sollhui commented on code in PR #67678:
URL: https://github.com/apache/doris/pull/67678#discussion_r4003336808
##########
fe/fe-core/src/main/java/org/apache/doris/load/routineload/kinesis/KinesisRoutineLoadJob.java:
##########
@@ -214,6 +316,10 @@ public void divideRoutineLoadJob(int
currentConcurrentTaskNum) throws UserExcept
writeLock();
try {
if (state == JobState.NEED_SCHEDULE) {
+ if (!getUnresolvedLatestShards().isEmpty()) {
Review Comment:
[P1] Keep task creation blocked until the pending initialization batch is
persisted
`getUnresolvedLatestShards()` only examines the current OPEN shards, but the
tasks below also include CLOSED shards. If an initial shard A is being scanned
and is split before that scan completes, the next metadata refresh moves A to
`closedKinesisShards` and initializes its children at `TRIM_HORIZON`. When no
other OPEN shard has an unresolved LATEST position, this guard passes even
though `latestSequenceFetch` is still pending. The job then enters RUNNING with
a task containing `A=LATEST`, before the initialization result has been
journaled. This bypasses the durable-start-position guarantee, and the normal
scheduler no longer calls `prepare()` for that RUNNING job to collect the
pending result.
I reproduced the gate bypass in an isolated Java harness using five
unchanged methods from this PR, with infrastructure stubbed: before the split
there are no tasks and the state is NEED_SCHEDULE; after refreshing the
topology and calling `divideRoutineLoadJob(1)`, the state is RUNNING, the scan
is still pending, and task positions are `{parent=LATEST, child-0=-2,
child-1=-2}`. This is not a full Doris/AWS integration test.
Please also block scheduling while an initialization batch remains
uncollected/unpersisted, including when its requested shards become CLOSED. A
regression test covering pending initialization -> split/merge -> scheduler
round should assert that no tasks are created until the result is journaled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
