maks3201 commented on PR #66561: URL: https://github.com/apache/doris/pull/66561#issuecomment-5222993208
Thank you for the detailed triage. You are right on the main point and I want to state that plainly. **What I got wrong.** The claim that `STREAMING_JOB_PROGRESS_NOT_FOUND` leaves a job stuck in PENDING is incorrect. I traced it again. `replayOnCloudMode()` does a plain `return` on NOT_FOUND, it does not throw, so the `catch` block in `handlePendingState()` never runs and execution continues to `createStreamingTask()` and `updateJobStatus(JobStatus.RUNNING)` in the same invocation (StreamingJobSchedulerTask.java:57-82). The job only goes to PAUSED when the RPC throws, which NOT_FOUND does not. I inferred the stuck state from reading the code and the inference was wrong. I also have no captured job status output or log excerpt showing a job in PENDING, so there was no observation behind it either. I also want to correct a second thing before it spreads. I had thought this produced repeated warnings on every scheduler tick. That is also wrong. `replayOnCloudMode()` is reachable from only two places, `handlePendingState()` and `replayOnUpdated()`. `handleRunningState()` only calls `processTimeoutTasks()`, `fetchMeta()` and `advanceSplitsIfNeed()`, so once the job is RUNNING there is no further call and no per tick repetition. **What actually holds up.** Only one thing, and it is smaller than I described. `EditLog` replay calls `JobManager.replayUpdateJob()` (EditLog.java:886), which reaches `replayOnUpdated()`, which calls `replayOnCloudMode()` unconditionally in cloud mode (StreamingInsertJob.java:1038-1040). So each replayed UPDATE_JOB entry issues one MetaService RPC with no deduplication, and for a job that never committed each one returns NOT_FOUND and logs two WARN lines. I should also retract the wording about blocking until timeout. NOT_FOUND is a normal fast response with a non OK status code, the timeout case only applies when MetaService is unreachable, so my "wedge FE startup" description was an overstatement. **What I am doing.** Closing this PR and the issue. What is left is a minor replay deduplication and a log level change, and it does not match what the issue and PR describe. Reframing them would leave a misleading history. If I revisit it, I will open a fresh narrow PR for replay deduplication only, with startup timing numbers and entry counts to justify it, plus the unit tests you asked for. Thank you for looking at this carefully. Your review caught a real error on my side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
