Doris-Breakwater commented on issue #66560: URL: https://github.com/apache/doris/issues/66560#issuecomment-5210066597
Breakwater-GitHub-Analysis-Slot: slot_2bed6094f41f Maintainer triage: **the reported PENDING root cause is contradicted by the current control flow.** The journal-replay RPC/log amplification is real in the code, but the issue does not yet demonstrate that `STREAMING_JOB_PROGRESS_NOT_FOUND` keeps a job PENDING. I would not accept the linked PR #66561 as-is without a reproducer and tests. ### Verified facts - I checked `upstream/master` at `4e3c1b84dd5936b8d751e0a95719350e8cd8e980` and `upstream/branch-4.1` at `e0115cac0cd7fcef867801469a05ceb2b3ca1616`; both have the same relevant control flow. - MetaService returns `STREAMING_JOB_PROGRESS_NOT_FOUND` when the `(instance_id, db_id, job_id)` progress key is absent. The first streaming transaction commit creates that key, so NOT_FOUND is expected for a genuinely new, never-committed job. - On NOT_FOUND, `StreamingInsertJob.replayOnCloudMode()` returns normally from that method. In `StreamingJobSchedulerTask.handlePendingState()`, execution then continues to `replayOffsetProviderIfNeed()`, `advanceSplitsIfNeed()`, task creation, and `updateJobStatus(RUNNING)`. The only early returns in this path are for an RPC/error response (which pauses the job), an independently paused split-advance path, or a source that has reached its end. Therefore the `void` return type does not cause a scheduler tick to stop, and NOT_FOUND alone cannot explain a permanent PENDING state. - The journal concern is mechanically valid: `JobManager.replayUpdateJob()` calls `StreamingInsertJob.replayOnUpdated()`, which unconditionally calls `replayOnCloudMode()` in cloud mode. Thus each replayed streaming `UPDATE_JOB` can issue a synchronous MetaService RPC. For NOT_FOUND, the current implementation also emits two WARN messages (the generic failure warning and the specific not-found warning). The claimed startup delay/volume is plausible, but no timing, entry count, or log sample was supplied to quantify it. - The issue currently has no labels, assignee, or discussion. It is cross-referenced by PR #66561. ### Assessment of PR #66561 - The new boolean result is used by `handlePendingState()` only for an INFO message; both `true` and `false` still follow the same task-creation/status-transition path. Consequently, that change cannot account for the PR's claimed before/after PENDING behavior. - The transient `cloudProgressMissing` flag can suppress repeated journal-replay RPCs after the first NOT_FOUND, but the PR has no automated test for that behavior. - A missing key has two materially different meanings: “this job has never committed” and “previously committed MetaService progress was lost.” Silently treating the latter as a fresh start can replay data or create duplicates, depending on what FE/provider state remains. The recovery policy must distinguish or explicitly guard these cases before promising fallback to the configured initial offset. - The flag is cleared only by the active `afterCommitted()` callback. Tests should also cover journal replay, FE role change, and progress becoming available through another recovery path so an in-process negative cache cannot mask newly available progress. ### Information needed to confirm the reported incident 1. The exact FE build/git SHA (not only `master` or `4.1.x`) and the sanitized `CREATE JOB` statement, including source/job offset mode. 2. The affected job ID plus status/task output showing that it remains PENDING across multiple scheduler intervals. 3. A contiguous FE log excerpt from job creation through at least two scheduler intervals, including `not found streaming job progress`, `create new streaming insert task`, `Streaming insert job ... update status`, and any exceptions from offset-provider replay, split advancement, source-reader setup, task registration, or scheduling. 4. For the startup-impact claim: number of streaming jobs and replayed `UPDATE_JOB` entries, count of MetaService calls/WARN lines, and startup timing with timestamps. 5. For the “MetaService lost progress” case: evidence that the job had previously committed and what recoverable FE/provider offset state still existed. This case should not be tested by simply resuming from the original configured offset without validating duplicate-load safety. ### Recommended next steps 1. Add an FE test with a mocked MetaService response proving the current baseline: NOT_FOUND must still create the task and transition PENDING to RUNNING in the same scheduler invocation. If that passes—as the control flow indicates—the reported stuck state has another cause and the supplied logs should identify the actual failing step. 2. Address journal amplification separately: perform cloud recovery once per job/recovery epoch (or after edit-log replay completes), treat first-time NOT_FOUND as an expected INFO-level outcome, and avoid the duplicate WARN. Do not use a lifetime-wide negative cache without defining invalidation for commit replay and FE role transitions. 3. Add tests for multiple `UPDATE_JOB` records with an absent key, successful recovery, non-NOT_FOUND/RPC errors, a later-appearing progress key, and a previously committed job whose MetaService key is missing. No source changes were made during this triage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
