maks3201 opened a new pull request, #66561:
URL: https://github.com/apache/doris/pull/66561
### What problem does this PR solve?
Issue Number: close #66560
Related PR: #66559 (independent fix in the same subsystem)
Problem Summary:
In compute-storage-decoupled (cloud) mode, `replayOnCloudMode()` asks
MetaService for a streaming job's persisted progress. When MetaService answers
`STREAMING_JOB_PROGRESS_NOT_FOUND` (expected for newly created jobs that have
not yet committed a transaction), the method logs a warning and returns `void`.
The caller — `handlePendingState()` in `StreamingJobSchedulerTask` — cannot
distinguish "no progress exists" from "progress loaded successfully", so it
repeats the RPC on every scheduler tick indefinitely.
**Consequences:**
- The job is stuck in PENDING forever
- FE log fills with repeated WARN messages
- During journal replay at FE startup, every `UPDATE_JOB` edit-log entry
triggers a doomed RPC
**Fix:**
- Change `replayOnCloudMode()` return type from `void` to `boolean` (returns
`false` on NOT_FOUND)
- Add a `transient volatile boolean cloudProgressMissing` flag that
short-circuits subsequent attempts
- Clear the flag in `afterCommitted()` when the job commits its first
transaction (progress now exists in MetaService)
- Guard the journal-replay call site with `!cloudProgressMissing` to avoid
repeated RPCs during startup
The flag is intentionally `transient` (not serialized): it resets on FE
restart, giving MetaService another chance if the issue was temporary. A
permanent NOT_FOUND is the expected state for newly created jobs — the fix
simply allows the job to proceed rather than spinning.
### Release note
Fix streaming insert job stuck in PENDING state with infinite MetaService
retries in compute-storage-decoupled (cloud) mode. When MetaService returns
STREAMING_JOB_PROGRESS_NOT_FOUND, the job now proceeds with its configured
offset instead of retrying indefinitely.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (all round) — verified on a compute-storage-decoupled
cluster: new job transitions from PENDING to RUNNING within one scheduler tick,
log spam eliminated
- [ ] No need to test or manually tested. Explain why:
- [ ] This is a refactor/code-cleanup without behavior change.
- Code quality <!-- Confirm the following: -->
- [x] Does not introduce new code style issues (checked with
`checkstyle`)
- [x] Does not introduce new `@Nullable` warnings
- [x] Boundary conditions and error handling are adequate
### Note to Reviewers
A unit test mocking `MetaServiceProxy` to return
`STREAMING_JOB_PROGRESS_NOT_FOUND` and asserting the method returns `false` /
the flag suppresses a second call is feasible and straightforward. I have not
included one in this PR — happy to add it if reviewers prefer. The behavioral
correctness was verified on a live cluster.
CC @JNSimba — as the streaming-job subsystem maintainer.
If this should be backported to 4.1, please apply the `dev/4.1.x` label (I
cannot as a non-committer).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
