maks3201 opened a new issue, #66560: URL: https://github.com/apache/doris/issues/66560
### Search before asking - [x] I searched in the [issues](https://github.com/apache/doris/issues) and found no similar issues. ### Version master (4.2-SNAPSHOT), also affects 4.1.x ### What's Wrong In compute-storage-decoupled (cloud) mode, when a streaming insert job is created for the first time (or after MetaService loses its progress data), `replayOnCloudMode()` queries MetaService for persisted job progress. MetaService responds with `STREAMING_JOB_PROGRESS_NOT_FOUND`, which is the expected answer when no transaction has been committed yet. However, `replayOnCloudMode()` returns `void`, so callers cannot distinguish "no progress exists" from "progress loaded successfully." As a result: 1. **In `handlePendingState()`** (StreamingJobSchedulerTask): the scheduler calls `replayOnCloudMode()` on every tick, receives the same NOT_FOUND response, and never transitions the job out of PENDING. 2. **In journal replay**: every `UPDATE_JOB` edit-log entry triggers another doomed RPC, flooding the FE log with WARN messages during startup. **Observed symptoms:** - Streaming job remains permanently stuck in PENDING state - FE log fills with repeated lines: `not found streaming job progress, response: ...` - In severe cases (many streaming jobs), the log volume during journal replay delays FE startup ### How to Reproduce 1. Deploy Doris in compute-storage-decoupled mode 2. Create a streaming insert job (e.g., CDC from MySQL) 3. Observe the job status — it stays PENDING indefinitely 4. Check `fe.log` for repeated warnings about `not found streaming job progress` The issue is 100% reproducible for any newly created streaming job that has not yet committed its first transaction. It also occurs for existing jobs if MetaService's FDB backing store loses the progress key. ### Root Cause `replayOnCloudMode()` has return type `void`. When it encounters `STREAMING_JOB_PROGRESS_NOT_FOUND`, it logs a warning and returns without any signal to the caller. The caller has no way to know the attempt was a permanent "not found" rather than a successful recovery, so it retries every scheduler tick. ### Expected Behavior - `replayOnCloudMode()` should return a boolean indicating whether progress was recovered - On NOT_FOUND, the job should proceed using its configured `offset` property rather than waiting for cloud-persisted progress - Repeated calls for the same job should be suppressed until the job actually commits a transaction (at which point progress will exist in MetaService) ### Anything Else This affects any deployment using the compute-storage-decoupled architecture. The fix is straightforward: change the return type to boolean and add a transient flag to prevent repeated RPCs. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
