maks3201 opened a new pull request, #66563: URL: https://github.com/apache/doris/pull/66563
## Proposed changes Related PR: #66559 (independent fix in the same subsystem) ### Problem Summary When an operator runs `PAUSE JOB` → `ALTER JOB ... offset='initial'` → `RESUME JOB` to force a fresh CDC snapshot, `JdbcSourceOffsetProvider` still holds the old binlog position and split progress in memory. On the next scheduler tick the job dispatches a task from the stale offset instead of starting a fresh snapshot. The operator believes they reset the job; they did not. **Why this is not just `clearSnapshotState()`:** The existing `clearSnapshotState()` method is invoked during the *normal* snapshot-to-binlog transition inside `updateOffset()`. It deliberately preserves `currentOffset`, `binlogOffsetPersist`, and `endBinlogOffset` because the incoming binlog phase depends on them. Our new `resetToInitialState()` is a strict superset: it calls `clearSnapshotState()` for the split/progress subset, then also clears the binlog-phase fields (`currentOffset`, `binlogOffsetPersist`, `endBinlogOffset`, `tableSchemas`, `hasMoreData`, `boundBackendId`), because the intent is a complete fresh start — not a phase transition within an active job. ### Fix 1. Add `SourceOffsetProvider.resetToInitialState()` as a `default` no-op method on the interface (S3 and other providers are unaffected). 2. Implement in `JdbcSourceOffsetProvider`: clear all cached split and binlog state (11 fields — see exhaustive list in the Javadoc). 3. In `StreamingInsertJob.alterJob()`: when the ALTER explicitly sets `offset` to `initial` or `snapshot`, call `resetToInitialState()` and null out `offsetProviderPersist` so that `replayIfNeed()` takes the fresh-start branch. After the reset, the next `handlePendingState()` tick sees empty state and initiates a full snapshot — identical to a brand-new job. ### Field-by-field reset audit | Field | Cleared? | Reason | |-------|----------|--------| | `remainingSplits` | ✅ via clearSnapshotState | Stale snapshot chunks | | `finishedSplits` | ✅ via clearSnapshotState | Old split records | | `chunkHighWatermarkMap` | ✅ via clearSnapshotState | Old HW would prevent re-splitting | | `committedSplitProgress` | ✅ via clearSnapshotState | Stale cursor | | `cdcSplitProgress` | ✅ via clearSnapshotState | Stale cursor; `noMoreSplits()` would return wrong answer | | `currentOffset` | ✅ → null | Prevents `getNextOffset` from returning stale binlog | | `binlogOffsetPersist` | ✅ → null | Prevents `replayIfNeed` from restoring old offset | | `endBinlogOffset` | ✅ → null | Stale end offset from previous phase | | `tableSchemas` | ✅ → null | Schema may have changed; will be re-fetched | | `hasMoreData` | ✅ → true | Fresh start should assume data exists | | `boundBackendId` | ✅ → 0 | Old pinning was for the binlog reader; snapshot may use different BE | | `jobId` | preserved | Identity | | `sourceType` | preserved | Configuration | | `sourceProperties` | preserved | Updated by alterJob itself | | `snapshotParallelism` | preserved | Configuration | | `cloudCluster` | preserved | Routing config, set externally | | `cachedSyncTables` | preserved | Re-set by replayIfNeed from job.getSyncTables() | ## Issue Number Closes #66562 ## Checklist - [x] I have read the [Contributing to Apache Doris](https://doris.apache.org/community/how-to-contribute) guide. - [x] I have completed the CLA agreement (via first-time PR bot prompt). - [x] I have reviewed the [PR naming conventions](https://doris.apache.org/community/how-to-contribute/pull-request#pull-request-title). - [x] I have added appropriate tests (unit test for `resetToInitialState` covering binlog-phase reset, snapshot-phase reset, idempotency, `noMoreSplits` correctness, and configuration preservation). --- cc @JNSimba — you maintain the streaming job subsystem; this touches the offset provider interface and the `alterJob()` flow. Please also consider applying `dev/4.1.x` label for backport — this bug affects any 4.1.x user attempting manual CDC recovery via ALTER JOB. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
