enocknkhoma opened a new pull request, #65555:
URL: https://github.com/apache/doris/pull/65555
### What problem does this PR solve?
Issue Number: No related issue
Related PR: #40728, #53768
Problem Summary:
In compute-storage decoupled (cloud) mode, if the edit log contains an
OP_CHANGE_ROUTINE_LOAD_JOB (Operation Type 201) entry whose routine load job no
longer exists in memory (the job was cleaned up while a later change op for it
remains in the journal), FE throws a NullPointerException during journal replay
and cannot start. All FEs replaying the same journal crash
identically, so the entire FE cluster becomes unrecoverable.
`force_skip_journal_id` cannot bypass it, as it only covers journals that fail
to deserialize, not replay exceptions.
Observed in production on 3.1.0 (replayer thread):
```
2026-07-11 07:35:25,666 INFO (replayer|15) [EditLog.loadJournal():379] Begin
to unprotect drop table: internal.1771306616047.stg_simreg
2026-07-11 07:35:25,666 INFO (replayer|15)
[CatalogRecycleBin.recycleTable():188] recycle table[1782934238385-stg_simreg],
is force drop: false
2026-07-11 07:35:25,666 INFO (replayer|15)
[InternalCatalog.unprotectDropTable():1059] finished dropping table[stg_simreg]
in db[cdrs] recycleTable cost: 0ms
2026-07-11 07:35:25,668 INFO (replayer|15) [EditLog.loadJournal():361] Begin
to unprotect create table. internal.cdrs.stg_simreg
2026-07-11 07:35:25,669 ERROR (replayer|15) [EditLog.loadJournal():1438]
replay Operation Type 201, log id: 24831862
java.lang.NullPointerException: Cannot invoke
"org.apache.doris.load.routineload.RoutineLoadJob.setCloudCluster()" because
the return value of
"org.apache.doris.cloud.load.CloudRoutineLoadManager.getJob(long)" is null
at
org.apache.doris.cloud.load.CloudRoutineLoadManager.replayChangeRoutineLoadJob(CloudRoutineLoadManager.java:70)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.persist.EditLog.loadJournal(EditLog.java:875)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.replayJournal(Env.java:3103)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env$4.runOneCycle(Env.java:2865)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.common.util.Daemon.run(Daemon.java:119)
~[doris-fe.jar:1.2-SNAPSHOT]
```
Reproduced identically on 3.1.1 and 3.1.4 (on 3.1.4 also via the state
transfer path during master election: Env.transferToMaster ->
Env.replayJournal, same NPE at CloudRoutineLoadManager.java:70). The unguarded
call is still present in 4.1.2 and master.
#40728 and #53768 fixed the illegal-state and null-field cases in this same
replay path; this PR covers the remaining null-job case with the same defensive
pattern (as used in CloudSystemInfoService.replayModifyBackend).
Note: the base RoutineLoadManager.replayChangeRoutineLoadJob has the same
unguarded getJob() dereference on the non-cloud path; this PR fixes the cloud
override where the crash was observed, the base class can be guarded in a
follow-up.
Fix: null-check the job and skip the edit log entry with a warning,
consistent with how a change op for an already-removed job should be treated.
### Release note
Fix FE start failure (NPE) in cloud mode when replaying
OP_CHANGE_ROUTINE_LOAD_JOB for a routine load job that no longer exists.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Manual test (add detailed scripts or steps below)
Replayed a production edit log containing an orphaned
OP_CHANGE_ROUTINE_LOAD_JOB entry in a 3-FE cloud-mode sandbox (FDB + Meta
Service). Without this fix, all FEs crash-loop at the same log id (verified on
3.1.0/3.1.1/3.1.4). With the fix, the FE logs a warning, replay completes,
master election succeeds, and routine load jobs resume normally.
- Behavior changed:
- [x] No.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
