morningman opened a new issue, #69783: URL: https://github.com/apache/doris/issues/69783
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version branch-4.1 (checked at `5fc51d21256`) and branch-4.2 (checked at `9f63be85cea`), with the routine load `compute_group` binding from #67010. ### What's Wrong? The routine load `compute_group` binding has three gaps, and the workload group check involved in the first one also fails without any compute group. Files: `fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateRoutineLoadInfo.java`, `fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java`, `fe/fe-core/src/main/java/org/apache/doris/resource/computegroup/ComputeGroupBindingUtil.java`. 1. **Metadata load can cancel a recoverable job.** On metadata load, `RoutineLoadJob.gsonPostProcess` re-parses the CREATE statement with `setReplay(true)`. That skips the compute group check, but not the workload group existence check that follows, and any exception there sets the job to the final `CANCELLED`, which RESUME cannot undo. - That check resolves the declared group through `ComputeGroupMgr.getComputeGroupByName`, which throws once the group is dropped, renamed or scaled to zero. `origStmt` keeps the CREATE-time `compute_group`, so a job moved to another group by ALTER is still cancelled on the next restart after its old group is dropped. - The check also fails on image load with no compute group at all. `PersistMetaModules` lists `routineLoadJobs` before `workloadGroups` (same order on both branches), so when the jobs are re-parsed the Env still holds the empty `WorkloadGroupMgr` its constructor created. A job created with `workload_group` gets `Can not find workload group <name> in compute group <tag>` and is cancelled by the first restart that loads it from an image. This part was shown by a unit test with an empty `WorkloadGroupMgr`; it was not reproduced on a cluster. 2. **A suspended declared group is not started.** `validateDeclaredComputeGroup` performs the USAGE and existence checks of `USE @<group>` but not its `waitForAutoStart`. A CREATE / ALTER that declares a suspended group from a session on another group therefore fails its Kafka metadata lookup (custom partitions, offset conversion). 3. **A combined ALTER converts offsets in the old group.** In a single `ALTER ROUTINE LOAD` that sets both `compute_group` and `kafka_offsets`, `KafkaRoutineLoadJob.modifyProperties` calls `convertOffset` before `modifyPropertiesInternal` installs the new properties. The conversion RPC goes to the job's old group and fails if that group is gone, although the requested group is usable. ### What You Expected? - A job survives a restart whatever happened to the groups it declared; the next task reports a missing or unusable group instead. - CREATE / ALTER that declares a suspended group starts it, as `USE @<group>` does. - A combined ALTER converts its offsets in the group it declares. ### How to Reproduce? 1. Cloud mode: create a Kafka routine load job with `compute_group = 'cg_a'` and `workload_group = 'wg_a'`, ALTER it to `compute_group = 'cg_b'`, drop `cg_a`, then restart the FE so the job is re-parsed from the journal or image. The job ends `CANCELLED`. 2. Any mode: create a routine load job with `workload_group = 'wg_a'`, let a checkpoint write an image, and restart the FE from that image. The job ends `CANCELLED` (inferred from the module order; shown by a unit test, not on a cluster). 3. Cloud mode with auto-start: from a session on a live group, CREATE a routine load job with custom `kafka_partitions` and `compute_group` set to a suspended group. The CREATE fails its Kafka metadata request. 4. Cloud mode: `ALTER ROUTINE LOAD ... PROPERTIES ('compute_group' = 'live_b') FROM KAFKA ('kafka_partitions' = '0', 'kafka_offsets' = 'OFFSET_END')` on a job whose current group has been dropped. The ALTER fails in the offset conversion. ### Anything Else? Fixed on master in #69749 (commit `4ed5856f57c`): the replay path adopts the workload group name without resolving it, CREATE / ALTER start the declared group with `waitForAutoStart` (on the statement thread, before the job lock), and the offset conversion receives the group the ALTER declares. Its unit tests in `RoutineLoadComputeGroupTest` (`testMetadataLoadDoesNotResolveWorkloadGroupInDeclaredComputeGroup`, `testImageLoadAdoptsWorkloadGroupBeforeWorkloadGroupsAreLoaded`, `testCreateWakesDeclaredComputeGroup`, `testAlterWakesDeclaredComputeGroup`, `testCombinedAlterConvertsOffsetsInTheGroupItDeclares`) each fail without the fix. Two related points raised for async MVs were left unchanged on master, matching branch-4.x: a `SET_VAR(compute_group = ...)` hint in the MV query overrides the declaration (the existing precedence of statement hints over object properties, which `workload_group` follows too), and the compute and workload groups are read in two separate reads at refresh start. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
