Doris-Breakwater commented on issue #69928: URL: https://github.com/apache/doris/issues/69928#issuecomment-6562934541
Breakwater-GitHub-Analysis-Slot: slot_23f973b75d65 The reported behavior is supported by the source at `8e10b235cf030d486b6840cfd64b6218ab515e6d`: there is a gap between persisted Routine Load properties and the runtime fields reconstructed during recovery. This warrants an FE Routine Load persistence fix. The issue is currently open with no labels; suggested triage areas are bug, ingestion/Routine Load, and FE metadata recovery. This assessment is based on read-only source inspection; I have not independently run the FE/Kafka restart reproduction, and it does not establish a production data-loss incident or the affected release range. **Verified code mechanism** - The four runtime fields lack `@SerializedName`, while `jobProperties` is serialized as `jp`. The configured Gson serialization strategy excludes unannotated fields. Their initialization defaults are `1.0`, `2147483648`, `1`, and `false`, respectively. See [RoutineLoadJob fields/defaults](https://github.com/apache/doris/blob/8e10b235cf030d486b6840cfd64b6218ab515e6d/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java#L121-L209), [Gson configuration](https://github.com/apache/doris/blob/8e10b235cf030d486b6840cfd64b6218ab515e6d/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java#L650-L662), and [exclusion strategy](https://github.com/apache/doris/blob/8e10b235cf030d486b6840cfd64b6218ab515e6d/fe/fe-common/src/main/java/org/apache/doris/persist/gson/GsonUtilsBase.java#L87-L102). - Creation through `setOptional()` sets all four runtime fields, but records only the ratio, parallelism, and single-tablet setting in `jobProperties`; it does not record `exec_mem_limit`. Recovery reparses and validates the original CREATE statement, then calls `setRoutineLoadDesc()`, without restoring these four fields. This explains successful reconstruction with default runtime values. See [creation](https://github.com/apache/doris/blob/8e10b235cf030d486b6840cfd64b6218ab515e6d/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java#L365-L408) and [serialization/recovery](https://github.com/apache/doris/blob/8e10b235cf030d486b6840cfd64b6218ab515e6d/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java#L1963-L2063). - These are operational fields: [Kafka task construction](https://github.com/apache/doris/blob/8e10b235cf030d486b6840cfd64b6218ab515e6d/fe/fe-core/src/main/java/org/apache/doris/load/routineload/kafka/KafkaRoutineLoadJob.java#L1094-L1124) passes them to `NereidsRoutineLoadTaskInfo`; [planning](https://github.com/apache/doris/blob/8e10b235cf030d486b6840cfd64b6218ab515e6d/fe/fe-core/src/main/java/org/apache/doris/nereids/load/NereidsStreamLoadPlanner.java#L317-L319) uses the memory limit, and [sink initialization](https://github.com/apache/doris/blob/8e10b235cf030d486b6840cfd64b6218ab515e6d/fe/fe-core/src/main/java/org/apache/doris/nereids/load/NereidsLoadPlanInfoCollector.java#L325-L333) uses parallelism and single-tablet loading. The FE's [error-rate check](https://github.com/apache/doris/blob/8e10b235cf030d486b6840cfd64b6218ab515e6d/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java#L931-L984) reads `maxFilterRatio` directly. **Important reproduction details** `SHOW ROUTINE LOAD` alone cannot prove that all four effective values survived recovery. Its [JSON renderer](https://github.com/apache/doris/blob/8e10b235cf030d486b6840cfd64b6218ab515e6d/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java#L1890-L1933) merges the persisted map into the output. Consequently, the ratio, parallelism, and single-tablet setting can still display their configured values while the corresponding runtime fields have reverted. `SHOW CREATE ROUTINE LOAD` reads the runtime ratio and memory limit, but omits the other two properties in its [DDL renderer](https://github.com/apache/doris/blob/8e10b235cf030d486b6840cfd64b6218ab515e6d/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java#L1808-L1828). Keep that rendering issue separate, and assert runtime fields/task options in recovery tests. ALTER ordering matters. [Updating the ratio](https://github.com/apache/doris/blob/8e10b235cf030d486b6840cfd64b6218ab515e6d/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java#L2083-L2086) updates both its field and persisted map. [ALTER replay](https://github.com/apache/doris/blob/8e10b235cf030d486b6840cfd64b6218ab515e6d/fe/fe-core/src/main/java/org/apache/doris/load/routineload/kafka/KafkaRoutineLoadJob.java#L919-L927) can therefore restore that field after deserialization, whereas loading an image that already incorporates the ALTER still encounters the missing restoration. Test both paths. At this commit, the [ALTER property allowlist](https://github.com/apache/doris/blob/8e10b235cf030d486b6840cfd64b6218ab515e6d/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterRoutineLoadCommand.java#L65-L87) permits `max_filter_ratio`, but not the other three properties; do not assume all four can be repaired with ALTER. The data-quality impact also depends on `max_error_number`: it independently triggers pausing and defaults to zero. A controlled behavioral test should select an error-count allowance that does not mask the ratio check. The supplied example establishes a configuration concern, but does not by itself demonstrate additional rows being filtered without pausing. **Suggested fix and validation** 1. Persist the effective `exec_mem_limit` on creation and restore these runtime fields during recovery. Resolve each property from the persisted map first, an explicitly supplied original CREATE property second, and the normal default only when neither exists. Do not blindly reapply all CREATE options: that would overwrite newer ALTER values. 2. Add tests exercising the real `write()`/`read()` recovery path with a valid catalog/table context, covering nondefault CREATE values, ALTER ratio precedence, older metadata missing individual keys, absent options, and explicit `0.0`/`false`. Include image recovery and subsequent ALTER journal replay, plus a second serialization round trip. 3. Assert job identity, persisted state, and Kafka progress/offset preservation before scheduling, then check recovered task options. Keep the change focused on configuration restoration and preserve existing recovery-failure handling. **Remaining evidence requested** Please attach the actual before/after outputs of both SHOW commands, the destination table DDL, any intervening ALTER statements, and the relevant FE startup image-loading/journal-replay log excerpt with job ID and timestamps. For parallelism and single-tablet behavior, provide a failing recovery unit test or captured runtime/task-option values because SHOW output can conceal the discrepancy. Redact broker addresses and credentials. A query profile is not needed to establish this serialization defect. The supplied commit and reproduction outline are sufficient to begin the focused fix; the additional evidence would validate the exact observed recovery path and impact. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
