Aranya7 opened a new issue, #69928: URL: https://github.com/apache/doris/issues/69928
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version Apache Doris `master`, commit `8e10b235cf0`. FE version: `doris-0.0.0-8e10b235cf0` Reproduced in a single-FE development environment using the unpatched implementation. ### What's Wrong? Routine Load jobs can silently lose their explicitly configured execution properties after an FE restart or metadata recovery. The following properties are affected: - `max_filter_ratio` - `exec_mem_limit` - `send_batch_parallelism` - `load_to_single_tablet` These properties are correctly applied when a Routine Load job is initially created. However, after the FE restarts and reconstructs the job from persisted metadata, their effective runtime values can revert to defaults instead of retaining the user-configured values. For example, a job configured with: ```sql "max_filter_ratio" = "0.0", "exec_mem_limit" = "3333333333", "send_batch_parallelism" = "2", "load_to_single_tablet" = "true" ``` can recover with default runtime settings, including `max_filter_ratio = 1.0` and `exec_mem_limit = 2147483648`. **Potential impact** This is concerning because FE restarts are part of normal cluster maintenance and recovery, but should not change the execution semantics of existing ingestion jobs. - **Data-quality enforcement:** A strict `max_filter_ratio` of `0.0` can revert to `1.0`, allowing a much higher proportion of filtered rows to be tolerated. This can weaken ingestion validation and potentially result in unexpected data omissions. - **Memory management:** `exec_mem_limit` can revert to the default, changing resource constraints and potentially causing unexpected memory consumption or task failures. - **Ingestion performance:** `send_batch_parallelism` can revert to its default, changing task execution parallelism and throughput. - **Tablet-loading behavior:** `load_to_single_tablet` can revert to `false`, changing the configured distribution behavior of load tasks. The issue is especially problematic because the affected job may otherwise recover successfully, without an explicit `ALTER ROUTINE LOAD` operation or an immediately visible error indicating that its runtime configuration has changed. ### What You Expected? <html><head></head><body><p><span>Routine Load jobs should preserve their effective execution configuration across FE restarts and metadata recovery.</span></p><p><span>Specifically:</span></p><ol start="1"><li><p><span>All explicitly configured properties should retain their original values after recovery.</span></p></li><li><p><span>Properties modified through </span><code><span>ALTER ROUTINE LOAD</span></code><span> should retain their latest values rather than reverting to the original CREATE statement or defaults.</span></p></li><li><p><span>Recovery of metadata written by older FE versions should preserve nondefault configuration values wherever the persisted properties or original CREATE statement provide sufficient information.</span></p></li><li><p><span>Job identity, state, Kafka partition progress, and stored offsets should remain unaffected by configuration recovery.</span></p></li></ol><p><span>For the reproduction below, the expected values after restarting the FE ar e:</span></p> Property | Expected value -- | -- max_filter_ratio | 0.0 exec_mem_limit | 3333333333 send_batch_parallelism | 2 load_to_single_tablet | true <p><span>An FE restart should not silently alter user-defined ingestion behavior.</span></p></body></html> ### How to Reproduce? <html><head></head><body><p><strong><span>Prerequisites:</span></strong><span> A Doris cluster with an available BE, an accessible Kafka topic, and an OLAP destination table using random distribution (required for </span><code><span>load_to_single_tablet = true</span></code><span>).</span></p><p><strong><span>Step 1: Create a Routine Load job with nondefault properties.</span></strong></p><pre><code><span>CREATE ROUTINE LOAD rl_config_recovery_test ON rl_recovery_events PROPERTIES ( "max_filter_ratio" = "0.0", "exec_mem_limit" = "3333333333", "send_batch_parallelism" = "2", "load_to_single_tablet" = "true", "format" = "json" ) FROM KAFKA ( "kafka_broker_list" = "<broker-host>:9092", "kafka_topic" = "<test-topic>" );</span></code></pre><p><strong><span>Step 2: Inspect the job configuration before restarting the FE.</span></strong></p><pre><code><span>SHOW ROUTINE LOAD FOR rl_config_recovery_test\G SHOW CREATE ROUTINE LOAD FOR rl_config_recovery_test\G</span></code></pre><p><span>Verify that the effective properties reflect the configured values.</span></p><p><strong><span>Step 3: Gracefully restart the FE</span></strong><span>, preserving its existing metadata directory.</span></p><p><strong><span>Step 4: Inspect the recovered job configuration.</span></strong></p><pre><code><span>SHOW ROUTINE LOAD FOR rl_config_recovery_test\G SHOW CREATE ROUTINE LOAD FOR rl_config_recovery_test\G</span></code></pre><p><strong><span>Step 5: Compare the properties before and after recovery.</span></strong></p><p><span>The job may recover with default values instead of the explicitly configured settings.</span></p><p><span>For example:</span></p> Property | Before restart | After restart (affected behavior) -- | -- | -- max_filter_ratio | 0.0 | 1.0 exec_mem_limit | 3333333333 | 2147483648 send_batch_parallelism | 2 | Default value load_to_single_tablet | true | false <p><span>The failure is reproducible through FE metadata recovery without requiring any user-initiated modification to the job configuration.</span></p></body></html> ### Anything Else? **Initial root cause analysis** The issue appears to originate in `RoutineLoadJob` metadata serialization and deserialization. Certain execution properties are maintained in runtime fields that are not independently serialized into FE metadata. Although some of the corresponding values are retained in the serialized `jobProperties` map, the existing `gsonPostProcess()` recovery path reconstructs the Routine Load description without restoring these runtime fields from their persisted configuration. Additionally, `exec_mem_limit` is not originally recorded in `jobProperties` during job creation. Consequently, deserialization can reconstruct a valid Routine Load job while leaving its effective runtime configuration initialized to defaults. **Backward compatibility considerations** Persisted configuration should take precedence over the original CREATE statement because properties such as `max_filter_ratio` may have been modified through `ALTER ROUTINE LOAD`. For older metadata that does not contain a particular persisted property, the original CREATE statement can provide a fallback where the configured value is available. **Related issue** `SHOW CREATE ROUTINE LOAD` also omits supported properties such as `send_batch_parallelism` and `load_to_single_tablet`. This is a separate DDL-rendering issue and should be addressed independently of the runtime metadata recovery bug. **Proposed scope** A focused fix can restore the affected runtime configuration during FE metadata recovery, persist the missing execution memory-limit property, maintain backward compatibility with existing job metadata, and introduce regression tests for serialization and recovery. The primary concern is the silent change to configured ingestion semantics. No specific production data-loss incident is being claimed. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
