Aranya7 opened a new pull request, #69929:
URL: https://github.com/apache/doris/pull/69929
### What problem does this PR solve?
Issue Number: close #69928
Related PR: N/A
**Problem Summary:**
Routine Load jobs can silently revert to default execution properties after
a Frontend (FE) restart or metadata recovery, even when nondefault values were
explicitly configured during job creation.
The affected properties include:
- `max_filter_ratio`
- `exec_mem_limit`
- `send_batch_parallelism`
- `load_to_single_tablet`
These properties are correctly initialized when a job is created. However,
their corresponding runtime fields are not independently serialized, and the
existing metadata recovery path does not restore them from the persisted
configuration.
As a result, a successfully recovered Routine Load job may execute with
settings that differ from its user-defined configuration, without an explicit
`ALTER ROUTINE LOAD` operation.
**Impact**
This issue can affect ingestion correctness and operational reliability:
- **Data-quality enforcement:** `max_filter_ratio` can revert from a strict
value such as `0.0` to the default `1.0`, allowing a substantially higher
proportion of filtered rows to be tolerated than intended.
- **Resource management:** `exec_mem_limit` can revert to its default,
potentially changing task memory consumption or causing unexpected memory-limit
failures.
- **Ingestion performance:** `send_batch_parallelism` can revert to its
default, changing batch submission parallelism and throughput.
- **Tablet-loading behavior:** `load_to_single_tablet` can revert to
`false`, changing the configured loading strategy.
The primary concern is that an FE restart can silently alter ingestion
semantics. This PR addresses that risk; it does not claim that a specific
production data-loss incident has occurred.
**Root Cause**
`RoutineLoadJob` maintains the affected configuration values in runtime
fields that are not independently persisted in FE metadata.
Although some of these values are available in the serialized
`jobProperties` map, `gsonPostProcess()` reconstructs the Routine Load
description without restoring the corresponding runtime fields.
Additionally, `exec_mem_limit` was not previously recorded in
`jobProperties` during job creation.
Consequently, metadata deserialization can reconstruct a valid job object
while leaving its execution properties initialized to defaults.
**Changes**
1. **Persist the effective execution memory limit**
- Store the resolved `exec_mem_limit` in `jobProperties` during Routine
Load job creation.
2. **Restore runtime configuration during metadata recovery**
- Restore `max_filter_ratio`, `exec_mem_limit`, `send_batch_parallelism`,
and `load_to_single_tablet` during `gsonPostProcess()`.
- Apply the following recovery precedence:
1. Persisted `jobProperties`, including values modified through ALTER.
2. Original CREATE statement, for properties missing from older
metadata.
3. Default configuration, when neither source provides a value.
3. **Preserve ALTER semantics**
- Persisted configuration takes precedence over the original CREATE
statement, which may no longer reflect the job's latest settings.
- For example, if `max_filter_ratio` was initially `0.5` and subsequently
altered to `0.0`, the recovered value remains `0.0`.
4. **Validate recovered properties**
- Parse and validate the recovered values before updating runtime fields.
- Invalid persisted configuration follows the existing recovery
error-handling path rather than being silently accepted.
5. **Add regression coverage**
- Cover configuration persistence and recovery through Gson
serialization/deserialization, legacy metadata compatibility, ALTER precedence,
invalid configuration, repeated recovery, and Kafka/Kinesis job types.
**Backward Compatibility**
The fix uses the existing serialized `jobProperties` map and the original
CREATE statement. It does not introduce new metadata fields or require a
serialization-format migration.
For existing jobs whose metadata does not contain `exec_mem_limit`, the
value can be recovered from the original CREATE statement when available.
This is particularly important during upgrades: an existing job may have
been executing with an unintended default after an earlier FE restart. Once the
fix is applied, its effective configuration can return to the value originally
specified by the user.
The change is limited to configuration persistence and recovery. It does not
intentionally modify job identity, Kafka partition offsets, ingestion progress,
or task scheduling logic.
### Release note
Fix Routine Load jobs unexpectedly reverting to default execution properties
after FE restart or metadata recovery.
### Check List (For Author)
- Test
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
**Unit Test**
Added `RoutineLoadConfigRecoveryTest` to cover:
- Recovery of nondefault runtime properties after
serialization/deserialization.
- Recovery of Kafka and Kinesis Routine Load jobs.
- Recovery of legacy metadata missing persisted configuration values.
- Precedence of ALTER-modified properties over the original CREATE statement.
- Handling of invalid persisted values.
- Repeated serialization/deserialization.
- Persistence of the resolved `exec_mem_limit` during job creation.
Existing Routine Load unit tests should also be executed to check for
regressions.
**Manual Test**
1. Start an unpatched FE and create a Routine Load job with:
```sql
PROPERTIES (
"max_filter_ratio" = "0.5",
"exec_mem_limit" = "3333333333",
"send_batch_parallelism" = "2",
"load_to_single_tablet" = "true"
)
```
2. Pause the job and modify its filter ratio:
```sql
ALTER ROUTINE LOAD FOR rl_recovery_legacy
PROPERTIES (
"max_filter_ratio" = "0.0"
);
```
3. Capture its configuration, job ID, state, and Kafka progress using `SHOW
ROUTINE LOAD`.
4. Gracefully stop the unpatched FE, preserve its metadata, and start the
patched FE using the same metadata.
5. Verify the recovered job retains:
| Property | Expected value |
|---|---|
| `max_filter_ratio` | `0.0` |
| `exec_mem_limit` | `3333333333` |
| `send_batch_parallelism` | `2` |
| `load_to_single_tablet` | `true` |
6. Verify that the job ID, paused state, and stored Kafka offsets remain
unchanged.
7. Create a new Routine Load job using the patched FE with the same
nondefault properties. Restart the patched FE and verify that its configuration
is also preserved.
**Test results:** Update the checkboxes and include actual execution results
before submitting or marking validation complete.
- Behavior changed:
- [ ] No.
- [x] Yes.
Routine Load jobs now recover their explicitly configured execution
properties instead of silently reverting to defaults.
For existing jobs previously affected by this issue, the effective
execution settings may change on the first FE restart with the fix, restoring
the user-configured values. This is the intended correctness improvement.
- Does this need documentation?
- [x] No.
- [ ] Yes.
No new SQL syntax, configuration options, or user-facing APIs are
introduced. This change restores the expected behavior of existing Routine Load
properties.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
