ryanmeowy opened a new pull request, #13147:
URL: https://github.com/apache/seatunnel/pull/13147

   ### Purpose of this pull request
   
   Fixes #13146
   
   On the Flink engine, `env.min-pause` is silently dropped whenever 
`checkpoint.timeout` is
   configured: the job starts, no warning is logged, and Flink keeps the 
default minimum pause of
   0 ms, so checkpoints are triggered back to back. The setting is documented
   (`docs/en/architecture/fault-tolerance/checkpoint-mechanism.md`) and 
accepted by `--check`, so the
   job looks correctly configured.
   
   **Root cause.** In `AbstractFlinkRuntimeEnvironment.setCheckpoint()` 
(`:108-117` on the parent
   commit), `min-pause` is the third `else if` branch behind the two 
`checkpoint.timeout` keys:
   
   ```java
   if (config.hasPath(ENV_CHECKPOINT_TIMEOUT)) {
       checkpointConfig.setCheckpointTimeout(...);
   } else if (EnvironmentUtil.hasPathAndWaring(config, 
LEGACY_CHECKPOINT_TIMEOUT)) {
       checkpointConfig.setCheckpointTimeout(...);
   } else if (config.hasPath(ENV_MIN_PAUSE)) {          // never reached when a 
timeout is set
       checkpointConfig.setMinPauseBetweenCheckpoints(...);
   }
   ```
   
   The two settings are independent `CheckpointConfig` fields in Flink, so 
chaining them is wrong.
   
   **Fix.** Move the `min-pause` handling out of the timeout chain and apply it 
in its own `if`,
   which is what the legacy `execution.checkpoint.min-pause` block 50 lines 
below already does.
   
   The change also guards a pre-existing trap that this reshape would otherwise 
make easier to hit:
   `-1` is the declared default of the option 
(`EnvCommonOptions.CHECKPOINT_MIN_PAUSE`) and the value
   Zeta uses for "no minimum pause", while Flink rejects any negative value with
   `IllegalArgumentException: Pause value must be zero or positive`. A negative 
`min-pause` is now
   skipped with a warning instead of failing the job at startup.
   
   The legacy `execution.checkpoint.min-pause` block later in the method is 
left unchanged.
   
   Diff is limited to `AbstractFlinkRuntimeEnvironment.java` and 
`TestFlinkParameter.java`.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, within the unreleased `dev` branch: a configuration that used to be 
accepted and then
   silently ignored now takes effect. No released version behaves differently 
after the fix than
   before it, since the bug is present in every version that has the env 
`min-pause` option.
   
   | config | before | after |
   |---|---|---|
   | `checkpoint.timeout` + `min-pause = 10000` | job starts, Flink min pause = 
**0** | job starts, Flink min pause = 10000 |
   | `execution.checkpoint.timeout` + `min-pause = 10000` | **0** | 10000 |
   | `min-pause = 10000` (no timeout) | 10000 | 10000 (unchanged) |
   | `checkpoint.timeout` + `min-pause = -1` | job starts, min pause = 0 | job 
starts, min pause = 0, one WARN log |
   | `min-pause = -1` (no timeout) | **startup fails** with 
`IllegalArgumentException` | job starts, min pause = 0, one WARN log |
   
   No option name, default value, public API or data format is changed, so 
`incompatible-changes.md`
   is not affected. The last two rows are the deliberate part: `-1` means "not 
set" for this option
   and for Zeta, so it is treated as unset instead of being passed to 
`CheckpointConfig`, which only
   accepts zero or a positive value. Note that Flink accepts `min-pause` 
together with
   `execution.checkpoint.max-concurrent` > 1 on 1.15 (verified with 
`CheckpointConfig#configure`), so
   no new conflict is introduced there.
   
   ### How was this patch tested?
   
   Unit tests, extending the existing `TestFlinkParameter` (the module has no 
other test that builds
   `FlinkRuntimeEnvironment`, so no new test class):
   
   - `testMinPauseAppliesTogetherWithCheckpointTimeout` — `checkpoint.interval 
= 60000`,
     `checkpoint.timeout = 120000`, `min-pause = 10000`; asserts the checkpoint 
timeout is 120000
     (deliberately not the Flink default 600000, which would pass even without 
the fix) and the min
     pause is 10000.
   - `testNegativeMinPauseIsIgnored` — same config with `min-pause = -1`; 
asserts the job builds and
     the min pause stays 0.
   
   `FlinkRuntimeEnvironment` is a process-wide singleton, so the test helper 
applies the env block
   explicitly (`setConfig` + `setJobMode` + `prepare`) instead of relying on 
whichever config created
   the singleton first.
   
   Commands run locally:
   
   ```bash
   # the fix and its neighbouring tests
   ./mvnw -pl 
seatunnel-core/seatunnel-flink-starter/seatunnel-flink-starter-common test \
     -Dspotless.check.skip=true
   # Tests run: 9, Failures: 0, Errors: 0   (TestFlinkParameter 4, 
TransformExecuteProcessorTest 3,
   #                                         FlinkConfValidateCommandTest 2)
   
   # format
   ./mvnw -pl 
seatunnel-core/seatunnel-flink-starter/seatunnel-flink-starter-common 
spotless:check
   # BUILD SUCCESS
   ```
   
   Neither test is vacuous:
   
   - on the parent commit (`a3743d7f5`), 
`testMinPauseAppliesTogetherWithCheckpointTimeout` fails with
     `expected: <10000> but was: <0>`;
   - `testNegativeMinPauseIsIgnored` guards the reshape this PR introduces: 
with the independent `if`
     but without the negative guard it fails with
     `IllegalArgumentException: Pause value must be zero or positive`. (On the 
parent commit the same
     config passes only because `min-pause` was ignored altogether, so the 
negative guard is not
     covered by the old code path.)
   
   I also checked the value survives 
`StreamExecutionEnvironment#getStreamGraph()`
   (`StreamGraph#getCheckpointConfig()` reports the same 10000), so the 
assertion reflects what the
   job runs with, not just a field on the environment object.
   
   Not covered here: no Flink cluster E2E test. A reproducible E2E would need a 
real Flink mini
   cluster and a checkpoint interval short enough to observe timing; the unit 
tests pin the
   configuration values that drive it. Happy to add one if maintainers prefer 
it.
   
   ### Check list
   
   * [x] If any new Jar binary package adding in your PR, please add License 
Notice according
     [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
     — not applicable, no new dependency or binary.
   * [x] If necessary, please update the documentation to describe the new 
feature.
     — not applicable. The documentation already describes `min-pause` (and 
shows it next to
     `checkpoint.timeout`, which is the combination that was broken); this 
change makes the code match
     it. No option is added, renamed or re-defaulted.
   * [x] If necessary, please update `incompatible-changes.md` to describe the 
incompatibility caused
     by this PR. — not applicable, no contract change: an accepted-but-ignored 
setting now applies.
   * [ ] If you are contributing the connector code, please check that the 
following files are updated
     — not applicable, this is not a connector change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to