ryanmeowy opened a new pull request, #13147:
URL: https://github.com/apache/seatunnel/pull/13147
### Purpose of this pull request
Fixes #13146
On the Flink engine, `env.min-pause` is silently dropped whenever
`checkpoint.timeout` is
configured: the job starts, no warning is logged, and Flink keeps the
default minimum pause of
0 ms, so checkpoints are triggered back to back. The setting is documented
(`docs/en/architecture/fault-tolerance/checkpoint-mechanism.md`) and
accepted by `--check`, so the
job looks correctly configured.
**Root cause.** In `AbstractFlinkRuntimeEnvironment.setCheckpoint()`
(`:108-117` on the parent
commit), `min-pause` is the third `else if` branch behind the two
`checkpoint.timeout` keys:
```java
if (config.hasPath(ENV_CHECKPOINT_TIMEOUT)) {
checkpointConfig.setCheckpointTimeout(...);
} else if (EnvironmentUtil.hasPathAndWaring(config,
LEGACY_CHECKPOINT_TIMEOUT)) {
checkpointConfig.setCheckpointTimeout(...);
} else if (config.hasPath(ENV_MIN_PAUSE)) { // never reached when a
timeout is set
checkpointConfig.setMinPauseBetweenCheckpoints(...);
}
```
The two settings are independent `CheckpointConfig` fields in Flink, so
chaining them is wrong.
**Fix.** Move the `min-pause` handling out of the timeout chain and apply it
in its own `if`,
which is what the legacy `execution.checkpoint.min-pause` block 50 lines
below already does.
The change also guards a pre-existing trap that this reshape would otherwise
make easier to hit:
`-1` is the declared default of the option
(`EnvCommonOptions.CHECKPOINT_MIN_PAUSE`) and the value
Zeta uses for "no minimum pause", while Flink rejects any negative value with
`IllegalArgumentException: Pause value must be zero or positive`. A negative
`min-pause` is now
skipped with a warning instead of failing the job at startup.
The legacy `execution.checkpoint.min-pause` block later in the method is
left unchanged.
Diff is limited to `AbstractFlinkRuntimeEnvironment.java` and
`TestFlinkParameter.java`.
### Does this PR introduce _any_ user-facing change?
Yes, within the unreleased `dev` branch: a configuration that used to be
accepted and then
silently ignored now takes effect. No released version behaves differently
after the fix than
before it, since the bug is present in every version that has the env
`min-pause` option.
| config | before | after |
|---|---|---|
| `checkpoint.timeout` + `min-pause = 10000` | job starts, Flink min pause =
**0** | job starts, Flink min pause = 10000 |
| `execution.checkpoint.timeout` + `min-pause = 10000` | **0** | 10000 |
| `min-pause = 10000` (no timeout) | 10000 | 10000 (unchanged) |
| `checkpoint.timeout` + `min-pause = -1` | job starts, min pause = 0 | job
starts, min pause = 0, one WARN log |
| `min-pause = -1` (no timeout) | **startup fails** with
`IllegalArgumentException` | job starts, min pause = 0, one WARN log |
No option name, default value, public API or data format is changed, so
`incompatible-changes.md`
is not affected. The last two rows are the deliberate part: `-1` means "not
set" for this option
and for Zeta, so it is treated as unset instead of being passed to
`CheckpointConfig`, which only
accepts zero or a positive value. Note that Flink accepts `min-pause`
together with
`execution.checkpoint.max-concurrent` > 1 on 1.15 (verified with
`CheckpointConfig#configure`), so
no new conflict is introduced there.
### How was this patch tested?
Unit tests, extending the existing `TestFlinkParameter` (the module has no
other test that builds
`FlinkRuntimeEnvironment`, so no new test class):
- `testMinPauseAppliesTogetherWithCheckpointTimeout` — `checkpoint.interval
= 60000`,
`checkpoint.timeout = 120000`, `min-pause = 10000`; asserts the checkpoint
timeout is 120000
(deliberately not the Flink default 600000, which would pass even without
the fix) and the min
pause is 10000.
- `testNegativeMinPauseIsIgnored` — same config with `min-pause = -1`;
asserts the job builds and
the min pause stays 0.
`FlinkRuntimeEnvironment` is a process-wide singleton, so the test helper
applies the env block
explicitly (`setConfig` + `setJobMode` + `prepare`) instead of relying on
whichever config created
the singleton first.
Commands run locally:
```bash
# the fix and its neighbouring tests
./mvnw -pl
seatunnel-core/seatunnel-flink-starter/seatunnel-flink-starter-common test \
-Dspotless.check.skip=true
# Tests run: 9, Failures: 0, Errors: 0 (TestFlinkParameter 4,
TransformExecuteProcessorTest 3,
# FlinkConfValidateCommandTest 2)
# format
./mvnw -pl
seatunnel-core/seatunnel-flink-starter/seatunnel-flink-starter-common
spotless:check
# BUILD SUCCESS
```
Neither test is vacuous:
- on the parent commit (`a3743d7f5`),
`testMinPauseAppliesTogetherWithCheckpointTimeout` fails with
`expected: <10000> but was: <0>`;
- `testNegativeMinPauseIsIgnored` guards the reshape this PR introduces:
with the independent `if`
but without the negative guard it fails with
`IllegalArgumentException: Pause value must be zero or positive`. (On the
parent commit the same
config passes only because `min-pause` was ignored altogether, so the
negative guard is not
covered by the old code path.)
I also checked the value survives
`StreamExecutionEnvironment#getStreamGraph()`
(`StreamGraph#getCheckpointConfig()` reports the same 10000), so the
assertion reflects what the
job runs with, not just a field on the environment object.
Not covered here: no Flink cluster E2E test. A reproducible E2E would need a
real Flink mini
cluster and a checkpoint interval short enough to observe timing; the unit
tests pin the
configuration values that drive it. Happy to add one if maintainers prefer
it.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
— not applicable, no new dependency or binary.
* [x] If necessary, please update the documentation to describe the new
feature.
— not applicable. The documentation already describes `min-pause` (and
shows it next to
`checkpoint.timeout`, which is the combination that was broken); this
change makes the code match
it. No option is added, renamed or re-defaulted.
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused
by this PR. — not applicable, no contract change: an accepted-but-ignored
setting now applies.
* [ ] If you are contributing the connector code, please check that the
following files are updated
— not applicable, this is not a connector change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]