SEPURI-SAI-KRISHNA opened a new pull request, #13261:
URL: https://github.com/apache/seatunnel/pull/13261
Yes, glad to contribute to Apache SeaTunnel.
### Purpose of this pull request
Fixes #13253.
`SingleSplitEnumerator` (`connector-common`, used by 29 connector modules,
among them `connector-http-base`, Cassandra, Redis, MQTT and Socket) failed a
Flink job on its second restart:
1. On the first restart, Flink gives back the split of the failed reader.
`addSplitsBack` stored it, but `assigned` stayed `true`, so `assignSplit()`
returned and the restarted reader never got the split again.
2. On the second restart, nothing had been assigned since the last
checkpoint, so Flink called `addSplitsBack` with an empty list, `splits.get(0)`
threw `IndexOutOfBoundsException`, and `SourceCoordinator` failed over the job
with it, replacing the error that caused the restarts. #13253 has the trace
from the Flink 1.13, 1.15 and 1.18 e2e images.
The fix, in `addSplitsBack`: return when the list is empty, and set
`assigned = false` before `assignSplit()`, so a split that comes back is
assigned again when the reader registers. Nothing else changes: `run()`,
`registerReader()` and the state stay as they are, and a split is still
assigned to one reader at a time.
Notes for reviewers:
- In the CI run of #13253, the same test passed on the Zeta and Spark
containers and on the Flink 1.20 image, which does not restart the job.
- The fix stays in `connector-common` and does not depend on #13222. The
`execution.restart.strategy = "no"` line in the GraphQL e2e config of #13222
stays as it is: there it makes the job report the GraphQL error without
restarts.
- The same empty list should also arrive on the first restart after a
completed checkpoint, when the split was assigned before it. The fix covers
that case the same way, but I have not run it on Flink.
### Does this PR introduce _any_ user-facing change?
Yes, for Flink jobs with a source on `SingleSplitEnumerator` that restart:
the restarted reader gets its split again, and a second restart no longer fails
the job with `IndexOutOfBoundsException`, so the job reports its own error. No
option, API or state change.
### How was this patch tested?
- New `SingleSplitEnumeratorTest` (no test for this class existed), as asked
in #13253. With a mocked `SourceSplitEnumerator.Context`, it drives the
enumerator through `run()`, `registerReader()` and `addSplitsBack()` the way
Flink does when the reader fails (the reader is removed, the splits are given
back, a new reader registers), and checks each `context.assignSplit` call: the
reader and the split.
- `testFirstAssignmentWhenTheReaderRegistersBeforeRun` and
`testFirstAssignmentWhenRunBeforeTheReaderRegisters`: the first assignment, in
both orders, happens once, also when `run()` or `registerReader()` is called
again.
- `testSplitGivenBackIsAssignedAgainWhenTheReaderRegisters`: the split
given back is not assigned while no reader is registered, and is assigned
again, the same split, when the reader registers.
- `testEmptyListGivenBackIsHarmless`: an empty list, on the second restart
or while the reader holds the split, does not throw and assigns nothing.
- `testEmptyListGivenBackKeepsAPendingSplit`: an empty list that follows a
split given back before the reader registered does not drop it; the reader
still gets that split.
- Against the code on dev, 3 of the 5 fail (the two first-assignment tests
pass, so that path is unchanged). I also changed the fix on purpose: without
`assigned = false` 3 tests fail (1 assignment instead of 2); without the empty
check 2 fail with `IndexOutOfBoundsException: Index: 0`, as in the CI log of
#13253; with an empty list clearing the pending split,
`testEmptyListGivenBackKeepsAPendingSplit` fails.
- `connector-common`: 21 tests pass; `spotless:check` and `verify` clean. No
local Flink run; the CI Flink jobs run the connectors that use this class.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
(no new jar)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs (no user-facing
option)
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR. (not incompatible)
* [ ] If you are contributing the connector code, please check that the
following files are updated: (not a new connector)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]