SEPURI-SAI-KRISHNA opened a new pull request, #13261:
URL: https://github.com/apache/seatunnel/pull/13261

   Yes, glad to contribute to Apache SeaTunnel.
   
   ### Purpose of this pull request
   
   Fixes #13253.
   
   `SingleSplitEnumerator` (`connector-common`, used by 29 connector modules, 
among them `connector-http-base`, Cassandra, Redis, MQTT and Socket) failed a 
Flink job on its second restart:
   
   1. On the first restart, Flink gives back the split of the failed reader. 
`addSplitsBack` stored it, but `assigned` stayed `true`, so `assignSplit()` 
returned and the restarted reader never got the split again.
   2. On the second restart, nothing had been assigned since the last 
checkpoint, so Flink called `addSplitsBack` with an empty list, `splits.get(0)` 
threw `IndexOutOfBoundsException`, and `SourceCoordinator` failed over the job 
with it, replacing the error that caused the restarts. #13253 has the trace 
from the Flink 1.13, 1.15 and 1.18 e2e images.
   
   The fix, in `addSplitsBack`: return when the list is empty, and set 
`assigned = false` before `assignSplit()`, so a split that comes back is 
assigned again when the reader registers. Nothing else changes: `run()`, 
`registerReader()` and the state stay as they are, and a split is still 
assigned to one reader at a time.
   
   Notes for reviewers:
   
   - In the CI run of #13253, the same test passed on the Zeta and Spark 
containers and on the Flink 1.20 image, which does not restart the job.
   - The fix stays in `connector-common` and does not depend on #13222. The 
`execution.restart.strategy = "no"` line in the GraphQL e2e config of #13222 
stays as it is: there it makes the job report the GraphQL error without 
restarts.
   - The same empty list should also arrive on the first restart after a 
completed checkpoint, when the split was assigned before it. The fix covers 
that case the same way, but I have not run it on Flink.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes, for Flink jobs with a source on `SingleSplitEnumerator` that restart: 
the restarted reader gets its split again, and a second restart no longer fails 
the job with `IndexOutOfBoundsException`, so the job reports its own error. No 
option, API or state change.
   
   ### How was this patch tested?
   
   - New `SingleSplitEnumeratorTest` (no test for this class existed), as asked 
in #13253. With a mocked `SourceSplitEnumerator.Context`, it drives the 
enumerator through `run()`, `registerReader()` and `addSplitsBack()` the way 
Flink does when the reader fails (the reader is removed, the splits are given 
back, a new reader registers), and checks each `context.assignSplit` call: the 
reader and the split.
     - `testFirstAssignmentWhenTheReaderRegistersBeforeRun` and 
`testFirstAssignmentWhenRunBeforeTheReaderRegisters`: the first assignment, in 
both orders, happens once, also when `run()` or `registerReader()` is called 
again.
     - `testSplitGivenBackIsAssignedAgainWhenTheReaderRegisters`: the split 
given back is not assigned while no reader is registered, and is assigned 
again, the same split, when the reader registers.
     - `testEmptyListGivenBackIsHarmless`: an empty list, on the second restart 
or while the reader holds the split, does not throw and assigns nothing.
     - `testEmptyListGivenBackKeepsAPendingSplit`: an empty list that follows a 
split given back before the reader registered does not drop it; the reader 
still gets that split.
   - Against the code on dev, 3 of the 5 fail (the two first-assignment tests 
pass, so that path is unchanged). I also changed the fix on purpose: without 
`assigned = false` 3 tests fail (1 assignment instead of 2); without the empty 
check 2 fail with `IndexOutOfBoundsException: Index: 0`, as in the CI log of 
#13253; with an empty list clearing the pending split, 
`testEmptyListGivenBackKeepsAPendingSplit` fails.
   - `connector-common`: 21 tests pass; `spotless:check` and `verify` clean. No 
local Flink run; the CI Flink jobs run the connectors that use this class.
   
   ### Check list
   
   * [x] If any new Jar binary package adding in your PR, please add License 
Notice according
     [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
 (no new jar)
   * [x] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs (no user-facing 
option)
   * [x] If necessary, please update `incompatible-changes.md` to describe the 
incompatibility caused by this PR. (not incompatible)
   * [ ] If you are contributing the connector code, please check that the 
following files are updated: (not a new connector)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to