gkhnelbstn opened a new pull request, #12412:
URL: https://github.com/apache/seatunnel/pull/12412
### Purpose of this pull request
Closes #12411.
A streaming `Http` source — one with `poll_interval_millis` set — never
completes a checkpoint, and Zeta fails the job at the checkpoint timeout
while
the reader is demonstrably still polling.
`HttpSourceReader.pollNext` takes `output.getCheckpointLock()` around the
whole
poll, and the wait between polls is inside it, so the lock is held for the
whole interval:
```java
public void pollNext(Collector<SeaTunnelRow> output) throws Exception {
synchronized (output.getCheckpointLock()) {
internalPollNext(output); // finally {
Thread.sleep(pollIntervalMillis); }
}
}
```
That is the lock `SourceFlowLifeCycle#triggerBarrier` needs, as the comment
in
`SourceFlowLifeCycle` already says — and the engine adds a `Thread.sleep(0L)`
between polls precisely because the window between one `pollNext` releasing
the lock and the next taking it is narrow. A poll interval turns that into
one
narrow window every `poll_interval_millis`, and `synchronized` is not fair,
so
the barrier thread can lose the race indefinitely.
Raising `checkpoint.interval` or `checkpoint.timeout` does not help: a
contended lock is not a slow operation. The issue has the table of what was
tried.
`Object.wait(long)` releases the monitor while it waits and `Thread.sleep`
does not, so that is the whole change. The monitor is held at that point, so
it is legal; a spurious wake-up only means polling slightly early, which for
a
poll interval costs nothing; and if the method were ever called without the
lock it would throw `IllegalMonitorStateException` rather than fail quietly.
The two `Thread.sleep(10)` calls in the pagination loops are inside the same
lock. They are short enough not to matter and are left alone.
### Does this PR introduce _any_ user-facing change?
Yes, in the sense that a configuration which used to fail now works: a
streaming `Http` source with `poll_interval_millis` set can complete
checkpoints, so the job is no longer failed after a few minutes with
*"Checkpoint expired before completing"*. No option, default or schema
changes.
### How was this patch tested?
Against a running Zeta cluster built from 2.3.13 with this change applied,
polling an HTTP endpoint every ten seconds into a JDBC sink with
`checkpoint.interval = 20000`:
* **before** — the job reached `FAILED` at around two minutes, every time, at
every combination of `checkpoint.interval` and `checkpoint.timeout` tried
(the table is in #12411), while `TableSourceReceivedCount` kept growing by
four rows every 25 s;
* **after** — the same job ran continuously, checkpoints completing, rows
landing in the sink at each one. Watched for four and a half minutes at
45-second intervals, `RUNNING` throughout, and then used for an end-to-end
run of the pipeline it belongs to.
No unit test is added: the failure is a lock-contention race between the
reader thread and the checkpoint coordinator, and a test that reproduces it
reliably would have to drive both, which is an e2e shape rather than a unit
one. If a connector e2e case for a streaming `Http` source would be welcome I
am happy to add one — please say where it should live.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according [New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
— no new jars.
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs — no behaviour to
document; the option and its meaning are unchanged.
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR — none.
* [x] If you are contributing the connector code, please check that the
following files are updated — not a new connector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
