fhan688 commented on code in PR #19960:
URL: https://github.com/apache/hudi/pull/19960#discussion_r4023663166
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/common/AbstractStreamWriteFunction.java:
##########
@@ -301,7 +302,8 @@ public void handleOperatorEvent(OperatorEvent event) {
* @return The instant time
*/
protected String instantToWrite(boolean hasData) {
- return
Preconditions.checkNotNull(this.correspondent.requestInstantTime(this.checkpointId),
+ return Preconditions.checkNotNull(
Review Comment:
> can we just add the retry in write functions and avoid the complexities on
coordinator?
The retry alone doesn't fix the root cause. The timeout is the pekko **ask**
timeout on the coordinator's *response*, not on the worker — creation already
runs async today, so a client retry would still ride a 10s ask timeout (and log
an error) on every poll while a lock is held. For a clean retry the handler has
to return fast with a "not ready" signal — that's the `PENDING` status, so a
minimal poll protocol is unavoidable.
**The good news:** idempotency is already cheap — the single-thread executor
plus the authoritative `eventBuffers` map mean concurrent writers and retries
all converge on one instant, no extra machinery needed.
Beyond the fast-return, the only real additions are:
- **`epoch` fencing** — prevents a gen-N creation from publishing into
gen-N+1's buffer after a failover.
- **`FAILED` fast-fail** — surfaces a permanently failing creation
immediately instead of retrying until the budget is exhausted.
If you'd prefer to keep the coordinator minimal, I can drop the fencing /
`InstantOp` and keep just fast-return + `PENDING`, at the cost of reopening the
failover-during-creation window. Which would you prefer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
