zhaoyudi-creator commented on code in PR #19951:
URL: https://github.com/apache/hudi/pull/19951#discussion_r4017089916
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java:
##########
@@ -435,6 +439,15 @@ private CompletableFuture<CoordinationResponse>
handleInFlightInstantsRequest(Co
return
CompletableFuture.completedFuture(CoordinationResponseSerDe.wrap(coordinationResponse));
}
+ private CompletableFuture<CoordinationResponse>
handlePendingBucketFileIdsRequest(Correspondent.PendingBucketFileIdsRequest
request) {
Review Comment:
Thanks @cshuo. Your analysis is correct for the original version — with the
committed view loaded before the pending-fileId lookup, it does produce a
duplicate fileId.
The latest commit reverses the order in bootstrapIndexIfNeed: query pending
fileIds first, then load the committed view from a fresh getHoodieTable().
Since the lookup and the recommit run on the same single-threaded executor
(FIFO), and commit lands before the buffer reset in doCommit, both orderings
recover fileId-A: lookup before recommit → the buffer still holds it; lookup
after recommit → it returns empty, but "empty" already implies the commit
landed, so the committed view loaded afterwards is guaranteed to see it. The
"both miss" case can no longer be constructed.
That said, I don't want to just push my own direction — I'd like to weigh
the two approaches with you:
The reorder in this PR: lightweight, adds no coordinator state, but
correctness leans on two implicit assumptions (lookup and recommit on the same
executor; getHoodieTable() not caching a stale timeline) that a future refactor
could quietly break.
Your coordinator-lifetime partition → bucketId → fileId mapping: relies on
no timing reasoning and is more robust, at the cost of extra coordinator state
to maintain correctly.
Which do you think fits better here? Would love your read, based on the
latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
