cshuo commented on code in PR #20323:
URL: https://github.com/apache/hudi/pull/20323#discussion_r4270402532
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/StreamWriteOperatorCoordinator.java:
##########
@@ -587,11 +592,17 @@ private void handleBootstrapEvent(WriteMetadataEvent
event) {
this.eventBuffers.cleanLegacyEvents(event);
return;
}
- EventBuffer eventBuffer =
this.eventBuffers.getOrCreateBootstrapBuffer(event);
- eventBuffer.addBootstrapEvent(event);
- if (eventBuffer.allBootstrapEventsReceived()) {
+ // The writer may have observed a pending instant before its normal commit
finished.
+ // Do not recreate the committed buffer when that bootstrap event arrives
later.
+ if (this.eventBuffers.getEventBuffer(event.getCheckpointId()) == null
+ &&
this.metaClient.reloadActiveTimeline().filterCompletedInstants().containsInstant(event.getInstantTime()))
{
+ return;
+ }
+ EventBuffer bootstrapBuffer =
this.eventBuffers.getOrCreateBootstrapBuffer(event);
+ bootstrapBuffer.addBootstrapEvent(event);
Review Comment:
During partial failover, a restarted writer can resend bootstrap metadata
when the coordinator already holds its complete write event for the same
checkpoint and instant. This replaces `lastBatch=true` with the restored
event's default `false`, excluding the batch from subsequent coordinator
snapshots. Unaffected writers retain ordinary events, so
`allBootstrapEventsReceived()` also remains false. Could we preserve the
existing completion state when applying replay, while still allowing recovery
to fill missing or incomplete entries?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
