K1taIkuyo0421 commented on issue #19922: URL: https://github.com/apache/hudi/issues/19922#issuecomment-5660792710
> > In this scale-up recovery scenario, the bootstrap events do not fill all slots in the eventBuffer array corresponding to the new parallelism. Consequently, recommit is not triggered, and the old -1 → instant mapping remains in the coordinator. > > We might need to send the empty bootstrap event for tasks that does not recover from any checkpoint state, this is a function regression since 1.1. @danny0405 Thanks for the reply. Since 1.1, when the coordinator receives an event created by emptyBootstrap(), it calls cleanLegacyEvents() and returns without counting the event toward bootstrap completion. This is still the behavior on the current master branch. Simply sending such events therefore seems insufficient. If we keep the existing bootstrap handling, the event would need instantTime and checkpointId values matching the pending recovery batch. Additional coordinator-side handling would also be needed to help tasks without restored state obtain those values. Alternatively, could we introduce a separate notification, such as BootstrapNoStateEvent, for tasks that do not restore any state? The coordinator could track these tasks and combine their notifications with the actual restored metadata to determine whether it can recommit. This would preserve the cleanup semantics of the existing empty bootstrap event. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
