danny0405 commented on issue #19774: URL: https://github.com/apache/hudi/issues/19774#issuecomment-5453997880
After further compatibility analysis, I think **patch 1 is the correct fix** and patch 2 should not be used.\n\n- Patch 1: https://github.com/apache/hudi/issues/19774#issuecomment-5449553257\n- Patch 2: https://github.com/apache/hudi/issues/19774#issuecomment-5451608691\n\nThe failing state is produced by NBCC when a new, log-only file group is reconstructed with out-of-order completion:\n\n\n\nIf the logs are added one at a time, creates an uncommitted slice first and the completed log is then assigned to that slice. The existing check hides the whole slice.\n\nPatch 1 fixes the root cause during file-group construction. It sorts and adds the logs as a batch and, when the file group has no existing slice, establishes the earliest completed log as the initial slice anchor before assigning the pending logs. The pending log then follows the existing rule of attaching to the latest slice, while the slice remains visible through the committed anchor.\n\nThis is also compatible wi th the legitimate sources of uncommitted base files:\n\n1. A base file left by a failed is not published as a valid record location, so later writers do not append logs to that slice. The base-only uncommitted slice should continue to be ignored.\n2. A base file produced by pending compaction belongs to an intentional slice barrier. The compaction instant makes the slice eligible, and the filesystem view already removes the uncommitted base file: pre-V8 uses , while V8+ uses .\n\nPatch 2 instead broadens so that any committed log can make a slice with an uncommitted base instant visible. That changes a general visibility invariant to handle an NBCC construction-order bug and can expose inconsistent pre-V8 slices containing an actual uncommitted base file that is not covered by pending-compaction filtering. Pre-V8 log-block filtering cannot remove such a base file.\n\nTherefore patch 1 is the more targeted fix: it corrects the reachable NBCC log-only case at slice construction time , preserves the existing slice-visibility invariant, and leaves the established pre-V8 and V8+ file filtering behavior unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
