rangareddy commented on issue #17241: URL: https://github.com/apache/hudi/issues/17241#issuecomment-5366113146
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-7101). **Findings: needs a reproduction against a current build before it can be dispositioned.** The report is concrete: the MDT initialization writes a dummy delete-block log file at base commit `0000000` and then a bulk_insert base file with the same commit time, so what should be two file slices reads as one and the stale log file is read alongside the latest base file. Reproducing it looks tractable - reading the MDT file groups after initialization and inspecting the file slice boundaries is enough; no large dataset needed. I have not called this from the source. The surrounding code has been rewritten substantially since this was filed - table versions 8 and 9, the `BufferedRecordMerger` family replacing payload-based merging, the file-group-reader migration, the indexer abstraction in #18348 / #18372, and the `hudi-common` package moves in #19193 / #19195. Inferring either "still broken" or "silently fixed" through that much churn would be a guess. There is also no commit anywhere in the tree referencing HUDI-7101, so if it was fixed it was incidental rather than deliberate - another reason to verify rather than assume. There is a mechanism on `master` that may already address this, which is why this needs checking rather than fixing blind: `HoodieBackedTableMetadataWriter.generateUniqueInstantTime` (around `:507-522`) loops `for (int offset = 0; ; ++offset)` over `HoodieInstantTimeGenerator.instantTimePlusMillis(SOLO_COMMIT_TIMESTAMP, offset)` until it finds an instant not already in the MDT commits timeline - so initialization instants are now distinct by construction. `SOLO_COMMIT_TIMESTAMP` is `"00000000000000"` (`HoodieTableMetadata.java:65`). What I could not establish is whether the dummy delete-block log file and the following bulk_insert base file actually receive different times through that path. That is the thing to verify first. Keeping this open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
