deepakpanda93 commented on issue #19686: URL: https://github.com/apache/hudi/issues/19686#issuecomment-5409591486
cc @danny0405 @cshuo — flagging this one for your eyes since it sits squarely in the Flink + MOR compaction path. Quick summary: On table version < 8 (Timeline Layout V1), MOR + Avro log blocks written by Flink streaming with async instant generation can hit a deterministic data-loss race. The compaction planner freezes log file paths before an inflight deltacommit completes, so committed data can become invisible once compaction finishes. The race shows up in two ways with base instant A, completed deltacommit B, inflight deltacommit C, and planned compaction D (C < D): C hasn't written its file yet when D is planned → C later commits but is excluded from the compaction plan. C has written its file but is still inflight when D is planned → the block headers aren't readable yet, so C's data is stranded in the old slice. Repro details, versions (Hudi 1.1.1 / Flink 1.20), and the full analysis are in the issue above. Could you take a look and share whether this matches a known code path, and how you'd recommend approaching the fix ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
