nsivabalan commented on issue #10803: URL: https://github.com/apache/hudi/issues/10803#issuecomment-1979270249
So to summarize the issue, for the file group of interest, if not for spark speculative execution, only one log file(file ending w/ 24780) should have been added. but w/ spark speculative execution, we found two log files(file ending w/ 24780 and file ending w/ 25536) were added. The compaction plan shows two log files. From the reader side, hudi checks for log block sequences, and since both sequences have just 1 log file, hudi chooses one of them and there is no determinism here. But I am curious how the 2nd log file(file ending w/ 25536) was fully formed w/o any corruptness. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
