EgorFokin commented on issue #19290: URL: https://github.com/apache/hudi/issues/19290#issuecomment-5024655625
We see this issue pretty often when the data volume is large. Every time it is the same pattern: log file with instant older than the base file which gets scheduled for both cleaning and compaction. This is concerning since it might indicate data loss (log file being cleaned). Right now we are relying on the KEEP_LATEST_BY_HOURS policy with hoodie.clean.hours.retained=24 to mitigate it, but it is not very reliable if compaction, for example, doesn't run for 24 hours for some reason. Can you suggest some other workarounds for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
