HeartSaVioR commented on issue #27085: [SPARK-29779][CORE] Compact old event log files and cleanup - part 1 URL: https://github.com/apache/spark/pull/27085#issuecomment-570500300 Hmm... it seems hard to filter events accurately, assuming arbitrary out of order for events. If there's no idea to make it accurate, we may want to choose the safer approach which may filter out less events on out of order, as filtered out events will be lost once we compact and there's no way to restore them back. It may leave some unmatched events which might have been filtered out if they are in order, but (SQL) AppStatusListener have been tolerating events out of order so it seems acceptable.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
