tgravescs commented on pull request #28412: URL: https://github.com/apache/spark/pull/28412#issuecomment-656148922
yeah 10x definitely seems safe as most of the number are more at the 8x number for zstd. I'm fine with leaving the current logic for the small files, we can always follow up with more enhancements to skip them later if we see that its causing a lot of load. @HeartSaVioR I'm not sure if that is what you were agreeing with or your suggest was to change it here to skip the small ones? I think you were ok either way but wanted to clarify. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
