HeartSaVioR commented on PR #37474: URL: https://github.com/apache/spark/pull/37474#issuecomment-1211452718
Duplicating my comment in JIRA ticket here: Very interesting one to see! (Disclaimer: Abortable was something I worked with Steve.) Have you gone through some benchmarks to figure out this works with small to big files? One thing I wonder is whether multipart upload performs well with tiny file. We have lots of tiny files in checkpoint and all files could be pretty tiny for stateless query. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
