HeartSaVioR commented on PR #59777: URL: https://github.com/apache/spark/pull/59777#issuecomment-6514370642
Hi @pratham76, first of all, thanks for the contribution! I think the issue is that the demand of this config is mostly happening with one (or a couple) specific object stores. For other object stores (or HDFS based setup), flipping this could end up with unnecessary operation, adding cost and latency. That said, I'm not sure I'm supportive to turn this on by default. I'd probably suggest to find the docs for cloud/object store setup if we have any, and update the doc instead. Worth noting that the default Apache Spark setup is for Hadoop-compatible systems, rather than cloud vendors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
