hvanhovell commented on issue #25962: [SPARK-29285][Shuffle] Temporary shuffle files should be able to handle disk failures URL: https://github.com/apache/spark/pull/25962#issuecomment-605607619 @yaooqinn this is causing a performance regression for short running queries that only write to a few shuffle files, because now every shuffle file is created eagerly. In this case I think the cure is worse than the disease, so I think we should revert this. cc @cloud-fan @tgravescs @squito
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
