manuzhang commented on issue #24618: [SPARK-27734][CORE][SQL] Add memory based thresholds for shuffle spill URL: https://github.com/apache/spark/pull/24618#issuecomment-608206508 @amuraru I forgot `spark.memory.fraction` and `spark.memory.storageFaction`. How will they play together with the new configurations ? I've been testing Spark Adaptive Query Execution (AQE) recently, where contiguous shuffle partitions are coalesced to avoid too many small tasks. The problem is, IIUC, AQE makes coalesce decisions based on size of serialized map outputs. When data from multiple map tasks get deserialized into memory of one reduce task, it could easily blow up. I have to set extremely large `spark.executor.memory` to avoid being killed by YARN while wasting some resources. I think this patch is crucial for AQE to work steadily. @cloud-fan @maryannxue @JkSelf
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
