wForget commented on issue #11542: URL: https://github.com/apache/incubator-gluten/issues/11542#issuecomment-3834928453
I tried using sort shuffle and then got an OOM error. enable sort shuffle conf: ``` spark.sql.adaptive.coalescePartitions.initialPartitionNum=2000; spark.gluten.sql.columnar.shuffle.sort.partitions.threshold=1000; ``` error log: ``` Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2020 in stage 13.0 failed 4 times, most recent failure: Lost task 2020.3 in stage 13.0 (TID 4905) (xxx executor 16): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding physical memory limits. 4.0 GB of 4 GB physical memory used. Consider boosting spark.executor.memoryOverhead. ``` I will try the relevant PRs: + https://github.com/apache/incubator-gluten/pull/11059 + https://github.com/apache/incubator-gluten/pull/11123 + https://github.com/apache/incubator-gluten/pull/11101 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
