andygrove commented on issue #2321: URL: https://github.com/apache/datafusion-ballista/issues/2321#issuecomment-5310844332
Follow-up: I re-ran Q10 with \`ballista.scheduler.max_partitions_per_task=1\` (the pre-#2315 default) — same cluster, same config otherwise. It still fails, but the failure mode moved: **Before (default post-#2315: unbounded packing):** \`\`\` Failed to allocate 111.2 MB for SortPreservingMergeExec[0] with 5.5 GB already allocated for this reservation - 19.4 MB remain available for the total memory pool: fair(pool_size: 5.6 GB) \`\`\` **With \`max_partitions_per_task=1\`:** \`\`\` Failed to allocate 135.6 MB for RangeShuffleReaderExec[stage=5,out=45] with 0.0 B already allocated for this reservation - 118.2 MB remain available for the total memory pool: fair(pool_size: 5.6 GB) \`\`\` So \`max_partitions_per_task=1\` did prevent the sort-merge from ballooning past 5.5 GB in a single reservation — that specific OOM is gone. But a different operator (\`RangeShuffleReaderExec\` at stage=5, out=45) now fails to secure its initial 135 MB because the executor's fair pool is already near-exhausted by other concurrent tasks. Both failure modes point at the same root cause: **the AQE plan produced by #2315 needs more per-task memory than the pre-PR plan for Q10 at SF1000**, and the executor's per-task pool of ~5.6 GB (44.8 GB / 8 concurrent tasks) isn't sufficient. Restoring the old \`max_partitions_per_task\` default alone doesn't recover the pre-PR plan's memory shape. I'll continue with plan-diffing against a pre-PR build to identify the specific shape change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
