andygrove commented on issue #1996: URL: https://github.com/apache/datafusion-ballista/issues/1996#issuecomment-5049884443
This just bit PR #2064 in CI, so here is a concrete reproduction to go with the code analysis above. Run [29923718090](https://github.com/apache/datafusion-ballista/actions/runs/29923718090), leg `TPC-H SF10 (AQE on, multi-partition tasks)`. Query 21 failed about 2 seconds in with `No space left on device (os error 28)` while writing shuffle output. Queries 1 through 20 all passed and verified. The AQE-off leg passed, which is exactly the asymmetry this issue predicts. ``` Task 1 failed: DataFusion error: Arrow error: External error: Execution error: Execution("ArrowError(IoError(\"No space left on device (os error 28)\", Os { code: 28, kind: StorageFull, message: \"No space left on device\" }), Some(\"\"))") ``` The scheduler log shows 20 `Job finished successfully` events between 13:57:06 and 13:59:02. The executor log tail covers three of those completions (jobs `ypJm7Ol`, `qPMmk2t`, `V3bztez`, queries 18 through 20) and contains zero `Remove data for job` and zero `Remove intermediate data for job` lines. The only reclaim anywhere in the window is for the failed job, which goes through `clean_up_failed_job` and so calls the undelayed `clean_up_job_data`. What makes this unrecoverable rather than merely suboptimal is the timing. `finished_job_data_clean_up_interval_seconds` defaults to 300 (`ballista/scheduler/src/config.rs:302`), and the full 21 query suite finishes in 116 seconds. With the immediate reclaim no-oping under AQE, the delayed whole-job cleanup never fires either, so shuffle output grows monotonically from query 1 through query 21 and nothing is ever freed during the run. For scale, one q18 stage was spilling around 235 MB per event across 4 partitions repeatedly, and the executor reported 69.7 GB available at registration. Also worth flagging that #1981 is currently closed as completed, but #1982 only fixed the static planner path. The flake it describes is still reachable through this issue, and it is showing up as red X's on unrelated PRs again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
