andygrove commented on PR #7066: URL: https://github.com/apache/datafusion-comet/pull/7066#issuecomment-6234845319
I switched to the alternative you pointed to, done in Comet rather than waiting for DataFusion. DataFusion 55.2 already lets an engine supply its own spill backend (`DiskManagerMode::Custom` with a `TempFileFactory`), so each plan's `DiskManager` now builds its `Directories` mode `DiskManager` on the plan's first spill file and creates every spill file through it (5514cce2a). - A plan that never spills creates no directory. A plan that spills gets the same `datafusion-*` directories, the same per-plan `spark.comet.maxTempDirectorySize` limit and the same cleanup as on main. - The global map, the shared limit and the doc changes are gone, so there is no semantic change for users, now or later: if DataFusion makes `Directories` mode lazy (apache/datafusion#26446), Comet can drop the factory with nothing visible changing. - The `CometNativeSuite` test checks the new contract: a plan that does not spill creates no temp dir, a sort that spills creates one in a Spark local dir, and none remain once the plans are released. It fails against the eager behavior. Linux numbers, from a Ryzen 7950X3D on Ubuntu 22.04 (kernel 6.8, ext4 on NVMe): - Per plan, the eager directories cost about 35 µs per local dir (11 syscalls each): 147 µs with 4 dirs and 339 µs with 8. With the factory, building and dropping a plan's disk manager takes 0.1 µs. The work is kernel CPU, and it serializes on the filesystem: with one local dir, 8 or 16 threads building plans at once top out near 30k plans/s, and the median per plan rises from 35 µs to about 175 µs. While another writer flushed about 0.7 GB/s to the same filesystem, 8 threads with 4 local dirs saw p99 per plan go from 1.1 ms to 22 ms. - End to end, Spark 4.1.1 `local[8]`, `SELECT count(*) FROM lineitem WHERE l_quantity > 49`, 3 interleaved runs per arm: | Input | Tasks | Local dirs | Wall, main → PR | CPU, main → PR | |---|---|---|---|---| | SF10, default 128 MB splits | 20 | 1 or 8 | no change (±1%) | no change | | SF1 as 500 files of 0.5 MB | 501 | 1 | 177 → 176 ms | 1710 → 1675 ms | | SF1 as 500 files of 0.5 MB | 501 | 8 | 197 → 173 ms (−12%) | 1850 → 1680 ms (−9%) | With one local disk at a typical split size the gain is in the noise, as you expected. It shows up for many short tasks on executors with several local disks, which YARN and Kubernetes nodes often configure. On EBS-backed local dirs I'd expect the same CPU cost, since the loop above runs at 95% CPU in the kernel rather than waiting on the device, plus two costs that grow there: the waits behind other writers' journal commits get longer on a volume that flushes at gp3's 125 MB/s baseline, and the churn's metadata writes (about 150 small writes/s at 3k plans/s with 8 dirs here) come out of the volume's IOPS budget. I haven't measured on EBS. With no semantic change or process-wide state left, I think the factory is worth keeping for that case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
