andygrove opened a new issue, #26446: URL: https://github.com/apache/datafusion/issues/26446
### Is your feature request related to a problem or challenge? `DiskManagerBuilder::build` in `DiskManagerMode::Directories` calls `create_local_dirs`, which creates a `datafusion-*` directory in every configured directory, and each one is removed recursively when the `DiskManager` is dropped. `DiskManagerMode::OsTmpDirectory` creates its directory on the first `create_tmp_file` instead (#1695). An engine that builds a `RuntimeEnv` per query or per task pays for this on every one, whether or not it spills. DataFusion Comet builds one per native plan in a Spark task, over the executor's local disks, and measured the directory syscalls at 4.7% of task samples for a small-task workload on macOS (apache/datafusion-comet#7060). On Linux (ext4 on NVMe, kernel 6.8), each configured directory costs 11 syscalls per `DiskManager` (`mkdir`, `unlinkat`, two `statx`, two `getdents64`, two `fcntl`, `openat`, `close` and `newfstatat`), about 35 µs. The operations also serialize on the filesystem: with one directory, 8 or 16 threads building `DiskManager`s at once top out near 30k per second. ### Describe the solution you'd like Create the `Directories` mode directories on the first `create_tmp_file`, as `OsTmpDirectory` does. Two visible differences: `temp_dir_paths()` returns nothing until the first spill, and an unusable directory fails the first spill rather than `build()`. ### Describe alternatives you've considered Comet now does this itself with a `TempFileFactory` (`DiskManagerMode::Custom`) that builds a `Directories` mode `DiskManager` on its first spill file (apache/datafusion-comet#7066). Every engine with a per-query `RuntimeEnv` would otherwise need the same wrapper. ### Additional context In Comet, with 8 local directories, a Spark query of 501 short tasks ran 12% faster in wall time, with 9% less CPU, once the directories were created lazily. With one directory, or with fewer and longer tasks, the difference was in the noise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
