zhuxiangyi opened a new issue, #67062: URL: https://github.com/apache/doris/issues/67062
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? On a backend configured with several spill disks, all spill files created within any two-second window land on the same disk while the other spill disks stay idle. Spill throughput is therefore capped at one disk's bandwidth, and that disk reaches its capacity limit long before the others. The disk is chosen deterministically. `_get_stores_for_spill()` sorts candidates by usage ascending: ```cpp // be/src/exec/spill/spill_file_manager.cpp std::ranges::sort(stores_with_usage, [](auto&& a, auto&& b) { return a.second < b.second; }); ``` and `create_spill_file()` always takes the front entry: ```cpp // Select the first available data dir (sorted by usage ascending) SpillDataDir* data_dir = data_dirs.front(); ``` Sorting by usage is the right intent, but the usage value it sorts on is stale. `SpillDataDir::_get_disk_usage()` is derived from `_available_bytes`: ```cpp double _get_disk_usage(int64_t incoming_data_size) const { return _disk_capacity_bytes == 0 ? 0 : (double)(_disk_capacity_bytes - _available_bytes + incoming_data_size) / (double)_disk_capacity_bytes; } ``` and `_available_bytes` is only refreshed by `SpillDataDir::update_capacity()`, which runs on the GC thread once per `config::spill_gc_interval_ms` (2s by default): ```cpp void SpillFileManager::_spill_gc_thread_callback() { while (!_stop_background_threads_latch.wait_for( std::chrono::milliseconds(config::spill_gc_interval_ms))) { gc(config::spill_gc_work_time_ms); for (auto& [path, dir] : _spill_store_map) { static_cast<void>(dir->update_capacity()); } } } ``` Note that `_spill_data_bytes`, which *is* updated on every write, does not feed into `_get_disk_usage()`. So nothing the manager hands out during those two seconds changes the ordering: every caller computes the same usage vector, sorts it the same way, and picks the same winner. This is easy to hit in practice — a single spilling query creates one spill file per partition per pipeline task, so hundreds of `create_spill_file()` calls can fall inside one refresh window. ### What You Expected? Spill files should be spread across the spill disks that are effectively equally empty, so that all configured spill disks contribute bandwidth and fill at a comparable rate. A disk that is genuinely emptier than the rest should still be preferred. ### How to Reproduce? Configure a backend with several spill disks of similar free space, then run a query that spills heavily: ```sql set enable_spill = true; set enable_force_spill = true; select l_orderkey, count(*) from lineitem group by l_orderkey; ``` Watch the per-disk `spill_disk_data_size` gauges (or the on-disk `spill/` directories). The bytes concentrate on one disk instead of being spread across them. ### Anything Else? StarRocks solves the same problem by picking a random start index and walking the directory list from there (`be/src/compute_env/spill/dir_manager.cpp`). Randomizing only among the disks that a stale snapshot cannot tell apart keeps Doris's existing "prefer the emptiest disk" behaviour intact. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
