zhuxiangyi opened a new pull request, #67063:
URL: https://github.com/apache/doris/pull/67063
### What problem does this PR solve?
Issue Number: close #67062
Problem Summary:
On a backend with several spill disks, every spill file created within a
two-second window lands on the same disk while the others stay idle. Spill
throughput is capped at one disk's bandwidth and that disk reaches its capacity
limit first.
The selection is deterministic *and* the input it sorts on is stale:
- `_get_stores_for_spill()` sorts candidates by usage ascending, and
`create_spill_file()` always takes `data_dirs.front()`.
- The usage comes from `SpillDataDir::_get_disk_usage()`, which reads
`_available_bytes`.
- `_available_bytes` is only refreshed by `SpillDataDir::update_capacity()`,
called from the GC thread once per `config::spill_gc_interval_ms` (2s by
default).
- `_spill_data_bytes`, which *is* updated on every write, does not feed into
`_get_disk_usage()`.
So nothing the manager hands out during those two seconds changes the
ordering: every caller computes the same usage vector, sorts it the same way,
and picks the same winner. A single spilling query creates one spill file per
partition per pipeline task, so hundreds of calls easily fall inside one
refresh window.
This PR shuffles the head of the sorted list. Disks whose usage is within
`USAGE_EQUIVALENCE_BAND` (2% of capacity) of the emptiest one are treated as
equally empty and drawn from at random; a disk that is genuinely emptier by
more than the band still wins outright, so the existing "prefer the emptiest
disk" intent is preserved. Only ties that a stale snapshot cannot distinguish
get spread out. The generator is `thread_local`, so concurrent callers draw
independent permutations without contending on a shared generator.
StarRocks addresses the same problem by picking a random start index and
walking the directory list from there
(`be/src/compute_env/spill/dir_manager.cpp`); randomizing only within the
near-equal band keeps Doris's usage-aware preference rather than replacing it
with plain round-robin.
### Release note
Fix spill files all landing on a single disk when multiple spill disks are
configured.
### Check List (For Author)
- Test
- [x] Unit Test
Two tests in `spill_file_test.cpp`:
- `EquallyEmptyDisksAreSelectedInVaryingOrder` — with four equally
empty disks, 200
draws must reach all four. The previous deterministic policy yields
exactly one
distinct winner, so this fails before the change. With a uniform
draw the odds of
missing a disk are ~1e-25, so it cannot flake.
- `ClearlyEmptiestDiskAlwaysWins` — with one disk far emptier than the
rest (well
beyond the band), it must win all 50 draws. This guards the
preference that the
shuffle must not destroy.
- Behavior changed:
- [x] Yes.
Which spill disk a given spill file lands on is no longer
deterministic when several
disks are equally empty. Disk preference is unchanged whenever one
disk is more than
2% of capacity emptier than the others. No configuration, format or
interface change.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
