adriangb opened a new pull request, #27621:
URL: https://github.com/apache/datafusion/pull/27621
## Which issue does this PR close?
- No issue filed yet. This is the hash join counterpart of the "allocate,
then reserve" fix in the hash aggregate.
## Rationale for this change
A hash join whose build side does not fit the memory pool should fail before
it allocates the build. Today it reserves each build batch as it arrives, then
reserves the concatenated copy, the hash map and the visited bitmap one at a
time, each just before it allocates them. Under memory pressure the late
request fails part way through, usually at the hash map, after the copy already
exists. With several partitions sharing a `FairSpillPool`, that untracked work
is what pushes other operators over the limit.
## What changes are included in this PR?
```diff
collect_left_input
reserve each input batch
build_in_memory
+ try_grow(max(copy, map + visited bitmap)) // one request, before any
allocation
- concat_build_batches: try_grow(copy)
+ concat_build_batches: take copy from the request, keep map + bitmap
- new_join_hashmap: try_grow(map)
+ map_reservation = reservation.split(map) // no pool call
- allocate_bitmap: try_grow(bitmap)
+ allocate_bitmap: already reserved
```
The input batches are dropped before the map is allocated, so the request is
the larger of the copy and the map plus bitmap, not their sum. The perfect hash
join (array map) decision moves ahead of the concat, so the request uses the
array map size when that map will be built.
Review commit by commit:
1. `refactor`: splits `join_hashmap_size`, `concat_copy_size` and
`alloc_join_hashmap` out of the existing functions. No behavior change.
2. `fix`: the single up-front reservation, and its test.
These parts still reserve late, because they depend on the data: NULL keys
that rule out an array map, key expressions that compute new arrays, the hashes
scratch buffer, the `NOT IN` state, and a concatenated batch that retains more
than was reserved.
## What is the testing strategy for this PR?
`collect_left_input_reserves_build_once` records every `try_grow` that
reaches the pool:
```text
build 4 x 1000 rows, two key columns, LEFT join (visited bitmap)
before: [12000, 12000, 12000, 12000, copy, map, bitmap]
after: [12000, 12000, 12000, 12000, map + bitmap] // 155828
and pool.reserved() == retained batch + map + bitmap == build_mem_used
```
Existing tests (`join_build_concat_is_reserved`,
`join_build_key_arrays_are_reserved`, `concat_build_batches_*`) pass unchanged,
so the limits at which a build fits are the same.
Reproduction: a 4M row build side, an 8M row probe side, `target_partitions
= 1`, a `FairSpillPool`, and a counting global allocator.
| limit (MB) | before: outcome, tracked / heap peak (MB) | after |
|---|---|---|
| 70 to 160 | fails at the map, after the copy: 61.0 / 30.7 | fails before
the copy: 30.5 / 0.2 |
| 200, 400 | ok: 181.8 / 182.2 | ok: 181.8 / 182.2 |
A refused build now fails before it allocates. The success threshold and the
peak are unchanged.
## Are there any user-facing changes?
No. No public API changes. A build that does not fit fails at the same
limits, but without first making the concatenated copy.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]