adriangb opened a new pull request, #27621:
URL: https://github.com/apache/datafusion/pull/27621

   ## Which issue does this PR close?
   
   - No issue filed yet. This is the hash join counterpart of the "allocate, 
then reserve" fix in the hash aggregate.
   
   ## Rationale for this change
   
   A hash join whose build side does not fit the memory pool should fail before 
it allocates the build. Today it reserves each build batch as it arrives, then 
reserves the concatenated copy, the hash map and the visited bitmap one at a 
time, each just before it allocates them. Under memory pressure the late 
request fails part way through, usually at the hash map, after the copy already 
exists. With several partitions sharing a `FairSpillPool`, that untracked work 
is what pushes other operators over the limit.
   
   ## What changes are included in this PR?
   
   ```diff
    collect_left_input
      reserve each input batch
      build_in_memory
   +    try_grow(max(copy, map + visited bitmap))   // one request, before any 
allocation
   -    concat_build_batches: try_grow(copy)
   +    concat_build_batches: take copy from the request, keep map + bitmap
   -    new_join_hashmap: try_grow(map)
   +    map_reservation = reservation.split(map)    // no pool call
   -    allocate_bitmap: try_grow(bitmap)
   +    allocate_bitmap: already reserved
   ```
   
   The input batches are dropped before the map is allocated, so the request is 
the larger of the copy and the map plus bitmap, not their sum. The perfect hash 
join (array map) decision moves ahead of the concat, so the request uses the 
array map size when that map will be built.
   
   Review commit by commit:
   
   1. `refactor`: splits `join_hashmap_size`, `concat_copy_size` and 
`alloc_join_hashmap` out of the existing functions. No behavior change.
   2. `fix`: the single up-front reservation, and its test.
   
   These parts still reserve late, because they depend on the data: NULL keys 
that rule out an array map, key expressions that compute new arrays, the hashes 
scratch buffer, the `NOT IN` state, and a concatenated batch that retains more 
than was reserved.
   
   ## What is the testing strategy for this PR?
   
   `collect_left_input_reserves_build_once` records every `try_grow` that 
reaches the pool:
   
   ```text
   build 4 x 1000 rows, two key columns, LEFT join (visited bitmap)
   before: [12000, 12000, 12000, 12000, copy, map, bitmap]
   after:  [12000, 12000, 12000, 12000, map + bitmap]   // 155828
   and pool.reserved() == retained batch + map + bitmap == build_mem_used
   ```
   
   Existing tests (`join_build_concat_is_reserved`, 
`join_build_key_arrays_are_reserved`, `concat_build_batches_*`) pass unchanged, 
so the limits at which a build fits are the same.
   
   Reproduction: a 4M row build side, an 8M row probe side, `target_partitions 
= 1`, a `FairSpillPool`, and a counting global allocator.
   
   | limit (MB) | before: outcome, tracked / heap peak (MB) | after |
   |---|---|---|
   | 70 to 160 | fails at the map, after the copy: 61.0 / 30.7 | fails before 
the copy: 30.5 / 0.2 |
   | 200, 400 | ok: 181.8 / 182.2 | ok: 181.8 / 182.2 |
   
   A refused build now fails before it allocates. The success threshold and the 
peak are unchanged.
   
   ## Are there any user-facing changes?
   
   No. No public API changes. A build that does not fit fails at the same 
limits, but without first making the concatenated copy.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to