jayzhan211 opened a new pull request, #25496:
URL: https://github.com/apache/datafusion/pull/25496
## Which issue does this PR close?
- Closes #25269.
## Rationale for this change
A hash join reserves every build-side batch as it arrives, then copies all
of them
into a single batch without reserving the copy, while the input batches stay
alive
until the build finishes. Near the memory limit the process therefore holds
about
twice the build side while the memory pool sees it once, so instead of a
`ResourcesExhausted` error the process can be killed by the OS. It also means
nothing that reacts to pool pressure can rely on what a hash join reports.
Measured on a 1.25 GB build side (TPC-H SF1 `lineitem`, 16 columns, as the
build
side): the pool was charged ~1.25 GB while peak RSS was ~2.46 GB.
## What changes are included in this PR?
- `concat_build_batches`: reserves the copy before `concat_batches`, drops
the input
batches, then trims the reservation to what the concatenated batch retains
(`get_record_batch_memory_size`), growing instead if the estimate was low.
`build_mem_used` follows every grow and shrink.
- Nothing extra is reserved when there is nothing to copy: concatenating a
single
batch is zero-copy, and for `Utf8View`/`BinaryView` columns only the views
are
copied while the data buffers stay shared with the inputs.
- `try_create_array_map` becomes `array_map_key_range`: it only decides
whether the
perfect hash join applies and reserves the `ArrayMap`. Both the hash map
and the
`ArrayMap` paths now hand the batches to the helper by value, so the
inputs are
gone before the visited bitmap and the null-aware scope maps are allocated.
- Join key arrays that do not share the buffers of the build batch (any
on-expression
that is not a plain column) are kept for the whole join and are now
reserved too.
This uses a new public `RecordBatchMemoryCounter::count_array`.
This does not reduce memory usage: the build still peaks at about twice the
build
side, it is now counted. A memory-limited query that only fit because the
copy was
not counted can now fail with `ResourcesExhausted`.
## What is the testing strategy for this PR?
- `join_build_concat_is_reserved` (CollectLeft and Partitioned, hash map and
`ArrayMap`): fails with `ResourcesExhausted` at map + 1.5x inputs, passes
at
map + 3x inputs. It fails on `main` in all four variants.
- `join_build_key_arrays_are_reserved` (hash map and `ArrayMap`): with a
single build
batch and an `a1 + 1` join key, the join fails at inputs + map + half the
key array
and passes with room for it; a plain column key passes at the lower limit.
It fails
when the reservation is removed.
- Unit tests of the helper: output equals `concat_batches` in both orders,
the copy
is reserved and the reservation then matches the retained batch, a single
batch
is not reserved twice, view data buffers are not reserved twice.
- Existing `single_partition_join_overallocation` /
`partitioned_join_overallocation`
are unchanged.
## Are there any user-facing changes?
One public addition: `RecordBatchMemoryCounter::count_array`. Hash joins
running close
to their memory limit may now report `ResourcesExhausted` where they
previously exceeded the limit unnoticed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
