akupchinskiy commented on issue #26606:
URL: https://github.com/apache/datafusion/issues/26606#issuecomment-6916896967
@osscm since you're working on the hash-join side of this in #27349, here is
another case for it
that the current heuristic (compact below 4 rows per buffer) doesn't catch:
a single join with a
small `batch_size`.
When the build side is read in small batches, each batch keeps a reference
to the Parquet page
buffer it came from, so the concatenated build batch ends up with about one
buffer entry per input
batch, i.e. ~`batch_size` rows per entry. 100k build rows at `batch_size =
37` give 2,704 entries
pointing to only 7 distinct allocations. `take` then attaches the whole list
to every output batch,
and everything above the join walks it per batch.
Self-contained repro for `datafusion-cli`:
```sql
SET datafusion.execution.batch_size = 37;
COPY (SELECT value AS k, concat('key-', lpad(CAST(value AS VARCHAR), 56,
'0')) AS s
FROM generate_series(1, 100000) AS t(value)) TO 'dim.parquet';
COPY (SELECT value % 100000 + 1 AS k, (value * 7919) % 600007 AS v
FROM generate_series(1, 600000) AS t(value)) TO 'fact.parquet';
CREATE EXTERNAL TABLE dim STORED AS PARQUET LOCATION 'dim.parquet';
CREATE EXTERNAL TABLE fact STORED AS PARQUET LOCATION 'fact.parquet';
SELECT count(*), max(rn), max(s)
FROM (SELECT d.s, row_number() OVER (ORDER BY f.v % 1000, f.v) AS rn
FROM dim d JOIN fact f ON f.k = d.k);
```
On `main` (d9752d6, `release-nonlto` build, 10 partitions): 0.11 s at the
default `batch_size`,
57 s at 37. With `-m 256m` it fails at 37: `Failed to allocate additional
2.6 GB for
ExternalSorter`, for a batch that holds a few KB. With #27349 applied on top
of the same commit
the numbers don't change (58.6 s, and the same error with `-m 256m`): ~37
rows per entry is well
above the 4-rows-per-buffer threshold.
Since the entries here are mostly duplicates of a few allocations,
collapsing repeated
`(ptr, len)` entries and remapping the view buffer indices (like
`dedup_buffer_entries` in the spill
code) removes them without copying string bytes; gating it on the number of
entries removed
(e.g. more than `batch_size`) keeps the default path untouched. A quick
local experiment along
those lines took this query from 57 s to 9 s, and it completes under `-m
256m`; the rest is
`BoundedWindowAggExec`, which grows its own buffer list by concatenating
each input batch onto its
partition buffer.
For context, we ran into this running a benchmark under different
`batch_size` settings: on two
real queries (joins on string-view keys over ~100 MB of Parquet) `batch_size
= 37` took one from
0.06 s to 5 s, and the other was killed after 300 s at 5.3 GB RSS while the
memory pool reported
almost nothing. Happy to run those again against your branch once it's back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]