akupchinskiy commented on issue #26606:
URL: https://github.com/apache/datafusion/issues/26606#issuecomment-6916896967

   @osscm since you're working on the hash-join side of this in #27349, here is 
another case for it
   that the current heuristic (compact below 4 rows per buffer) doesn't catch: 
a single join with a
   small `batch_size`.
   
   When the build side is read in small batches, each batch keeps a reference 
to the Parquet page
   buffer it came from, so the concatenated build batch ends up with about one 
buffer entry per input
   batch, i.e. ~`batch_size` rows per entry. 100k build rows at `batch_size = 
37` give 2,704 entries
   pointing to only 7 distinct allocations. `take` then attaches the whole list 
to every output batch,
   and everything above the join walks it per batch.
   
   Self-contained repro for `datafusion-cli`:
   
   ```sql
   SET datafusion.execution.batch_size = 37;
   
   COPY (SELECT value AS k, concat('key-', lpad(CAST(value AS VARCHAR), 56, 
'0')) AS s
         FROM generate_series(1, 100000) AS t(value)) TO 'dim.parquet';
   COPY (SELECT value % 100000 + 1 AS k, (value * 7919) % 600007 AS v
         FROM generate_series(1, 600000) AS t(value)) TO 'fact.parquet';
   
   CREATE EXTERNAL TABLE dim STORED AS PARQUET LOCATION 'dim.parquet';
   CREATE EXTERNAL TABLE fact STORED AS PARQUET LOCATION 'fact.parquet';
   
   SELECT count(*), max(rn), max(s)
   FROM (SELECT d.s, row_number() OVER (ORDER BY f.v % 1000, f.v) AS rn
         FROM dim d JOIN fact f ON f.k = d.k);
   ```
   
   On `main` (d9752d6, `release-nonlto` build, 10 partitions): 0.11 s at the 
default `batch_size`,
   57 s at 37. With `-m 256m` it fails at 37: `Failed to allocate additional 
2.6 GB for
   ExternalSorter`, for a batch that holds a few KB. With #27349 applied on top 
of the same commit
   the numbers don't change (58.6 s, and the same error with `-m 256m`): ~37 
rows per entry is well
   above the 4-rows-per-buffer threshold.
   
   Since the entries here are mostly duplicates of a few allocations, 
collapsing repeated
   `(ptr, len)` entries and remapping the view buffer indices (like 
`dedup_buffer_entries` in the spill
   code) removes them without copying string bytes; gating it on the number of 
entries removed
   (e.g. more than `batch_size`) keeps the default path untouched. A quick 
local experiment along
   those lines took this query from 57 s to 9 s, and it completes under `-m 
256m`; the rest is
   `BoundedWindowAggExec`, which grows its own buffer list by concatenating 
each input batch onto its
   partition buffer.
   
   For context, we ran into this running a benchmark under different 
`batch_size` settings: on two
   real queries (joins on string-view keys over ~100 MB of Parquet) `batch_size 
= 37` took one from
   0.06 s to 5 s, and the other was killed after 300 s at 5.3 GB RSS while the 
memory pool reported
   almost nothing. Happy to run those again against your branch once it's back.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to