FreakingMind commented on issue #25804:
URL: https://github.com/apache/datafusion/issues/25804#issuecomment-6670830749
We ran into this on real jobs on Sail (0.7.2, with a limited pool), and I
managed to get it down to plain datafusion-cli with no external data:
```sql
SET datafusion.execution.target_partitions = 16;
SET datafusion.sql_parser.map_string_types_to_utf8view = false;
COPY (
SELECT (value * 2654435761) % 1000003 AS k,
(value * 40503) % 100000007 AS v,
concat('payload-', CAST(value AS VARCHAR),
'-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx') AS s
FROM generate_series(0, 99999999)
ORDER BY v, s
) TO '/tmp/out/' STORED AS PARQUET;
```
```
datafusion-cli -m 8g --mem-pool-type greedy -f repro.sql
```
This fails in about a second on both 55.2.0 and main (b6b2e0e) with `Failed
to allocate additional 696.1 KB for ExternalSorterMerge[N]`. Same query with
`--mem-pool-type fair` finishes and writes all 100M rows.
I also tried a window version, `sum(v) OVER (PARTITION BY k ORDER BY v)`
instead of the ORDER BY. Runs that passed, 16 partitions, 12-core Mac:
| | 55.2.0 | main |
|---|---|---|
| window, greedy 8G | 0/6 | 6/6 |
| window, greedy 12G | 0/6 | 5/6 |
| window, greedy 16G | 6/6 | 6/6 |
| sort, greedy 8G | 0/4 | 0/4 |
| sort, fair 8G | 4/4 | 4/4 |
So main is a lot better for the window case, but the plain sort still fails
and 12G is still flaky. On 55.2, greedy 24G doesn't spill at all
(`spill_count=0`), while fair 8G spills 96 times and is fine, so it does look
tied to the spill path.
With default Utf8View strings it fails earlier: the window query still fails
at greedy 24G on both versions, only 2 runs each though. On 55.2, one partition
passed at greedy 8G (2/2) and two partitions was 1/2, which would fit
partitions competing for the pool, but I haven't confirmed that's the cause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
