mzabaluev opened a new issue, #26354:
URL: https://github.com/apache/datafusion/issues/26354
### Describe the bug
`memory_limit::memory_limit_validation::smj_mem_validation::smj_with_mem_limit_1`,
added in #23988 and first released in 55.1.0, fails intermittently on 32-vCPU
Linux hosts. In the failures the measured RSS growth is about 200 MB, well
above the 160 MB allowance (`40_000_000 * 4`). Across repeated runs the result
is bimodal: it either fits under the allowance or lands near 200 MB.
The test leaves little headroom even where it passes. On an Apple Silicon
Mac it measures 149.4–149.8 MB against the 152.6 MiB allowance in five out of
five runs, a margin of about 2%.
### To Reproduce
On a 32-vCPU x86_64 Linux host (we see it on Amazon Linux 2023, EC2
c7a.8xlarge):
```sh
git checkout 55.1.0
cargo test --profile ci -p datafusion --features extended_tests --test
core_integration -- \
memory_limit::memory_limit_validation::smj_mem_validation::smj_with_mem_limit_1_runner
```
The failure is intermittent, so repeat the run several times.
Example failure output:
```
Memory before: 140.9 MB, Memory after: 340.9 MB
Query: SELECT sum(rr.v) FROM generate_series(0, 0) AS l(k) JOIN (SELECT i %
1 AS k, i AS v FROM generate_series(1, 5000000) AS r(i)) rr ON l.k = rr.k,
Memory usage: 200.0 MB, Memory limit: 152.6 MB
thread
'memory_limit::memory_limit_validation::smj_mem_validation::smj_with_mem_limit_1'
panicked at
datafusion/core/tests/memory_limit/memory_limit_validation/utils.rs:286:5:
Memory usage exceeded the theoretical limit. Actual: 200.0 MB, Expected
limit: 152.6 MB
```
Measured usage in three failing runs: 197.8 MB, 199.7 MB, 200.0 MB. Four
other runs of the same suite passed.
### Expected behavior
The test passes deterministically on supported CI hosts, or its allowance
leaves enough headroom for allocator variance between platforms.
### Additional context
- **Spilling is not the problem.** The assertion that `SortMergeJoinExec`
spilled passes in the failing runs, and `smj_with_mem_limit_2` and
`smj_no_mem_limit` pass.
- **The baseline is much higher on Linux.** The baseline query peaks at
about 141 MB of RSS on the Linux host, against about 62 MB on macOS. The jump
of roughly 50 MB between passing and failing runs suggests glibc malloc
retaining an extra arena heap, which is 64 MB on 64-bit. We have not verified
this, for example with `MALLOC_ARENA_MAX`.
- **Our measurement helper was modified.** We patch `measure_max_rss` in
`utils.rs` to refresh only the current PID instead of calling
`System::new_all()` inside the monitor task, and to seed the peak with a
reading taken before the query. In our environment the full refresh could take
long enough that the short baseline query finished before any sample, leaving
the baseline RSS at 0 and turning the "delta" into absolute RSS. This may
deserve a separate issue, and we can open a PR if that's useful.
Possible fixes:
- raise the allowance for `smj_with_mem_limit_1`, for example to `40_000_000
* 5`;
- base the allowance on the spill metrics rather than a fixed RSS factor;
- run the memory-validation subprocesses with `MALLOC_ARENA_MAX` set, if the
arena explanation holds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
