gurpreetmultanii opened a new pull request, #57263:
URL: https://github.com/apache/spark/pull/57263
### What changes were proposed in this pull request?
This PR shrinks the input size and batch-size configs used by the
`test_arrow_batch_slicing` test in the Arrow and pandas grouped-aggregate
UDF test suites:
- `python/pyspark/sql/tests/arrow/test_arrow_udf_grouped_agg.py`
- `python/pyspark/sql/tests/pandas/test_pandas_udf_grouped_agg.py`
Specifically, in both files the identical change is applied (a proportional
10x shrink that preserves the multi-slice behavior the test exercises):
- `range(10000000)` -> `range(1000000)`
- `assert len(v) == 10000000 / 2` -> `assert len(v) == 1000000 / 2`
- batch-size configs `[(1000, 2**31 - 1), (0, 1048576), (1000, 1048576)]`
-> `[(100, 2**31 - 1), (0, 104858), (100, 104858)]`
### Why are the changes needed?
The tests materialize a 10-million-row DataFrame and slice it into batches.
Under constrained driver heap this is unnecessarily memory-hungry and can
flake. A 10x smaller dataset with proportionally smaller
`maxRecordsPerBatch` / `maxBytesPerBatch` still produces multiple batches
per group, so the batch-slicing path is exercised exactly as before while
using far less memory.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Existing tests (`test_arrow_batch_slicing` in both suites) continue to pass
with the reduced sizes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
