Gurpreet Singh Multani created SPARK-58132:
----------------------------------------------
Summary: Reduce memory footprint of grouped-agg UDF batch-slicing
tests
Key: SPARK-58132
URL: https://issues.apache.org/jira/browse/SPARK-58132
Project: Spark
Issue Type: Test
Components: PySpark
Affects Versions: 4.2.1
Reporter: Gurpreet Singh Multani
The test_arrow_batch_slicing tests in the Arrow and pandas grouped-aggregate
UDF test suites materialize a 10-million-row DataFrame and slice it into
batches:
- python/pyspark/sql/tests/arrow/test_arrow_udf_grouped_agg.py
- python/pyspark/sql/tests/pandas/test_pandas_udf_grouped_agg.py
Under constrained driver heap this is unnecessarily memory-hungry and can
flake. A proportional 10x shrink of the dataset and the maxRecordsPerBatch /
maxBytesPerBatch configs still produces multiple batches per group, so the
batch-slicing code path is exercised exactly as before while using far less
memory:
Test-only change; no user-facing impact.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
