Yicong-Huang opened a new pull request, #53002: URL: https://github.com/apache/spark/pull/53002
### What changes were proposed in this pull request? We use these tests to ensure that large datasets are processed exclusively through the new iterator API and that loading them entirely into memory would fail. However, the original tests were too slow and resource-intensive. This PR optimizes the test to achieve the same validation goal while using significantly fewer resources. ### Why are the changes needed? To save time Before: ``` 7d04fd12050e/python3.11__pyspark.sql.tests.connect.pandas.test_parity_pandas_grouped_map__6iwxrtx4.log) Finished test(python3.11): pyspark.sql.tests.connect.pandas.test_parity_pandas_grouped_map **(291s)** Starting test(python3.11): pyspark.sql.tests.pandas.test_pandas_grouped_map (temp output: /__w/spark/spark/python/target/379cea6f-7421-4f81-ae7b-1c22baddaa92/python3.11__pyspark.sql.tests.pandas.test_pandas_grouped_map__km0800fd.log) Finished test(python3.11): pyspark.sql.tests.pandas.test_pandas_grouped_map **(300s)** ``` After: ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI test. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
