HyukjinKwon opened a new pull request, #57155:
URL: https://github.com/apache/spark/pull/57155
### What changes were proposed in this pull request?
This PR splits several large/slow PySpark test suites into smaller modules so
they run as separate test targets and parallelize better, reducing wall-clock
time. No test logic is changed; test methods are only relocated, following
the
existing convention already used in the repo (e.g. `test_split_apply_*`,
`test_stat_*`).
The following suites are split, each with a matching Spark Connect parity
module:
- `pyspark.sql.tests.pandas.test_pandas_cogrouped_map`: move the heavier,
self-contained tests (`test_with_window_function`, `test_with_local_data`,
`test_arrow_batch_slicing`, `test_cogroup_apply_in_pandas_with_logging`)
into
a new `test_pandas_cogrouped_map_misc` module.
- `pyspark.sql.tests.arrow.test_arrow_cogrouped_map`: extract shared helpers
into `CogroupedMapInArrowTestsFuncMixin` and move the heavier tests into a
new
`test_arrow_cogrouped_map_misc` module.
- `pyspark.pandas.tests.data_type_ops.test_num_ops`: move
`IntegralExtensionOpsTestsMixin` and `FractionalExtensionOpsTestsMixin`
into
their own modules.
- `pyspark.pandas.tests.groupby.test_stat`: move `test_median` into a new
`test_stat_median` module.
- `pyspark.pandas.tests.groupby.test_split_apply`: move the `mean` case into
a
new `test_split_apply_mean` module.
New modules are registered in `dev/sparktestsupport/modules.py`.
### Why are the changes needed?
These suites are slow in CI. Splitting them lets the test runner schedule the
pieces in parallel, lowering total wall-clock time without dropping coverage.
### Does this PR introduce _any_ user-facing change?
No. Test-only change.
### How was this patch tested?
Existing tests, only relocated across modules. Verified that the total number
of test methods is unchanged for each split suite, that all new and modified
modules import and collect, and that `flake8` passes.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
