HyukjinKwon commented on PR #57155: URL: https://github.com/apache/spark/pull/57155#issuecomment-4923606823
### CI timing results (run on this PR) All split modules pass. The one red job (`pyspark-sql, pyspark-core, ...`) failed only on `pyspark.tests.test_import_spark` (`Unexpected 3rd party package 'numpy' imported during 'import pyspark'`), which is unrelated to this test-only change and not part of this diff. Measured per-module durations (classic / Spark Connect): | Suite | base | new module(s) | | --- | --- | --- | | arrow cogrouped map | 22s | misc 23s | | pandas cogrouped map | 41s | misc 25s | | num_ops | 27s / 28s | integral_ext 37s / 50s, fractional_ext 26s / 40s | | groupby stat | 27s / 36s | stat_median 12s / 15s | | split_apply | 29s / 41s | split_apply_mean 29s / 41s | Each split lowers the longest single-module wall-clock via parallelism. `arrow` and `split_apply` are well balanced; `num_ops`, `groupby stat`, and `pandas cogrouped map` are functional splits but not tightly balanced (documented here for transparency; can be rebalanced in a follow-up if desired). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
