HyukjinKwon commented on PR #57155: URL: https://github.com/apache/spark/pull/57155#issuecomment-4925139868
### num_ops rebalance — measured before/after (CI durations, classic / Spark Connect) The earlier revision left the extension modules heavier than the base (Connect: integral_ext 50s, fractional_ext 40s vs base 28s). The heavy work was `test_astype` (looped dtype conversions) plus the 6 comparison ops (`eq`/`ne`/`lt`/`le`/`gt`/`ge`, each using `ops_on_diff_frames`). These were split into dedicated `*_astype_cmp` modules: | Module | before | after | | --- | --- | --- | | num_ops (base) | 27s / 28s | 26s / 25s | | num_ops_integral_ext | 37s / 50s | 16s / 21s | | num_ops_integral_ext_astype_cmp | — | 34s / 47s | | num_ops_fractional_ext | 26s / 40s | 13s / 14s | | num_ops_fractional_ext_astype_cmp | — | 26s / 36s | Peak single-module time drops (Connect 50s→47s, classic 37s→34s) and the spread is flatter. The remaining ~47s in `integral_ext_astype_cmp` is dominated by `test_astype`'s dtype-conversion loop (a single test method), which is the practical floor without rewriting the test itself. All split/rebalanced modules pass. (The unrelated `test_import_spark` numpy failure is pre-existing on master, as noted above.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
