Yicong-Huang opened a new pull request, #57047:
URL: https://github.com/apache/spark/pull/57047
### What changes were proposed in this pull request?
Add an ASV microbenchmark for the `SQL_TRANSFORM_WITH_STATE_PYTHON_ROW_UDF`
eval type (served by `TransformWithStateInPySparkRowSerializer`), mirroring the
existing `SQL_TRANSFORM_WITH_STATE_PANDAS_UDF` benchmark. It reuses the
existing `bench_eval_type.py` harness (`MockProtocolWriter`, `MockDataFactory`,
`MockUDFFactory`, `_StubStateServer`, `_TimeBenchBase`/`_PeakmemBenchBase`) and
adds `TransformWithStateRowUDF{Time,Peakmem}Bench` over 7 scenarios (few/many
groups small/large, wide columns, mixed types, nested struct) x 3 UDFs. Unlike
the Pandas variant, the input `Row`s carry every column including the grouping
key, and output `Row`s round-trip through `row.asDict(True)` +
`pa.RecordBatch.from_pylist` -- the per-row Python object path this eval type
is built around.
### Why are the changes needed?
Establishes a no-regression baseline for the upcoming refactor that moves
the Row transformWithState logic out of the serializer into `read_udfs()`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Benchmark-only change. Ran `asv run --bench 'TransformWithStateRowUDF' -a
repeat=3 --python=same` twice; results were stable across runs. One
representative run:
```text
[time_worker]
================ ============== ============= ============
-- udf
---------------- -----------------------------------------
scenario identity_udf rebuild_udf count_udf
================ ============== ============= ============
few_groups_sm 505+-1ms 573+-2ms 403+-0.6ms
few_groups_lg 4.93+-0.01s 5.70+-0.05s 3.94+-0.01s
many_groups_sm 2.33+-0.01s 2.64+-0.01s 1.83+-0.01s
many_groups_lg 2.08+-0s 2.39+-0.02s 1.64+-0.01s
wide_cols 5.48+-0.03s 6.08+-0.02s 4.52+-0.02s
mixed_cols 2.06+-0.01s 2.36+-0.01s 1.61+-0.02s
nested_struct 3.26+-0.01s 3.65+-0.02s 2.44+-0.01s
================ ============== ============= ============
[peakmem_worker]
================ ============== ============= ===========
scenario identity_udf rebuild_udf count_udf
================ ============== ============= ===========
few_groups_sm 89.4M 89.4M 86M
few_groups_lg 103M 103M 89.8M
many_groups_sm 92.2M 92.2M 86.8M
many_groups_lg 92.5M 92.5M 86.1M
wide_cols 104M 104M 94.2M
mixed_cols 96.1M 96.1M 90.4M
nested_struct 101M 101M 93.7M
================ ============== ============= ===========
```
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
