L. C. Hsieh created SPARK-61422:
-----------------------------------
Summary: Make the generated function dispatch and wrappers of
spark-connect-rust reproducible from their generator, and check them in CI
Key: SPARK-61422
URL: https://issues.apache.org/jira/browse/SPARK-61422
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: L. C. Hsieh
In spark-connect-rust, crates/pyspark-rs/src/dispatch_generated.rs and
python/pyspark/sql/functions_generated.py both start with "Generated by
scripts/gen_fn_dispatch.py - do not edit directly", but neither can be trusted
to be what the generator writes:
* The Rust dispatch was edited by hand. The generator keeps
{{--write-dispatch}} off by default because regenerating would regress the
hand-tuned arms (round -> round_scale, instr -> instr_with_start,
count_distinct, ...), and it would in fact emit arms that do not compile.
* The generator finds functions with a regex that misses every generic function
({{pub fn array<C: Into<Column>>(...)}}) and lets a function that does not
return a Column swallow the next one (broadcast -> cume_dist).
* The set of Python wrappers is read back from functions_generated.py itself,
so a wrapper deleted from that file is not regenerated.
* Nothing in CI compares either file with the generator, so drift is
undetectable.
This issue makes the generator reproduce both checked-in files byte for byte
(every arm that does not follow from a Rust signature is declared in a table of
the generator, and a function that fits no rule fails the generation instead of
being dropped), adds a {{--check}} mode that exits non-zero with a diff, and
adds a CI workflow and a unit test that run it. The generated files themselves
do not change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]