[ 
https://issues.apache.org/jira/browse/SPARK-61422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

L. C. Hsieh updated SPARK-61422:
--------------------------------
    Fix Version/s: connect-rust-4.3.0

> Make the generated function dispatch and wrappers of spark-connect-rust 
> reproducible from their generator, and check them in CI
> -------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61422
>                 URL: https://issues.apache.org/jira/browse/SPARK-61422
>             Project: Spark
>          Issue Type: Improvement
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: L. C. Hsieh
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> In spark-connect-rust, crates/pyspark-rs/src/dispatch_generated.rs and 
> python/pyspark/sql/functions_generated.py both start with "Generated by 
> scripts/gen_fn_dispatch.py - do not edit directly", but neither can be 
> trusted to be what the generator writes:
> * The Rust dispatch was edited by hand. The generator keeps 
> {{--write-dispatch}} off by default because regenerating would regress the 
> hand-tuned arms (round -> round_scale, instr -> instr_with_start, 
> count_distinct, ...), and it would in fact emit arms that do not compile.
> * The generator finds functions with a regex that misses every generic 
> function ({{pub fn array<C: Into<Column>>(...)}}) and lets a function that 
> does not return a Column swallow the next one (broadcast -> cume_dist).
> * The set of Python wrappers is read back from functions_generated.py itself, 
> so a wrapper deleted from that file is not regenerated.
> * Nothing in CI compares either file with the generator, so drift is 
> undetectable.
> This issue makes the generator reproduce both checked-in files byte for byte 
> (every arm that does not follow from a Rust signature is declared in a table 
> of the generator, and a function that fits no rule fails the generation 
> instead of being dropped), adds a {{--check}} mode that exits non-zero with a 
> diff, and adds a CI workflow and a unit test that run it. The generated files 
> themselves do not change.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to