[
https://issues.apache.org/jira/browse/SPARK-61422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh resolved SPARK-61422.
---------------------------------
Resolution: Fixed
Issue resolved by pull request 498
[https://github.com/apache/spark-connect-rust/pull/498]
> Make the generated function dispatch and wrappers of spark-connect-rust
> reproducible from their generator, and check them in CI
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61422
> URL: https://issues.apache.org/jira/browse/SPARK-61422
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In spark-connect-rust, crates/pyspark-rs/src/dispatch_generated.rs and
> python/pyspark/sql/functions_generated.py both start with "Generated by
> scripts/gen_fn_dispatch.py - do not edit directly", but neither can be
> trusted to be what the generator writes:
> * The Rust dispatch was edited by hand. The generator keeps
> {{--write-dispatch}} off by default because regenerating would regress the
> hand-tuned arms (round -> round_scale, instr -> instr_with_start,
> count_distinct, ...), and it would in fact emit arms that do not compile.
> * The generator finds functions with a regex that misses every generic
> function ({{pub fn array<C: Into<Column>>(...)}}) and lets a function that
> does not return a Column swallow the next one (broadcast -> cume_dist).
> * The set of Python wrappers is read back from functions_generated.py itself,
> so a wrapper deleted from that file is not regenerated.
> * Nothing in CI compares either file with the generator, so drift is
> undetectable.
> This issue makes the generator reproduce both checked-in files byte for byte
> (every arm that does not follow from a Rust signature is declared in a table
> of the generator, and a function that fits no rule fails the generation
> instead of being dropped), adds a {{--check}} mode that exits non-zero with a
> diff, and adds a CI workflow and a unit test that run it. The generated files
> themselves do not change.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]