Hyukjin Kwon created SPARK-61876:
------------------------------------

             Summary: Make the catalog and ML golden tests compare with goldens 
of PySpark 4.3.0, and send PySpark's requests from the Rust API's 
create_external_table, variant_set and tuple_union functions
                 Key: SPARK-61876
                 URL: https://issues.apache.org/jira/browse/SPARK-61876
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


As readwriter_golden.rs did before SPARK-61853, three golden tests claim parity 
with PySpark but read no golden: catalog_golden.rs refers to 
tests/golden/catalog.jsonl, ml_golden.rs to ml.jsonl and ml_extra_golden.rs to 
ml_extra.jsonl, none of which exists, so they pass without comparing anything. 
Their capture scripts, and capture_golden.py and capture_window_golden.py, read 
FieldDescriptor.label, which protobuf 7 removed, and capture_catalog_golden.py 
captured no case. functions_golden.rs has no case for 7 of its goldens.
With real goldens, the Rust API differs from PySpark 4.3.0 in:
- Catalog::create_external_table sends a CreateExternalTable, where PySpark's 
createExternalTable calls createTable, so it sends a CreateTable whose 
description option is the table's description;
- variant_set and try_variant_set of three arguments send no create_if_missing, 
where PySpark sends lit(True);
- tuple_union_agg_double, tuple_union_agg_integer, tuple_union_double, 
tuple_union_integer, tuple_union_theta_double and tuple_union_theta_integer 
send no lgNomEntries and mode, where PySpark sends lit(12) and lit("sum").




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to