Hyukjin Kwon created SPARK-61996:
------------------------------------

             Summary: Make the Rust API's replace, repartitions, 
approx_quantile and stream reader send and check as PySpark's, and add jdbc, 
the stream xml and PySpark's shortcuts
                 Key: SPARK-61996
                 URL: https://issues.apache.org/jira/browse/SPARK-61996
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


>From a review of the Rust API (spark_connect) against PySpark 4.3.0's surface 
>and requests:
- DataFrame::replace, NaFunctions::replace and plan::na_replace send a string 
that reads as a number ("1", "nan", "inf") as a double, so they replace the 
number in the numeric columns and leave the string columns as they are, or fail 
with CAST_INVALID_INPUT, where PySpark's replace of strings sends strings. 
replace_literals sends integers as they are, where PySpark sends them as 
doubles, and has no MIXED_TYPE_REPLACEMENT check.
- repartition(0), coalesce(0) and repartition_by_id(0) fail on the server, and 
repartition_by_expressions of a negative count sends no count, where PySpark 
raises VALUE_NOT_POSITIVE; StatFunctions::approx_quantile sends probabilities 
outside [0, 1] and a negative relative error, where PySpark raises 
NOT_EXPECTED_TYPE and NEGATIVE_VALUE; DataStreamReader::load of an empty path 
and the stream format methods send [""], where PySpark raises 
VALUE_NOT_NON_EMPTY_STR.
- The core has no DataFrameWriter::jdbc or DataStreamReader::xml, no DataFrame 
shortcuts of the stat functions (approx_quantile, corr, cov, crosstab, 
freq_items, sample_by), no NaFunctions fill of a value or a map, and no 
Column::outer or Column::transform.
- The Rust migration notes list stale differences of the 4.3 line (interval map 
keys, the Value variants, the Pipeline stages, PipelineModel's fields, Raw and 
MAX_PLAN_DEPTH, the seed), and known-differences.md lacks that the drop-in 
raises RecursionError for a plan of 746 to about 992 levels, which PySpark 
sends.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to