[ 
https://issues.apache.org/jira/browse/SPARK-61996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61996.
----------------------------------
    Fix Version/s: connect-rust-4.4.0
       Resolution: Fixed

Issue resolved by pull request 846
[https://github.com/apache/spark-connect-rust/pull/846]

> Make the Rust API's replace, repartitions, approx_quantile and stream reader 
> send and check as PySpark's, and add jdbc, the stream xml and PySpark's 
> shortcuts
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61996
>                 URL: https://issues.apache.org/jira/browse/SPARK-61996
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> From a review of the Rust API (spark_connect) against PySpark 4.3.0's surface 
> and requests:
> - DataFrame::replace, NaFunctions::replace and plan::na_replace send a string 
> that reads as a number ("1", "nan", "inf") as a double, so they replace the 
> number in the numeric columns and leave the string columns as they are, or 
> fail with CAST_INVALID_INPUT, where PySpark's replace of strings sends 
> strings. replace_literals sends integers as they are, where PySpark sends 
> them as doubles, and has no MIXED_TYPE_REPLACEMENT check.
> - repartition(0), coalesce(0) and repartition_by_id(0) fail on the server, 
> and repartition_by_expressions of a negative count sends no count, where 
> PySpark raises VALUE_NOT_POSITIVE; StatFunctions::approx_quantile sends 
> probabilities outside [0, 1] and a negative relative error, where PySpark 
> raises NOT_EXPECTED_TYPE and NEGATIVE_VALUE; DataStreamReader::load of an 
> empty path and the stream format methods send [""], where PySpark raises 
> VALUE_NOT_NON_EMPTY_STR.
> - The core has no DataFrameWriter::jdbc or DataStreamReader::xml, no 
> DataFrame shortcuts of the stat functions (approx_quantile, corr, cov, 
> crosstab, freq_items, sample_by), no NaFunctions fill of a value or a map, 
> and no Column::outer or Column::transform.
> - The Rust migration notes list stale differences of the 4.3 line (interval 
> map keys, the Value variants, the Pipeline stages, PipelineModel's fields, 
> Raw and MAX_PLAN_DEPTH, the seed), and known-differences.md lacks that the 
> drop-in raises RecursionError for a plan of 746 to about 992 levels, which 
> PySpark sends.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to