[
https://issues.apache.org/jira/browse/SPARK-61996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61996.
----------------------------------
Fix Version/s: connect-rust-4.4.0
Resolution: Fixed
Issue resolved by pull request 846
[https://github.com/apache/spark-connect-rust/pull/846]
> Make the Rust API's replace, repartitions, approx_quantile and stream reader
> send and check as PySpark's, and add jdbc, the stream xml and PySpark's
> shortcuts
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61996
> URL: https://issues.apache.org/jira/browse/SPARK-61996
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> From a review of the Rust API (spark_connect) against PySpark 4.3.0's surface
> and requests:
> - DataFrame::replace, NaFunctions::replace and plan::na_replace send a string
> that reads as a number ("1", "nan", "inf") as a double, so they replace the
> number in the numeric columns and leave the string columns as they are, or
> fail with CAST_INVALID_INPUT, where PySpark's replace of strings sends
> strings. replace_literals sends integers as they are, where PySpark sends
> them as doubles, and has no MIXED_TYPE_REPLACEMENT check.
> - repartition(0), coalesce(0) and repartition_by_id(0) fail on the server,
> and repartition_by_expressions of a negative count sends no count, where
> PySpark raises VALUE_NOT_POSITIVE; StatFunctions::approx_quantile sends
> probabilities outside [0, 1] and a negative relative error, where PySpark
> raises NOT_EXPECTED_TYPE and NEGATIVE_VALUE; DataStreamReader::load of an
> empty path and the stream format methods send [""], where PySpark raises
> VALUE_NOT_NON_EMPTY_STR.
> - The core has no DataFrameWriter::jdbc or DataStreamReader::xml, no
> DataFrame shortcuts of the stat functions (approx_quantile, corr, cov,
> crosstab, freq_items, sample_by), no NaFunctions fill of a value or a map,
> and no Column::outer or Column::transform.
> - The Rust migration notes list stale differences of the 4.3 line (interval
> map keys, the Value variants, the Pipeline stages, PipelineModel's fields,
> Raw and MAX_PLAN_DEPTH, the seed), and known-differences.md lacks that the
> drop-in raises RecursionError for a plan of 746 to about 992 levels, which
> PySpark sends.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]