Hyukjin Kwon created SPARK-61996:
------------------------------------
Summary: Make the Rust API's replace, repartitions,
approx_quantile and stream reader send and check as PySpark's, and add jdbc,
the stream xml and PySpark's shortcuts
Key: SPARK-61996
URL: https://issues.apache.org/jira/browse/SPARK-61996
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
>From a review of the Rust API (spark_connect) against PySpark 4.3.0's surface
>and requests:
- DataFrame::replace, NaFunctions::replace and plan::na_replace send a string
that reads as a number ("1", "nan", "inf") as a double, so they replace the
number in the numeric columns and leave the string columns as they are, or fail
with CAST_INVALID_INPUT, where PySpark's replace of strings sends strings.
replace_literals sends integers as they are, where PySpark sends them as
doubles, and has no MIXED_TYPE_REPLACEMENT check.
- repartition(0), coalesce(0) and repartition_by_id(0) fail on the server, and
repartition_by_expressions of a negative count sends no count, where PySpark
raises VALUE_NOT_POSITIVE; StatFunctions::approx_quantile sends probabilities
outside [0, 1] and a negative relative error, where PySpark raises
NOT_EXPECTED_TYPE and NEGATIVE_VALUE; DataStreamReader::load of an empty path
and the stream format methods send [""], where PySpark raises
VALUE_NOT_NON_EMPTY_STR.
- The core has no DataFrameWriter::jdbc or DataStreamReader::xml, no DataFrame
shortcuts of the stat functions (approx_quantile, corr, cov, crosstab,
freq_items, sample_by), no NaFunctions fill of a value or a map, and no
Column::outer or Column::transform.
- The Rust migration notes list stale differences of the 4.3 line (interval map
keys, the Value variants, the Pipeline stages, PipelineModel's fields, Raw and
MAX_PLAN_DEPTH, the seed), and known-differences.md lacks that the drop-in
raises RecursionError for a plan of 746 to about 992 levels, which PySpark
sends.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]