[ 
https://issues.apache.org/jira/browse/SPARK-61417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61417.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 495
[https://github.com/apache/spark-connect-rust/pull/495]

> Fix review findings: Spark's ML column defaults, the multiclass metric, the 
> depth of a cross-session transform, empty observe, done-call errors over 
> batch errors, and unencodable option strings
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61417
>                 URL: https://issues.apache.org/jira/browse/SPARK-61417
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> Follow-ups of the s17 review of SPARK-61362/61363/61367/61385/61386, each 
> checked against PySpark 4.3.0 or Spark v4.3.0:
> - The input/output column getters of StandardScaler, MaxAbsScaler, 
> StringIndexer and VectorAssembler returned made-up defaults ("features", 
> "scaled_features", "maxAbs_scaled", "indexed", "assembled"), which 
> SPARK-61386 documented as Spark's defaults. Spark's HasInputCol has no 
> default and HasOutputCol defaults to <uid>__output, the column that the 
> server writes, so select(col(scaler.output_col())) failed with 
> UNRESOLVED_COLUMN. input_col now returns Option<&str> and output_col the set 
> value or <uid>__output. VectorAssembler::set_input_cols and 
> Pipeline::set_stages take impl IntoIterator<Item = impl Into<String>>, as 
> AGENTS.md asks of string lists.
> - pyspark.ml.connect's MulticlassClassificationEvaluator() evaluated the JVM 
> default f1, where PySpark's constructor sets metricName "accuracy", so a 
> no-argument evaluator gave another number. The evaluators' signatures now are 
> PySpark's, with their defaults.
> - The drop-in's transform of a DataFrame of another session put the input 
> relation inline, so its levels counted as one: levels added over it were not 
> bounded, and a deep input could overflow a small stack when the relation was 
> cloned. It now passes the DataFrame's placeholder, which the core resolves 
> counting its levels, as the plans of pyspark.ml do.
> - DataFrame::observe_with with no expressions failed at the action with the 
> server's INTERNAL_ERROR; PySpark's observe raises CANNOT_BE_EMPTY at the call.
> - When the decoding of a batch (or of the schema) failed, collect, 
> collect_record_batches, the local iterator and catalog operations returned 
> that error and made the done call of the progress handlers on drop, losing 
> its error; PySpark's Progress.__exit__ makes it in the with of the execution, 
> and its error replaces the decoding error.
> - Reader, writer, streaming and session option keys and values, and fill 
> values, converted a str that UTF-8 cannot encode (a lone surrogate) lossily 
> (U+FFFD), so that a write could go to another path, or raised a misleading 
> TypeError; PySpark's protobuf string fields raise ValueError.
> - Docs: the empty agg of the Rust API, PipelineModel's public fields, and the 
> year-month interval columns that create_dataframe takes.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to