[
https://issues.apache.org/jira/browse/SPARK-61629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61629:
-----------------------------------
Labels: pull-request-available (was: )
> Reject pandas DataFrames in native Rust-backed Python ML methods
> ----------------------------------------------------------------
>
> Key: SPARK-61629
> URL: https://issues.apache.org/jira/browse/SPARK-61629
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> The native Rust-backed Python ML methods currently accept pandas DataFrames
> by implicitly creating a Spark DataFrame with the active session, and model
> transforms may collect results back to pandas. This follows the deprecated
> pyspark.ml.connect input contract rather than standard pyspark.ml over Spark
> Connect.
> Require Spark DataFrames for native fit, transform and evaluate. Reject
> pandas inputs consistently with TypeError before parameter validation, schema
> access or RPCs, and direct callers to convert explicitly with
> spark.createDataFrame. Remove the implicit pandas conversion and model-output
> conversion helpers. Cover empty inputs, pandas subclasses, vector cells, all
> native estimator/evaluator entry points, and explicit conversion against a
> live Spark server. Keep vendored Python scaler behavior unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]