Ruifeng Zheng created SPARK-61629:
-------------------------------------
Summary: Reject pandas DataFrames in native Rust-backed Python ML
methods
Key: SPARK-61629
URL: https://issues.apache.org/jira/browse/SPARK-61629
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Ruifeng Zheng
The native Rust-backed Python ML methods currently accept pandas DataFrames by
implicitly creating a Spark DataFrame with the active session, and model
transforms may collect results back to pandas. This follows the deprecated
pyspark.ml.connect input contract rather than standard pyspark.ml over Spark
Connect.
Require Spark DataFrames for native fit, transform and evaluate. Reject pandas
inputs consistently with TypeError before parameter validation, schema access
or RPCs, and direct callers to convert explicitly with spark.createDataFrame.
Remove the implicit pandas conversion and model-output conversion helpers.
Cover empty inputs, pandas subclasses, vector cells, all native
estimator/evaluator entry points, and explicit conversion against a live Spark
server. Keep vendored Python scaler behavior unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]