Hyukjin Kwon created SPARK-61391:
------------------------------------
Summary: Convert the params of the native pyspark.ml.connect
classes as the type converters of PySpark do
Key: SPARK-61391
URL: https://issues.apache.org/jira/browse/SPARK-61391
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
A follow-up of SPARK-61358, SPARK-61334 and SPARK-61324, found in a post-merge
review. The native ML classes of the Python client take their params as fixed
Rust types, which differ from PySpark's Params type converters: an integral
float such as maxDepth=3.0 is rejected where PySpark converts it to 3, a bool
is accepted for an int param where PySpark raises TypeError, None is ignored or
rejected where PySpark stores it and sends a null literal
(RandomForestClassifier(bootstrap=None) fits with bootstrap true, PySpark's
with false), an int out of the 32-bit range raises OverflowError at
construction where PySpark lets the server validate it, and an empty list
(VectorAssembler inputCols, setThresholds) is sent where PySpark raises
CANNOT_INFER_ARRAY_ELEMENT_TYPE. The setters also fail with 'Already borrowed'
while another thread is in fit.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]