Hyukjin Kwon created SPARK-61202:
------------------------------------
Summary: Send the relation of a DataFrame into the plans of
pyspark.ml without parsing it, so deep DataFrames fit and transform, in the
Python client as PySpark does
Key: SPARK-61202
URL: https://issues.apache.org/jira/browse/SPARK-61202
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
pyspark.ml fails on DataFrames whose plan nests past about 100 protobuf levels.
LogisticRegression().fit(df) over a DataFrame of about 50 chained withColumn
calls raises "DecodeError: Error parsing message: too many levels of nesting",
and so does transform of a OneVsRest model with 10 or more classes. The Python
plan of a DataFrame parses the relation with the Python protobuf decoder, and
DataFrame(plan) parses it again with the Rust decoder, both with a recursion
limit of 100. PySpark builds the nested message without parsing it, so it works
at these depths.
Repro: a DataFrame of 60 withColumn calls, then
LogisticRegression(maxIter=2).fit(df). The fix sends a placeholder of the
relation from Python, and the core replaces it with the relation of the
DataFrame when the plan reaches it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
