[
https://issues.apache.org/jira/browse/SPARK-61202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61202:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Send the relation of a DataFrame into the plans of pyspark.ml without parsing
> it, so deep DataFrames fit and transform, in the Python client as PySpark does
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61202
> URL: https://issues.apache.org/jira/browse/SPARK-61202
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> pyspark.ml fails on DataFrames whose plan nests past about 100 protobuf
> levels. LogisticRegression().fit(df) over a DataFrame of about 50 chained
> withColumn calls raises "DecodeError: Error parsing message: too many levels
> of nesting", and so does transform of a OneVsRest model with 10 or more
> classes. The Python plan of a DataFrame parses the relation with the Python
> protobuf decoder, and DataFrame(plan) parses it again with the Rust decoder,
> both with a recursion limit of 100. PySpark builds the nested message without
> parsing it, so it works at these depths.
> Repro: a DataFrame of 60 withColumn calls, then
> LogisticRegression(maxIter=2).fit(df). The fix sends a placeholder of the
> relation from Python, and the core replaces it with the relation of the
> DataFrame when the plan reaches it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
