[ 
https://issues.apache.org/jira/browse/SPARK-61520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon reassigned SPARK-61520:
------------------------------------

    Assignee: Hyukjin Kwon

> Unpickle deep Columns and DataFrames, and call session hooks on deep plans, 
> in the Python client as PySpark does
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61520
>                 URL: https://issues.apache.org/jira/browse/SPARK-61520
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.3.0
>
>
> A Column of more than about 50 nested expressions, or a DataFrame of more 
> than about 50 plan levels, pickles but does not unpickle: the client decodes 
> its spark.connect bytes with prost, and a Python worker of the server with 
> protobuf's FromString, and both stop at 100 levels of nesting ("failed to 
> decode Protobuf message" / DecodeError). With hook_factories, a plan of about 
> 50 levels fails the same way, as the request is decoded for the hooks and 
> decoded again after them. PySpark pickles its Python expression and plan 
> objects, and gives the hooks its Python request, so the same Columns and 
> DataFrames round-trip (e.g. 100 additions, 90 withColumn levels) and the 
> hooks run on deep plans, up to Python's own recursion limit.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to