Hyukjin Kwon created SPARK-61520:
------------------------------------

             Summary: Unpickle deep Columns and DataFrames, and call session 
hooks on deep plans, in the Python client as PySpark does
                 Key: SPARK-61520
                 URL: https://issues.apache.org/jira/browse/SPARK-61520
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


A Column of more than about 50 nested expressions, or a DataFrame of more than 
about 50 plan levels, pickles but does not unpickle: the client decodes its 
spark.connect bytes with prost, and a Python worker of the server with 
protobuf's FromString, and both stop at 100 levels of nesting ("failed to 
decode Protobuf message" / DecodeError). With hook_factories, a plan of about 
50 levels fails the same way, as the request is decoded for the hooks and 
decoded again after them. PySpark pickles its Python expression and plan 
objects, and gives the hooks its Python request, so the same Columns and 
DataFrames round-trip (e.g. 100 additions, 90 withColumn levels) and the hooks 
run on deep plans, up to Python's own recursion limit.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to