[ 
https://issues.apache.org/jira/browse/SPARK-62130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62130:
-----------------------------------
    Labels: pull-request-available  (was: )

> Decode the responses of the server within upb's limit of nested messages, of 
> PySpark's error, without a crash at any depth or on threads of small stacks
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62130
>                 URL: https://issues.apache.org/jira/browse/SPARK-62130
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: 5.0.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>
> The Rust client decodes the responses of the server with prost, whose 
> no-recursion-limit feature the workspace enables for the client's own deep 
> plans, so the responses decode at any depth:
> - A response nested very deeply, such as the schema of a struct nested 10^4 
> levels or a literal of 10^5, overflows the stack of the decode and the 
> process ends with SIGSEGV, in the Python client and the Rust API alike.
> - A response of more than 100 levels of nested messages decodes, where 
> PySpark 4.3.0, whose grpc deserializer is upb's FromString, raises the 
> SparkConnectGrpcException of INTERNAL "Exception deserializing response!": a 
> struct column nested 32 levels deep, say.
> - The conversions of deep responses and types recurse on the calling thread's 
> stack, so calls on threads of small stacks (threading.stack_size of 256 KiB) 
> crash, even a trivial spark.range(1).schema.
> Expected: as pyspark-client, the responses decode within upb's limit of 100 
> nested messages (65535 once upb's SetAllowOversizeProtos(True) is called), of 
> PySpark's error in the drop-in and of prost's error in the Rust API, without 
> a crash at any depth, and calls on threads of small stacks work as PySpark's.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to