[
https://issues.apache.org/jira/browse/SPARK-61695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61695:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Clone the expression of an unpickled Column on a stack that holds its levels,
> so that a deep one is sent from a small thread
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61695
> URL: https://issues.apache.org/jira/browse/SPARK-61695
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> An unpickled Column of the Python client keeps the `spark.connect.Expression`
> that it was pickled with (`ExpressionNode::Sent`), and serializing it clones
> that proto with prost's derived `Clone`, which recurses once per level of the
> expression, on the caller's stack. The clone of a `LogicalPlan::Raw` relation
> runs on a stack grown for its levels; the clone of a sent expression did not.
> A Column of 300 levels, unpickled and then used on a Python thread of 512 KiB
> (for example `spark.range(1).select(column).schema`), overflows the thread's
> stack in that clone and crashes the process with SIGBUS (seen on macOS
> x86_64, where
> `test_a_deep_column_unpickles_and_is_used_on_a_thread_of_512_kib` fails). The
> clone should run on a stack that holds its levels, as the clone of a raw
> relation does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]