[
https://issues.apache.org/jira/browse/SPARK-61469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61469:
------------------------------------
Assignee: Hyukjin Kwon
> Drive the future of a blocking call from the heap in Spark Connect Rust, so
> that a request does not overflow a small stack of the calling thread
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61469
> URL: https://issues.apache.org/jira/browse/SPARK-61469
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> runtime::block_on and block_on_abortable of Spark Connect Rust move the
> future they drive by value through several frames on the calling thread
> (block_on, drive, the runtime's block_on, the closure of block_in_place). The
> future of a request holds the state of its gRPC call and of its retries, tens
> of KiB (the analyze_plan future is about 45-75 KiB), so each of those frames
> keeps a copy of it on the calling thread unless the compiler inlines it.
> Which frames are inlined depends on the code of the calling crate, so an
> unrelated change can tip it: on macOS arm64 release builds, DataFrame::schema
> of a shallow plan used about 355 KiB of the stack before the request was
> cloned, and with other open changes (#495, #503, #515) about 526 KiB, so the
> deep_request_stack_tests of SPARK-61400, which run on a 512 KiB thread (the
> smallest stack a Python thread may have), overflow (SIGABRT/SIGBUS), even for
> spark.range(1).schema().
> Fix: pin the future on the heap in block_on and block_on_abortable, so that
> those frames hold only its pointer. The calling thread then uses about 127
> KiB up to the request clone in schema(), and schema() and collect() of
> shallow and 746-level plans pass down to a 192 KiB stack. The cost is one
> heap allocation per blocking call.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]