Hyukjin Kwon created SPARK-61469:
------------------------------------

             Summary: Drive the future of a blocking call from the heap in 
Spark Connect Rust, so that a request does not overflow a small stack of the 
calling thread
                 Key: SPARK-61469
                 URL: https://issues.apache.org/jira/browse/SPARK-61469
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


runtime::block_on and block_on_abortable of Spark Connect Rust move the future 
they drive by value through several frames on the calling thread (block_on, 
drive, the runtime's block_on, the closure of block_in_place). The future of a 
request holds the state of its gRPC call and of its retries, tens of KiB (the 
analyze_plan future is about 45-75 KiB), so each of those frames keeps a copy 
of it on the calling thread unless the compiler inlines it. Which frames are 
inlined depends on the code of the calling crate, so an unrelated change can 
tip it: on macOS arm64 release builds, DataFrame::schema of a shallow plan used 
about 355 KiB of the stack before the request was cloned, and with other open 
changes (#495, #503, #515) about 526 KiB, so the deep_request_stack_tests of 
SPARK-61400, which run on a 512 KiB thread (the smallest stack a Python thread 
may have), overflow (SIGABRT/SIGBUS), even for spark.range(1).schema().

Fix: pin the future on the heap in block_on and block_on_abortable, so that 
those frames hold only its pointer. The calling thread then uses about 127 KiB 
up to the request clone in schema(), and schema() and collect() of shallow and 
746-level plans pass down to a 192 KiB stack. The cost is one heap allocation 
per blocking call.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to