Francisco A. B. Sampaio created SPARK-59661:
-----------------------------------------------
Summary: Let downstream crates run actions asynchronously
Key: SPARK-59661
URL: https://issues.apache.org/jira/browse/SPARK-59661
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: connect-rust-0.1.0
Reporter: Francisco A. B. Sampaio
Hi, and thanks for the Rust client!
*Problem.* The public API of apache-spark-connect is synchronous. Actions go
through spark_connect_core::runtime::block_on, which panics when called from an
async context ("Cannot start a runtime from within a runtime"), even for
SparkSessionBuilder::get_or_create(). The workaround,
tokio::task::spawn_blocking, works, but it can't support cancellation: dropping
the future leaves the operation running on the server.
apache-spark-connect-core is already fully async
(SparkConnectClient::execute_plan_reattachable, analyze_plan, interrupt_*), so
async callers could drive actions directly. They're blocked only by visibility:
everything that turns a DataFrame into a request is pub(crate).
*Request.* The smallest change that would let downstream crates run actions
asynchronously without duplicating any plan-building logic:
*1.* DataFrame::build_execute_request(&self) ->
Result<proto::ExecutePlanRequest> → pub
*2.* DataFrame::build_analyze_request(&self) ->
Result<proto::AnalyzePlanRequest> → pub
*3.* SparkSession::client(&self) -> &Arc<SparkConnectClient> → pub
(1) and (2) rather than plan(), because the builders apply assign_plan_ids, the
session id and the tags. Downstream code shouldn't reimplement those.
Nice to have, for the same use case:
*4.* A public way to get the proto::Command behind side-effecting operations
(DataFrameWriter::save/save_as_table/insert_into, DataFrameWriterV2,
MergeIntoWriter), e.g. a to_command() next to each action. That would let
writes run through the async client as well.
*5.* decode_arrow_record_batches as a public helper, so downstream decoding
matches the crate's.
*Question.* Would you consider first-class async actions upstream, for example
async variants of the DataFrame actions behind a feature flag, since the core
client is already async? I'd be happy to contribute that instead if it fits the
project's direction. The visibility changes above would still be the minimal
step.
Happy to open a PR for items 1–3 if that's welcome.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
