Lisa Cao created SPARK-60611:
--------------------------------
Summary: Add a Spark Connect programming guide
Key: SPARK-60611
URL: https://issues.apache.org/jira/browse/SPARK-60611
Project: Spark
Issue Type: Documentation
Components: Documentation, Connect
Affects Versions: 4.3.0
Reporter: Lisa Cao
The Spark Connect docs (overview, setup, server libraries, and the Eager vs
Lazy page) explain what Spark Connect is and how to run a server, but no page
covers writing and running an application against one. That information is
spread across the PySpark quickstart, the setup page and the client source, and
parts of it differ by client and by PySpark package.
This issue adds {{docs/spark-connect-programming-guide.md}}, for people writing
an application against a Connect server, with sections on:
* where each component runs in Spark Connect and Spark Classic
* installing the client: the PySpark packages ({{pyspark}},
{{pyspark[connect]}}, {{pyspark-connect}}, {{pyspark-client}}) and what each
contains, the Scala dependency and JVM flag, the Rust crate, other clients
* starting a server and connecting to it from Python, Scala, Rust and Go;
connection strings; sessions
* running an application: {{spark-submit}} option mapping, {{spark-submit
--remote}}, {{spark.api.mode}}, {{spark-pipelines --remote}}
* configuration a client can and cannot set, artifacts, Python UDFs, reading
and writing data
* migrating from Spark Classic, client and server versions, troubleshooting
It also adds two entries to {{spark-connect-gotchas.md}}: session reuse across
connection strings, and static configuration set from the client.
It builds on the docs reorganization in SPARK-52598.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
