[
https://issues.apache.org/jira/browse/SPARK-60611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60611:
-----------------------------------
Labels: pull-request-available (was: )
> Add a Spark Connect programming guide
> -------------------------------------
>
> Key: SPARK-60611
> URL: https://issues.apache.org/jira/browse/SPARK-60611
> Project: Spark
> Issue Type: Documentation
> Components: Connect, Documentation
> Affects Versions: 4.3.0
> Reporter: Lisa Cao
> Priority: Major
> Labels: pull-request-available
>
> The Spark Connect docs (overview, setup, server libraries, and the Eager vs
> Lazy page) explain what Spark Connect is and how to run a server, but no page
> covers writing and running an application against one. That information is
> spread across the PySpark quickstart, the setup page and the client source,
> and parts of it differ by client and by PySpark package.
> This issue adds {{docs/spark-connect-programming-guide.md}}, for people
> writing an application against a Connect server, with sections on:
> * where each component runs in Spark Connect and Spark Classic
> * installing the client: the PySpark packages ({{pyspark}},
> {{pyspark[connect]}}, {{pyspark-connect}}, {{pyspark-client}}) and what each
> contains, the Scala dependency and JVM flag, the Rust crate, other clients
> * starting a server and connecting to it from Python, Scala, Rust and Go;
> connection strings; sessions
> * running an application: {{spark-submit}} option mapping, {{spark-submit
> --remote}}, {{spark.api.mode}}, {{spark-pipelines --remote}}
> * configuration a client can and cannot set, artifacts, Python UDFs, reading
> and writing data
> * migrating from Spark Classic, client and server versions, troubleshooting
> It also adds two entries to {{spark-connect-gotchas.md}}: session reuse
> across connection strings, and static configuration set from the client.
> It builds on the docs reorganization in SPARK-52598.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
