lisancao opened a new pull request, #59888: URL: https://github.com/apache/spark/pull/59888
### What changes were proposed in this pull request? This PR is stacked on #59613 (SPARK-52598, the Spark Connect docs reorg), which it was split out of. Until #59613 merges, the diff here includes its commits; the commit that belongs to this PR is the last one, "[SPARK-60611][DOCS][CONNECT] Add a Spark Connect programming guide". New page `spark-connect-programming-guide.md`, for people writing an application against a Connect server rather than operating one. Sections: - Spark Connect and Spark Classic: where each component runs - installing the client: the PySpark packages and what each contains, the Scala dependency and JVM flag, the Rust crate, other clients - starting a server: from PySpark, from a distribution, in a container - connecting from Python, Scala, Rust and Go; connection strings; sessions (`getOrCreate`, `create`, `newSession`, `stop`) - running an application: plain launch, `spark-submit` option mapping, `spark-submit --remote`, `spark.api.mode`, `spark-pipelines --remote` - configuration (what a client can and cannot set, and what happens when it tries), artifacts, Python UDFs and the server's Python, reading and writing data - migrating from Spark Classic, client and server versions, troubleshooting Also: - two gotchas in `spark-connect-gotchas.md`, in that page's format: session reuse across connection strings, and static configs set from the client - the guide's entry and section links in the Spark Connect sidebar - links to the guide from the overview and setup pages ### Why are the changes needed? No page walks through writing and running a client application end to end. What a reader needs is spread across the PySpark quickstart, the setup page and the client source, and several behaviours differ by client and by PySpark package. ### Does this PR introduce _any_ user-facing change? Docs only. ### How was this patch tested? Built the docs with `SKIP_API=1 SKIP_SCALADOC=1 SKIP_PYTHONDOC=1 SKIP_RDOC=1 SKIP_SQLDOC=1 SKIP_ERRORDOC=1 bundle exec jekyll build` and checked the sidebar entries and anchors in `_site/`. Ran the examples against a 4.3.0 RC2 Connect server: Python with the RC2 `pyspark` and `pyspark-client` packages on 3.11 and 3.12 (the `getOrCreate()` behaviour without a remote was checked for each of `pyspark`, `pyspark[connect]`, `pyspark-connect` and `pyspark-client`), Scala through `spark-shell --remote`, Rust with the `apache-spark-connect` 4.2.0 crate. The `spark-submit --remote` forms were run with the RC2 distribution for a Python file and a jar, and inside the `apache/spark:4.2.0` and `4.1.3` images. The client and server version rule was checked with the 4.3.0 client against 4.1.3 and 4.2.0 servers. The Scala builder behaviour for static keys (`spark.sql.catalogImplementation` dropped with a warning, `spark.sql.extensions` raising `CANNOT_MODIFY_STATIC_CONFIG`) was run through `spark-shell --remote`. One thing that came out of this: `spark-submit --remote` on its own does not work for Python files (SPARK-60365); the guide documents the forms that do. The review on #59613 before the split (from the spark-test reviewer) covered the guide and gotchas; those findings are addressed in the commits carried here. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code. The examples were run against real servers as described above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
