lisancao commented on code in PR #59613:
URL: https://github.com/apache/spark/pull/59613#discussion_r4415967165
##########
docs/spark-connect-overview.md:
##########
@@ -17,17 +17,18 @@ license: |
See the License for the specific language governing permissions and
limitations under the License.
---
-**Building client-side Spark applications**
-In Apache Spark 3.4, Spark Connect introduced a decoupled client-server
-architecture that allows remote connectivity to Spark clusters using the
-DataFrame API and unresolved logical plans as the protocol. The separation
-between client and server allows Spark and its open ecosystem to be
-leveraged from everywhere. It can be embedded in modern data applications,
-in IDEs, Notebooks and programming languages.
+Spark Connect is a decoupled client-server architecture for Apache Spark. A
thin client
+library speaks to a Spark cluster over gRPC using the DataFrame API and
unresolved logical
+plans as the protocol, so Spark and its open ecosystem can be used from
anywhere:
+embedded in modern data applications, in IDEs, in notebooks and in programming
languages.
+Spark Connect has been part of Apache Spark since the 3.4 release.
To get started, see [Quickstart: Spark
Connect](api/python/getting_started/quickstart_connect.html).
Review Comment:
Changed it to point at [Setting up Spark Connect](spark-connect-setup.html)
in feb75ffba01. The PySpark quickstart is still linked under "Where to go from
here", as the notebook walkthrough.
##########
docs/spark-connect-overview.md:
##########
@@ -70,387 +71,49 @@ that developers need to be aware of when using Spark
Connect:
2. By design, the Spark Connect protocol uses Sparks logical
plans as the abstraction to be able to declaratively describe the
operations to be executed
on the server. Consequently, the Spark Connect protocol does not support
all the
- execution APIs of Spark, most importantly RDDs.
+ execution APIs of Spark, most importantly RDDs. Alternatives for missing
RDD functionality
+ are provided as part of the higher-level DataFrame API.
3. Spark Connect provides a session-based client for its consumers. This means
that the
client does not have access to properties of the cluster that manipulate the
- environment for all connected clients. Most importantly, the client does
not have access
- to the static Spark configuration or the SparkContext.
+ environment for all connected clients. Most importantly, the client cannot
change the
+ static Spark configuration and has no access to the SparkContext.
+
+Apart from these differences the Spark Connect API is code-compatible with
classic Spark
+applications: the DataFrame code they contain does not change. What changes is
around that code:
+how the application is launched and configured, where its dependencies go, and
when a plan is
+analyzed. [Setting up Spark Connect](spark-connect-setup.html) covers starting
a server,
+connecting to it and switching an application between the two modes; the
+[Eager vs Lazy](spark-connect-gotchas.html) page covers the behavioral
differences, starting with
+the deferred analysis of query plans.
# Operational benefits of Spark Connect
With this new architecture, Spark Connect mitigates several multi-tenant
operational issues:
**Stability**: Applications that use too much memory will now only impact their
-own environment as they can run in their own processes. Users can define their
-own dependencies on the client and don't need to worry about potential
conflicts
-with the Spark driver.
+own environment as they can run in their own processes. Dependencies installed
on
+the client do not conflict with the Spark driver's, although anything a UDF
imports
+still has to be available on the server.
**Upgradability**: The Spark driver can now seamlessly be upgraded
independently
of applications, for example to benefit from performance improvements and
security fixes.
This means applications can be forward-compatible, as long as the server-side
RPC
definitions are designed to be backwards compatible.
+**Remote connectivity**: The decoupled architecture allows remote connectivity
to Spark beyond SQL
+and JDBC: any application can now interactively use Spark "as a service".
+
+**Simplicity**: The number of APIs exposed to the user is reduced from 3 to 2.
The Spark Connect
Review Comment:
Yes, it meant RDDs (RDD, DataFrame and SQL down to DataFrame and SQL).
Reworded in feb75ffba01 to say what the API is, the DataFrame API and SQL
without the RDD API, and linked the section on how client applications differ,
which already covers the missing RDD functionality. I kept it neutral rather
than selling the RDD gap as a benefit.
##########
docs/spark-connect-server-libs.md:
##########
@@ -137,16 +72,19 @@ A Spark Server Library consists of the following
components, illustrated in Fig.
4. The client package that exposes the Spark Server Library application logic
to the Spark Client
Application, alongside PySpark or the Scala Spark Client.
<p style="text-align: center;">
- <img src="img/extending-spark-connect-labelled.png" title="Figure 2:
Labelled Architecture" alt="Extending Spark
-Connect Diagram - Labelled Steps"/>
-</p>
+ <img
+ src="img/extending-spark-connect-labelled.png"
+ alt="Extending Spark Connect Diagram - Labelled Steps"
+ />
+ <figcaption style="text-align: center;">Figure 2: Labelled
Architecture</figcaption>
+</p>
-#### (1) Spark Connect Protocol Extension
+# Spark Connect Protocol Extension
Review Comment:
Done in feb75ffba01. I did the same on the overview while I was there, so
its top-level sections are `##` and the client-differences section is `###`
under "How Spark Connect works".
##########
docs/spark-connect-server-libs.md:
##########
@@ -122,12 +59,10 @@ exposed to a client, which differs from existing extension
points in Spark such
[SparkSession
extensions](api/java/org/apache/spark/sql/SparkSessionExtensions.html) or
[Spark Plugins](api/java/org/apache/spark/api/plugin/SparkPlugin.html).
-### Getting Started: Extending Spark with Spark Server Libraries
+* This will become a table of contents (this text will be scraped).
+{:toc}
-Spark Connect is available and supports PySpark and Scala
-applications. We will walk through how to run an Apache Spark server with Spark
-Connect and connect to it from a client application using the Spark Connect
client
-library.
+# Extending Spark with Spark Server Libraries
Review Comment:
Dropped it in feb75ffba01; the components list now follows the contents list
directly.
##########
docs/spark-connect-server-libs.md:
##########
@@ -137,16 +72,19 @@ A Spark Server Library consists of the following
components, illustrated in Fig.
4. The client package that exposes the Spark Server Library application logic
to the Spark Client
Application, alongside PySpark or the Scala Spark Client.
<p style="text-align: center;">
- <img src="img/extending-spark-connect-labelled.png" title="Figure 2:
Labelled Architecture" alt="Extending Spark
-Connect Diagram - Labelled Steps"/>
-</p>
+ <img
+ src="img/extending-spark-connect-labelled.png"
+ alt="Extending Spark Connect Diagram - Labelled Steps"
+ />
+ <figcaption style="text-align: center;">Figure 2: Labelled
Architecture</figcaption>
Review Comment:
Agreed, consolidated in feb75ffba01. The labelled diagram has everything the
first one shows plus the numbered components, so it replaces Figure 1 and the
prose refers to its boxes (_Plugin_, _App Logic_, _Proto API_). The unlabelled
image was not used anywhere else, so I removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
