This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new 1c7d78042f9 [SPARK-42731][CONNECT][DOCS] Document Spark Connect
configurations
1c7d78042f9 is described below
commit 1c7d78042f94b8177609d1f8c4ce2617432715c7
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Wed Mar 15 08:40:30 2023 +0900
[SPARK-42731][CONNECT][DOCS] Document Spark Connect configurations
### What changes were proposed in this pull request?
This PR proposes to document the configuration of Spark Connect defined in
https://github.com/apache/spark/blob/master/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
### Why are the changes needed?
To let users know which configuration are supported for Spark Connect.
### Does this PR introduce _any_ user-facing change?
Yes, it documents the configurations for Spark Connect.
### How was this patch tested?
Linters in CI should verify this change.
Also manually built the docs as below:
Closes #40416 from HyukjinKwon/SPARK-42731.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit e986fb0767eb64f1db6cf30f8c9f3e01c192171d)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
docs/configuration.md | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 63 insertions(+)
diff --git a/docs/configuration.md b/docs/configuration.md
index 932cd4d503c..0ca53acbeb6 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -3138,6 +3138,69 @@ like shuffle, just replace "rpc" with "shuffle" in the
property names except
The default value for number of thread-related config keys is the minimum of
the number of cores requested for
the driver or executor, or, in the absence of that value, the number of cores
available for the JVM (with a hardcoded upper limit of 8).
+### Spark Connect
+
+#### Server Configuration
+
+Server configurations are set in Spark Connect server, for example, when you
start the Spark Connect server with `./sbin/start-connect-server.sh`.
+They are typically set via the config file and command-lineoptions with
`--conf/-c`.
+
+<table class="table">
+<tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since
Version</th></tr>
+<tr>
+ <td><code>spark.connect.grpc.binding.port</code></td>
+ <td>
+ 15002
+ </td>
+ <td>Port for Spark Connect server to bind.</td>
+ <td>3.4.0</td>
+</tr>
+<tr>
+ <td><code>spark.connect.grpc.interceptor.classes</code></td>
+ <td>
+ (none)
+ </td>
+ <td>Comma separated list of class names that must implement the
<code>io.grpc.ServerInterceptor</code> interface</td>
+ <td>3.4.0</td>
+</tr>
+<tr>
+ <td><code>spark.connect.grpc.arrow.maxBatchSize</code></td>
+ <td>
+ 4m
+ </td>
+ <td>When using Apache Arrow, limit the maximum size of one arrow batch that
can be sent from server side to client side. Currently, we conservatively use
70% of it because the size is not accurate but estimated.</td>
+ <td>3.4.0</td>
+</tr>
+<tr>
+ <td><code>spark.connect.extensions.relation.classes</code></td>
+ <td>
+ (none)
+ </td>
+ <td>Comma separated list of classes that implement the trait
<code>org.apache.spark.sql.connect.plugin.RelationPlugin</code> to support
custom
+Relation types in proto.</td>
+ <td>3.4.0</td>
+</tr>
+<tr>
+ <td><code>spark.connect.extensions.expression.classes</code></td>
+ <td>
+ (none)
+ </td>
+ <td>Comma separated list of classes that implement the trait
+<code>org.apache.spark.sql.connect.plugin.ExpressionPlugin</code> to support
custom
+Expression types in proto.</td>
+ <td>3.4.0</td>
+</tr>
+<tr>
+ <td><code>spark.connect.extensions.command.classes</code></td>
+ <td>
+ (none)
+ </td>
+ <td>Comma separated list of classes that implement the trait
+<code>org.apache.spark.sql.connect.plugin.CommandPlugin</code> to support
custom
+Command types in proto.</td>
+ <td>3.4.0</td>
+</tr>
+</table>
### Security
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
