This is an automated email from the ASF dual-hosted git repository.
felixybw pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 789b81f4aa [VL] Add Spark configuration horhored status in Gluten
(#10442)
789b81f4aa is described below
commit 789b81f4aa14c4ee006d6572a21b247a7f4708ee
Author: BInwei Yang <[email protected]>
AuthorDate: Fri Aug 15 17:31:33 2025 -0700
[VL] Add Spark configuration horhored status in Gluten (#10442)
The PR started to draft the table of whether Spark configs are hornored by
Gluten or not. The columns are still left as blank. To be filled in future PRs.
---
README.md | 6 +
docs/velox-parquet-write-configuration.md | 228 +-
docs/velox-spark-configuration.md | 3541 +++++++++++++++++++++++++++++
3 files changed, 3730 insertions(+), 45 deletions(-)
diff --git a/README.md b/README.md
index bc5911a918..98667945f8 100644
--- a/README.md
+++ b/README.md
@@ -86,6 +86,12 @@ For **ClickHouse** backend, please refer to
[ClickHouse.md](./docs/get-started/C
Gluten jar will be generated under `/PATH/TO/GLUTEN/package/target/` after the
build.
+### Configurations
+Common configurations used by Gluten is listed in
[Configuration.md](./docs/Configuration.md). Velox specific configurations is
listed in [velox-configuration.md](./docs/velox-configuration.md)
+
+Some of the spark configurations are hornored by Gluten Velox backend, some of
them are ignored, and many are transparent to Gluten. The detail can be found
in [velox-spark-configuration.md](./docs/velox-spark-configuration.md) and
parquet write ones can be found in
[velox-parquet-write-configuration.md](./docs/velox-parquet-write-configuration.md)
+
+
# 4. Gluten Website
https://gluten.apache.org/
diff --git a/docs/velox-parquet-write-configuration.md
b/docs/velox-parquet-write-configuration.md
index 1e71f21344..cb82c5adc7 100644
--- a/docs/velox-parquet-write-configuration.md
+++ b/docs/velox-parquet-write-configuration.md
@@ -3,48 +3,186 @@ title: Parquet write configuration
nav_order: 17
## Parquet write configurations in Spark/Velox/Gluten
-| | parquet-mr default
| Spark default | Velox Default | Gluten Support |
-|---------------------------------------------------|
------------------------------------------ |---------------| -------------
|----------------|
-| -------------------Spark---------------- |
| | | |
-| spark.sql.parquet.outputTimestampType |
| int96 | | |
-| spark.sql.parquet.writeLegacyFormat |
| false | | |
-| -------------------Velox/Arrow---------------- |
| | | |
-| write_batch_size |
| | 1024 | Y (batch size) |
-| rowgroup_length |
| | 1M | |
-| compression_level |
| | 0 | |
-| page_index |
| | false | |
-| decimal_as_integer |
| | false | |
-| statistics_enabled |
| | false | |
-| -------------------parquet-mr---------------- |
| | | |
-| parquet.summary.metadata.level | all
| | | |
-| parquet.enable.summary-metadata | true
| | | |
-| parquet.block.size | 128m
| | | Y |
-| parquet.page.size | 1m
| | 1M | Y |
-| parquet.compression | uncompressed
| snappy | uncompressed | Y |
-| parquet.write.support.class |
org.apache.parquet.hadoop.api.WriteSupport | | |
|
-| parquet.enable.dictionary | true
| | true | Y |
-| parquet.dictionary.page.size | 1m
| | 1m | |
-| parquet.validation | false
| | | |
-| parquet.writer.version | PARQUET_1_0
| | PARQUET_2_6 | Y |
-| parquet.memory.pool.ratio | 0.95
| | | |
-| parquet.memory.min.chunk.size | 1m
| | | |
-| parquet.writer.max-padding | 8m
| | | |
-| parquet.page.size.row.check.min | 100
| | | |
-| parquet.page.size.row.check.max | 10000
| | | |
-| parquet.page.value.count.threshold | Integer.MAX_VALUE / 2
| | | |
-| parquet.page.size.check.estimate | true
| | | |
-| parquet.columnindex.truncate.length | 64
| | | |
-| parquet.statistics.truncate.length | 2147483647
| | | |
-| parquet.bloom.filter.enabled | false
| | | |
-| parquet.bloom.filter.adaptive.enabled | false
| | | |
-| parquet.bloom.filter.candidates.number | 5
| | | |
-| parquet.bloom.filter.expected.ndv |
| | | |
-| parquet.bloom.filter.fpp | 0.01
| | | |
-| parquet.bloom.filter.max.bytes | 1m
| | | |
-| parquet.decrypt.off-heap.buffer.enabled | false
| | | |
-| parquet.page.row.count.limit | 20000
| | | |
-| parquet.page.write-checksum.enabled | true
| | false | |
-| parquet.crypto.factory.class | None
| | | |
-| parquet.compression.codec.zstd.bufferPool.enabled | true
| | | |
-| parquet.compression.codec.zstd.level | 3
| | 0 | Y |
-| parquet.compression.codec.zstd.workers | 0
| | | |
+<table class="spark-config">
+<thead>
+ <tr>
+ <th></th>
+ <th>parquet-mr default</th>
+ <th>Spark default</th>
+ <th>Velox Default</th>
+ <th>Gluten Support</th>
+ </tr>
+</thead>
+<tbody>
+ <tr>
+ <td colspan="5" align="center">
+ <hr/>Spark <hr/></td>
+ </tr>
+ <tr>
+ <td><code>spark.sql.parquet.outputTimestampType</code></td>
+ <td></td><td>int96</td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>spark.sql.parquet.writeLegacyFormat</code></td>
+ <td></td><td>false</td><td></td><td></td>
+ </tr>
+ <tr>
+ <td colspan="5" align="center"> <hr/>Velox/Arrow <hr/></td>
+ </tr>
+ <tr>
+ <td><code>write_batch_size</code></td>
+ <td></td><td></td><td>1024</td><td>Y (batch size)</td>
+ </tr>
+ <tr>
+ <td><code>rowgroup_length</code></td>
+ <td></td><td></td><td>1M</td><td></td>
+ </tr>
+ <tr>
+ <td><code>compression_level</code></td>
+ <td></td><td></td><td>0</td><td></td>
+ </tr>
+ <tr>
+ <td><code>page_index</code></td>
+ <td></td><td></td><td>false</td><td></td>
+ </tr>
+ <tr>
+ <td><code>decimal_as_integer</code></td>
+ <td></td><td></td><td>false</td><td></td>
+ </tr>
+ <tr>
+ <td><code>statistics_enabled</code></td>
+ <td></td><td></td><td>false</td><td></td>
+ </tr>
+ <tr>
+ <td colspan="5" align="center"> <hr/>parquet-mr <hr/></td>
+ </tr>
+ <tr>
+ <td><code>parquet.summary.metadata.level</code></td>
+ <td>all</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.enable.summary-metadata</code></td>
+ <td>true</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.block.size</code></td>
+ <td>128m</td><td></td><td></td><td>Y</td>
+ </tr>
+ <tr>
+ <td><code>parquet.page.size</code></td>
+ <td>1m</td><td></td><td>1M</td><td>Y</td>
+ </tr>
+ <tr>
+ <td><code>parquet.compression</code></td>
+
<td>uncompressed</td><td>snappy</td><td>uncompressed</td><td>Y</td>
+ </tr>
+ <tr>
+ <td><code>parquet.write.support.class</code></td>
+
<td>org.apache.parquet.hadoop.api.WriteSupport</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.enable.dictionary</code></td>
+ <td>true</td><td></td><td>true</td><td>Y</td>
+ </tr>
+ <tr>
+ <td><code>parquet.dictionary.page.size</code></td>
+ <td>1m</td><td></td><td>1m</td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.validation</code></td>
+ <td>false</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.writer.version</code></td>
+ <td>PARQUET_1_0</td><td></td><td>PARQUET_2_6</td><td>Y</td>
+ </tr>
+ <tr>
+ <td><code>parquet.memory.pool.ratio</code></td>
+ <td>0.95</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.memory.min.chunk.size</code></td>
+ <td>1m</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.writer.max-padding</code></td>
+ <td>8m</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.page.size.row.check.min</code></td>
+ <td>100</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.page.size.row.check.max</code></td>
+ <td>10000</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.page.value.count.threshold</code></td>
+ <td>Integer.MAX_VALUE / 2</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.page.size.check.estimate</code></td>
+ <td>true</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.columnindex.truncate.length</code></td>
+ <td>64</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.statistics.truncate.length</code></td>
+ <td>2147483647</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.bloom.filter.enabled</code></td>
+ <td>false</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.bloom.filter.adaptive.enabled</code></td>
+ <td>false</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.bloom.filter.candidates.number</code></td>
+ <td>5</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.bloom.filter.expected.ndv</code></td>
+ <td></td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.bloom.filter.fpp</code></td>
+ <td>0.01</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.bloom.filter.max.bytes</code></td>
+ <td>1m</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.decrypt.off-heap.buffer.enabled</code></td>
+ <td>false</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.page.row.count.limit</code></td>
+ <td>20000</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.page.write-checksum.enabled</code></td>
+ <td>true</td><td></td><td>false</td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.crypto.factory.class</code></td>
+ <td>None</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+
<td><code>parquet.compression.codec.zstd.bufferPool.enabled</code></td>
+ <td>true</td><td></td><td></td><td></td>
+ </tr>
+ <tr>
+ <td><code>parquet.compression.codec.zstd.level</code></td>
+ <td>3</td><td></td><td>0</td><td>Y</td>
+ </tr>
+ <tr>
+ <td><code>parquet.compression.codec.zstd.workers</code></td>
+ <td>0</td><td></td><td></td><td></td>
+ </tr>
+</tbody>
+</table>
diff --git a/docs/velox-spark-configuration.md
b/docs/velox-spark-configuration.md
new file mode 100644
index 0000000000..0c1e72faf2
--- /dev/null
+++ b/docs/velox-spark-configuration.md
@@ -0,0 +1,3541 @@
+layout: page
+title: Spark configurations status in Gluten Velox Backend
+nav_order: 17
+
+The file lists the if Spark configurations are hornored by Gluten velox
backend or not. Table is from Spark4.0 configuration page. The status are:
+- H: hornored
+- P: Transparent to Gluten
+- I: ignored. Gluten doesn't use it.
+- `<blank>`: unknown yet
+
+
+# Application Properties
+<table>
+<tr>
+ <th>Property Name</th>
+ <th>Default</th>
+ <th>Since Version</th>
+ <th>Gluten Status</th>
+</tr>
+<tr>
+ <td><code>spark.app.name</code></td>
+ <td>(none)</td>
+ <td>0.9.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.cores</code></td>
+ <td>1</td>
+ <td>1.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.maxResultSize</code></td>
+ <td>1g</td>
+ <td>1.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.memory</code></td>
+ <td>1g</td>
+ <td>1.1.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.memoryOverhead</code></td>
+ <td>driverMemory * <code>spark.driver.memoryOverheadFactor</code>, with
minimum of <code>spark.driver.minMemoryOverhead</code></td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.minMemoryOverhead</code></td>
+ <td>384m</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.memoryOverheadFactor</code></td>
+ <td>0.10</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.resource.{resourceName}.amount</code></td>
+ <td>0</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.resource.{resourceName}.discoveryScript</code></td>
+ <td>None</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.resource.{resourceName}.vendor</code></td>
+ <td>None</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.resources.discoveryPlugin</code></td>
+ <td>org.apache.spark.resource.ResourceDiscoveryScriptPlugin</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.memory</code></td>
+ <td>1g</td>
+ <td>0.7.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.pyspark.memory</code></td>
+ <td>Not set</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.memoryOverhead</code></td>
+ <td>executorMemory * <code>spark.executor.memoryOverheadFactor</code>, with
minimum of <code>spark.executor.minMemoryOverhead</code></td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.minMemoryOverhead</code></td>
+ <td>384m</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.memoryOverheadFactor</code></td>
+ <td>0.10</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.resource.{resourceName}.amount</code></td>
+ <td>0</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.resource.{resourceName}.discoveryScript</code></td>
+ <td>None</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.resource.{resourceName}.vendor</code></td>
+ <td>None</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.extraListeners</code></td>
+ <td>(none)</td>
+ <td>1.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.local.dir</code></td>
+ <td>/tmp</td>
+ <td>0.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.logConf</code></td>
+ <td>false</td>
+ <td>0.9.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.master</code></td>
+ <td>(none)</td>
+ <td>0.9.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.submit.deployMode</code></td>
+ <td>client</td>
+ <td>1.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.log.callerContext</code></td>
+ <td>(none)</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.log.level</code></td>
+ <td>(none)</td>
+ <td>3.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.supervise</code></td>
+ <td>false</td>
+ <td>1.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.timeout</code></td>
+ <td>0min</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.log.localDir</code></td>
+ <td>(none)</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.log.dfsDir</code></td>
+ <td>(none)</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.log.persistToDfs.enabled</code></td>
+ <td>false</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.log.layout</code></td>
+ <td>%d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.log.allowErasureCoding</code></td>
+ <td>false</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.decommission.enabled</code></td>
+ <td>false</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.decommission.killInterval</code></td>
+ <td>(none)</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.decommission.forceKillTimeout</code></td>
+ <td>(none)</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.decommission.signal</code></td>
+ <td>PWR</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.maxNumFailures</code></td>
+ <td>numExecutors * 2, with minimum of 3</td>
+ <td>3.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.failuresValidityInterval</code></td>
+ <td>(none)</td>
+ <td>3.5.0</td>
+ <td></td>
+</tr>
+</table>
+
+# Runtime Environment
+
+<table class="spark-config">
+<thead><tr><th>Property Name</th><th>Default</th><th>Since
Version</th><th>Gluten Status</th></tr></thead>
+<tr>
+ <td><code>spark.driver.extraClassPath</code></td>
+ <td>(none)</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.defaultJavaOptions</code></td>
+ <td>(none)</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.extraJavaOptions</code></td>
+ <td>(none)</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.extraLibraryPath</code></td>
+ <td>(none)</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.driver.userClassPathFirst</code></td>
+ <td>false</td>
+ <td>1.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.extraClassPath</code></td>
+ <td>(none)</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.defaultJavaOptions</code></td>
+ <td>(none)</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.extraJavaOptions</code></td>
+ <td>(none)</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.extraLibraryPath</code></td>
+ <td>(none)</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.logs.rolling.maxRetainedFiles</code></td>
+ <td>-1</td>
+ <td>1.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.logs.rolling.enableCompression</code></td>
+ <td>false</td>
+ <td>2.0.2</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.logs.rolling.maxSize</code></td>
+ <td>1024 * 1024</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.logs.rolling.strategy</code></td>
+ <td>"" (disabled)</td>
+ <td>1.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.logs.rolling.time.interval</code></td>
+ <td>daily</td>
+ <td>1.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.userClassPathFirst</code></td>
+ <td>false</td>
+ <td>1.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executorEnv.[EnvironmentVariableName]</code></td>
+ <td>(none)</td>
+ <td>0.9.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.redaction.regex</code></td>
+ <td>(?i)secret|password|token|access[.]?key</td>
+ <td>2.1.2</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.redaction.string.regex</code></td>
+ <td>(none)</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.python.profile</code></td>
+ <td>false</td>
+ <td>1.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.python.profile.dump</code></td>
+ <td>(none)</td>
+ <td>1.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.python.worker.memory</code></td>
+ <td>512m</td>
+ <td>1.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.python.worker.reuse</code></td>
+ <td>true</td>
+ <td>1.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.files</code></td>
+ <td></td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.submit.pyFiles</code></td>
+ <td></td>
+ <td>1.0.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.jars</code></td>
+ <td></td>
+ <td>0.9.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.jars.packages</code></td>
+ <td></td>
+ <td>1.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.jars.excludes</code></td>
+ <td></td>
+ <td>1.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.jars.ivy</code></td>
+ <td></td>
+ <td>1.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.jars.ivySettings</code></td>
+ <td></td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+ <tr>
+ <td><code>spark.jars.repositories</code></td>
+ <td></td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.archives</code></td>
+ <td></td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.pyspark.driver.python</code></td>
+ <td></td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.pyspark.python</code></td>
+ <td></td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+</table>
+
+# Shuffle Behavior
+
+<table class="spark-config">
+<thead><tr><th>Property Name</th><th>Default</th><th>Since
Version</th><th>Gluten Status</th></tr></thead>
+<tr>
+ <td><code>spark.reducer.maxSizeInFlight</code></td>
+ <td>48m</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.reducer.maxReqsInFlight</code></td>
+ <td>Int.MaxValue</td>
+ <td>2.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.reducer.maxBlocksInFlightPerAddress</code></td>
+ <td>Int.MaxValue</td>
+ <td>2.2.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.compress</code></td>
+ <td>true</td>
+ <td>0.6.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.file.buffer</code></td>
+ <td>32k</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.file.merge.buffer</code></td>
+ <td>32k</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.unsafe.file.output.buffer</code></td>
+ <td>32k</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.localDisk.file.output.buffer</code></td>
+ <td>32k</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.spill.diskWriteBufferSize</code></td>
+ <td>1024 * 1024</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.io.maxRetries</code></td>
+ <td>3</td>
+ <td>1.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.io.numConnectionsPerPeer</code></td>
+ <td>1</td>
+ <td>1.2.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.io.preferDirectBufs</code></td>
+ <td>true</td>
+ <td>1.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.io.retryWait</code></td>
+ <td>5s</td>
+ <td>1.2.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.io.backLog</code></td>
+ <td>-1</td>
+ <td>1.1.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.io.connectionTimeout</code></td>
+ <td>value of <code>spark.network.timeout</code></td>
+ <td>1.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.io.connectionCreationTimeout</code></td>
+ <td>value of <code>spark.shuffle.io.connectionTimeout</code></td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.service.enabled</code></td>
+ <td>false</td>
+ <td>1.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.service.port</code></td>
+ <td>7337</td>
+ <td>1.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.service.name</code></td>
+ <td>spark_shuffle</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.service.index.cache.size</code></td>
+ <td>100m</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.service.removeShuffle</code></td>
+ <td>true</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.maxChunksBeingTransferred</code></td>
+ <td>Long.MAX_VALUE</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.sort.bypassMergeThreshold</code></td>
+ <td>200</td>
+ <td>1.1.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.sort.io.plugin.class</code></td>
+ <td>org.apache.spark.shuffle.sort.io.LocalDiskShuffleDataIO</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.spill.compress</code></td>
+ <td>true</td>
+ <td>0.9.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.accurateBlockThreshold</code></td>
+ <td>100 * 1024 * 1024</td>
+ <td>2.2.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.accurateBlockSkewedFactor</code></td>
+ <td>-1.0</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.registration.timeout</code></td>
+ <td>5000</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.registration.maxAttempts</code></td>
+ <td>3</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.reduceLocality.enabled</code></td>
+ <td>true</td>
+ <td>1.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.mapOutput.minSizeForBroadcast</code></td>
+ <td>512k</td>
+ <td>2.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.detectCorrupt</code></td>
+ <td>true</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.detectCorrupt.useExtraMemory</code></td>
+ <td>false</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.useOldFetchProtocol</code></td>
+ <td>false</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.readHostLocalDisk</code></td>
+ <td>true</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.files.io.connectionTimeout</code></td>
+ <td>value of <code>spark.network.timeout</code></td>
+ <td>1.6.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.files.io.connectionCreationTimeout</code></td>
+ <td>value of <code>spark.files.io.connectionTimeout</code></td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.checksum.enabled</code></td>
+ <td>true</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.checksum.algorithm</code></td>
+ <td>ADLER32</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.service.fetch.rdd.enabled</code></td>
+ <td>false</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.service.db.enabled</code></td>
+ <td>true</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.service.db.backend</code></td>
+ <td>ROCKSDB</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+</table>
+
+# Spark UI
+
+<table class="spark-config">
+<thead><tr><th>Property Name</th><th>Default</th><th>Since
Version</th><th>Gluten Status</th></tr></thead>
+<tr>
+ <td><code>spark.eventLog.logBlockUpdates.enabled</code></td>
+ <td>false</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.eventLog.longForm.enabled</code></td>
+ <td>false</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.eventLog.compress</code></td>
+ <td>true</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.eventLog.compression.codec</code></td>
+ <td>zstd</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.eventLog.erasureCoding.enabled</code></td>
+ <td>false</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.eventLog.dir</code></td>
+ <td>file:///tmp/spark-events</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.eventLog.enabled</code></td>
+ <td>false</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.eventLog.overwrite</code></td>
+ <td>false</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.eventLog.buffer.kb</code></td>
+ <td>100k</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.eventLog.rolling.enabled</code></td>
+ <td>false</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.eventLog.rolling.maxFileSize</code></td>
+ <td>128m</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.dagGraph.retainedRootRDDs</code></td>
+ <td>Int.MaxValue</td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.groupSQLSubExecutionEnabled</code></td>
+ <td>true</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.enabled</code></td>
+ <td>true</td>
+ <td>1.1.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.store.path</code></td>
+ <td>None</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.killEnabled</code></td>
+ <td>true</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.threadDumpsEnabled</code></td>
+ <td>true</td>
+ <td>1.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.threadDump.flamegraphEnabled</code></td>
+ <td>true</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.heapHistogramEnabled</code></td>
+ <td>true</td>
+ <td>3.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.liveUpdate.period</code></td>
+ <td>100ms</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.liveUpdate.minFlushPeriod</code></td>
+ <td>1s</td>
+ <td>2.4.2</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.port</code></td>
+ <td>4040</td>
+ <td>0.7.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.retainedJobs</code></td>
+ <td>1000</td>
+ <td>1.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.retainedStages</code></td>
+ <td>1000</td>
+ <td>0.9.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.retainedTasks</code></td>
+ <td>100000</td>
+ <td>2.0.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.reverseProxy</code></td>
+ <td>false</td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.reverseProxyUrl</code></td>
+ <td></td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.proxyRedirectUri</code></td>
+ <td></td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.showConsoleProgress</code></td>
+ <td>false</td>
+ <td>1.2.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.consoleProgress.update.interval</code></td>
+ <td>200</td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.custom.executor.log.url</code></td>
+ <td>(none)</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.prometheus.enabled</code></td>
+ <td>true</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.worker.ui.retainedExecutors</code></td>
+ <td>1000</td>
+ <td>1.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.worker.ui.retainedDrivers</code></td>
+ <td>1000</td>
+ <td>1.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.sql.ui.retainedExecutions</code></td>
+ <td>1000</td>
+ <td>1.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.streaming.ui.retainedBatches</code></td>
+ <td>1000</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.retainedDeadExecutors</code></td>
+ <td>100</td>
+ <td>2.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.filters</code></td>
+ <td>None</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.requestHeaderSize</code></td>
+ <td>8k</td>
+ <td>2.2.3</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.timelineEnabled</code></td>
+ <td>true</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.timeline.executors.maximum</code></td>
+ <td>250</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.timeline.jobs.maximum</code></td>
+ <td>500</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.timeline.stages.maximum</code></td>
+ <td>500</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.ui.timeline.tasks.maximum</code></td>
+ <td>1000</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.appStatusStore.diskStoreDir</code></td>
+ <td>None</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+</table>
+
+# Compression and Serialization
+
+<table class="spark-config">
+<thead><tr><th>Property Name</th><th>Default</th><th>Since
Version</th><th>Gluten Status</th></tr></thead>
+<tr>
+ <td><code>spark.broadcast.compress</code></td>
+ <td>true</td>
+ <td>0.6.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.checkpoint.dir</code></td>
+ <td>(none)</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.checkpoint.compress</code></td>
+ <td>false</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.io.compression.codec</code></td>
+ <td>lz4</td>
+ <td>0.8.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.io.compression.lz4.blockSize</code></td>
+ <td>32k</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.io.compression.snappy.blockSize</code></td>
+ <td>32k</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.io.compression.zstd.level</code></td>
+ <td>1</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.io.compression.zstd.bufferSize</code></td>
+ <td>32k</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.io.compression.zstd.bufferPool.enabled</code></td>
+ <td>true</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.io.compression.zstd.workers</code></td>
+ <td>0</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.io.compression.lzf.parallel.enabled</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.kryo.classesToRegister</code></td>
+ <td>(none)</td>
+ <td>1.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.kryo.referenceTracking</code></td>
+ <td>true</td>
+ <td>0.8.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.kryo.registrationRequired</code></td>
+ <td>false</td>
+ <td>1.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.kryo.registrator</code></td>
+ <td>(none)</td>
+ <td>0.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.kryo.unsafe</code></td>
+ <td>true</td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.kryoserializer.buffer.max</code></td>
+ <td>64m</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.kryoserializer.buffer</code></td>
+ <td>64k</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.rdd.compress</code></td>
+ <td>false</td>
+ <td>0.6.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.serializer</code></td>
+ <td>
+ org.apache.spark.serializer.<br />JavaSerializer
+ </td>
+ <td>0.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.serializer.objectStreamReset</code></td>
+ <td>100</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+</table>
+
+# Memory Management
+
+<table class="spark-config">
+<thead><tr><th>Property Name</th><th>Default</th><th>Since
Version</th><th>Gluten Status</th></tr></thead>
+<tr>
+ <td><code>spark.memory.fraction</code></td>
+ <td>0.6</td>
+ <td>1.6.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.memory.storageFraction</code></td>
+ <td>0.5</td>
+ <td>1.6.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.memory.offHeap.enabled</code></td>
+ <td>false</td>
+ <td>1.6.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.memory.offHeap.size</code></td>
+ <td>0</td>
+ <td>1.6.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.storage.unrollMemoryThreshold</code></td>
+ <td>1024 * 1024</td>
+ <td>1.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.storage.replication.proactive</code></td>
+ <td>true</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.storage.localDiskByExecutors.cacheSize</code></td>
+ <td>1000</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.cleaner.periodicGC.interval</code></td>
+ <td>30min</td>
+ <td>1.6.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.cleaner.referenceTracking</code></td>
+ <td>true</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.cleaner.referenceTracking.blocking</code></td>
+ <td>true</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.cleaner.referenceTracking.blocking.shuffle</code></td>
+ <td>false</td>
+ <td>1.1.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.cleaner.referenceTracking.cleanCheckpoints</code></td>
+ <td>false</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+</table>
+
+# Execution Behavior
+
+<table class="spark-config">
+<thead><tr><th>Property Name</th><th>Default</th><th>Since
Version</th><th>Gluten Status</th></tr></thead>
+<tr>
+ <td><code>spark.broadcast.blockSize</code></td>
+ <td>4m</td>
+ <td>0.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.broadcast.checksum</code></td>
+ <td>true</td>
+ <td>2.1.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.broadcast.UDFCompressionThreshold</code></td>
+ <td>1 * 1024 * 1024</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.cores</code></td>
+ <td>
+ 1 in YARN mode, all the available cores on the worker in standalone mode.
+ </td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.default.parallelism</code></td>
+ <td>
+ For distributed shuffle operations like <code>reduceByKey</code> and
<code>join</code>, the
+ largest number of partitions in a parent RDD. For operations like
<code>parallelize</code>
+ with no parent RDDs, it depends on the cluster manager:
+ <ul>
+ <li>Local mode: number of cores on the local machine</li>
+ <li>Others: total number of cores on all executor nodes or 2, whichever
is larger</li>
+ </ul>
+ </td>
+ <td>0.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.executor.heartbeatInterval</code></td>
+ <td>10s</td>
+ <td>1.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.files.fetchTimeout</code></td>
+ <td>60s</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.files.useFetchCache</code></td>
+ <td>true</td>
+ <td>1.2.2</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.files.overwrite</code></td>
+ <td>false</td>
+ <td>1.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.files.ignoreCorruptFiles</code></td>
+ <td>false</td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.files.ignoreMissingFiles</code></td>
+ <td>false</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.files.maxPartitionBytes</code></td>
+ <td>134217728 (128 MiB)</td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.files.openCostInBytes</code></td>
+ <td>4194304 (4 MiB)</td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.hadoop.cloneConf</code></td>
+ <td>false</td>
+ <td>1.0.3</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.hadoop.validateOutputSpecs</code></td>
+ <td>true</td>
+ <td>1.0.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.storage.memoryMapThreshold</code></td>
+ <td>2m</td>
+ <td>0.9.2</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.storage.decommission.enabled</code></td>
+ <td>false</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.storage.decommission.shuffleBlocks.enabled</code></td>
+ <td>true</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.storage.decommission.shuffleBlocks.maxThreads</code></td>
+ <td>8</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.storage.decommission.rddBlocks.enabled</code></td>
+ <td>true</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.storage.decommission.fallbackStorage.path</code></td>
+ <td>(none)</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.storage.decommission.fallbackStorage.cleanUp</code></td>
+ <td>false</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.storage.decommission.shuffleBlocks.maxDiskSize</code></td>
+ <td>(none)</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+
<td><code>spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version</code></td>
+ <td>1</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+</table>
+
+# Executor Metrics
+These configurations are handled by Spark and do not affect Gluten’s behavior.
+
+# Networking
+These configurations are handled by Spark and do not affect Gluten’s behavior.
+
+# Scheduling
+<table class="spark-config">
+<thead><tr><th>Property Name</th><th>Default</th><th>Since
Version</th><th></th></tr></thead>
+<tr>
+ <td><code>spark.cores.max</code></td>
+ <td>(not set)</td>
+ <td>0.6.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.locality.wait</code></td>
+ <td>3s</td>
+ <td>0.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.locality.wait.node</code></td>
+ <td>spark.locality.wait</td>
+ <td>0.8.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.locality.wait.process</code></td>
+ <td>spark.locality.wait</td>
+ <td>0.8.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.locality.wait.rack</code></td>
+ <td>spark.locality.wait</td>
+ <td>0.8.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.scheduler.maxRegisteredResourcesWaitingTime</code></td>
+ <td>30s</td>
+ <td>1.1.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.scheduler.minRegisteredResourcesRatio</code></td>
+ <td>0.8 for KUBERNETES mode; 0.8 for YARN mode; 0.0 for standalone mode</td>
+ <td>1.1.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.scheduler.mode</code></td>
+ <td>FIFO</td>
+ <td>0.8.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.scheduler.revive.interval</code></td>
+ <td>1s</td>
+ <td>0.8.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
+ <td>10000</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.scheduler.listenerbus.eventqueue.shared.capacity</code></td>
+ <td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+
<td><code>spark.scheduler.listenerbus.eventqueue.appStatus.capacity</code></td>
+ <td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+
<td><code>spark.scheduler.listenerbus.eventqueue.executorManagement.capacity</code></td>
+ <td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+
<td><code>spark.scheduler.listenerbus.eventqueue.eventLog.capacity</code></td>
+ <td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.scheduler.listenerbus.eventqueue.streams.capacity</code></td>
+ <td><code>spark.scheduler.listenerbus.eventqueue.capacity</code></td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.scheduler.resource.profileMergeConflicts</code></td>
+ <td>false</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+<tr>
+
<td><code>spark.scheduler.excludeOnFailure.unschedulableTaskSetTimeout</code></td>
+ <td>120s</td>
+ <td>2.4.1</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.standalone.submit.waitAppCompletion</code></td>
+ <td>false</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.excludeOnFailure.enabled</code></td>
+ <td>
+ false
+ </td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.excludeOnFailure.application.enabled</code></td>
+ <td>
+ false
+ </td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.excludeOnFailure.taskAndStage.enabled</code></td>
+ <td>
+ false
+ </td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.excludeOnFailure.timeout</code></td>
+ <td>1h</td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.excludeOnFailure.task.maxTaskAttemptsPerExecutor</code></td>
+ <td>1</td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.excludeOnFailure.task.maxTaskAttemptsPerNode</code></td>
+ <td>2</td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.excludeOnFailure.stage.maxFailedTasksPerExecutor</code></td>
+ <td>2</td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.excludeOnFailure.stage.maxFailedExecutorsPerNode</code></td>
+ <td>2</td>
+ <td>2.1.0</td>
+ <td></td>
+</tr>
+<tr>
+
<td><code>spark.excludeOnFailure.application.maxFailedTasksPerExecutor</code></td>
+ <td>2</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+<tr>
+
<td><code>spark.excludeOnFailure.application.maxFailedExecutorsPerNode</code></td>
+ <td>2</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.excludeOnFailure.killExcludedExecutors</code></td>
+ <td>false</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.excludeOnFailure.application.fetchFailure.enabled</code></td>
+ <td>false</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.speculation</code></td>
+ <td>false</td>
+ <td>0.6.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.speculation.interval</code></td>
+ <td>100ms</td>
+ <td>0.6.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.speculation.multiplier</code></td>
+ <td>3</td>
+ <td>0.6.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.speculation.quantile</code></td>
+ <td>0.9</td>
+ <td>0.6.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.speculation.minTaskRuntime</code></td>
+ <td>100ms</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.speculation.task.duration.threshold</code></td>
+ <td>None</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.speculation.efficiency.processRateMultiplier</code></td>
+ <td>0.75</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.speculation.efficiency.longRunTaskFactor</code></td>
+ <td>2</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.speculation.efficiency.enabled</code></td>
+ <td>true</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.task.cpus</code></td>
+ <td>1</td>
+ <td>0.5.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.task.resource.{resourceName}.amount</code></td>
+ <td>1</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.task.maxFailures</code></td>
+ <td>4</td>
+ <td>0.8.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.task.reaper.enabled</code></td>
+ <td>false</td>
+ <td>2.0.3</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.task.reaper.pollingInterval</code></td>
+ <td>10s</td>
+ <td>2.0.3</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.task.reaper.threadDump</code></td>
+ <td>true</td>
+ <td>2.0.3</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.task.reaper.killTimeout</code></td>
+ <td>-1</td>
+ <td>2.0.3</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.stage.maxConsecutiveAttempts</code></td>
+ <td>4</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.stage.ignoreDecommissionFetchFailure</code></td>
+ <td>true</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+</table>
+
+# Barrier Execution Mode
+These configurations are handled by Spark and do not affect Gluten’s behavior.
+
+# Dynamic Allocation
+These configurations are handled by Spark and do not affect Gluten’s behavior.
+
+# Thread Configurations
+These configurations are handled by Spark and do not affect Gluten’s behavior.
+
+# Spark Connect
+## Server Configuration
+These configurations are handled by Spark and do not affect Gluten’s behavior.
+
+# Security
+These configurations are handled by Spark and do not affect Gluten’s behavior.
+
+# Spark SQL
+
+## Runtime SQL Configuration
+<table class="spark-config">
+<thead><tr><th>Property Name</th><th>Default</th><th>Since
Version</th><th>Gluten Status</th></tr></thead>
+
+<tr>
+ <td><code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code></td>
+ <td>(value of
<code>spark.sql.adaptive.shuffle.targetPostShuffleInputSize</code>)</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.adaptive.autoBroadcastJoinThreshold</code></td>
+ <td>(none)</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.adaptive.coalescePartitions.enabled</code></td>
+ <td>true</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.adaptive.coalescePartitions.initialPartitionNum</code></td>
+ <td>(none)</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.adaptive.coalescePartitions.minPartitionSize</code></td>
+ <td>1MB</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.adaptive.coalescePartitions.parallelismFirst</code></td>
+ <td>true</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.adaptive.customCostEvaluatorClass</code></td>
+ <td>(none)</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.adaptive.enabled</code></td>
+ <td>true</td>
+ <td>1.6.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.adaptive.forceOptimizeSkewedJoin</code></td>
+ <td>false</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.adaptive.localShuffleReader.enabled</code></td>
+ <td>true</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.adaptive.maxShuffledHashJoinLocalMapThreshold</code></td>
+ <td>0b</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.adaptive.optimizeSkewsInRebalancePartitions.enabled</code></td>
+ <td>true</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.adaptive.optimizer.excludedRules</code></td>
+ <td>(none)</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.adaptive.rebalancePartitionsSmallPartitionFactor</code></td>
+ <td>0.2</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.adaptive.skewJoin.enabled</code></td>
+ <td>true</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.adaptive.skewJoin.skewedPartitionFactor</code></td>
+ <td>5.0</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes</code></td>
+ <td>256MB</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.allowNamedFunctionArguments</code></td>
+ <td>true</td>
+ <td>3.5.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.ansi.doubleQuotedIdentifiers</code></td>
+ <td>false</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.ansi.enabled</code></td>
+ <td>true</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.ansi.enforceReservedKeywords</code></td>
+ <td>false</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.ansi.relationPrecedence</code></td>
+ <td>false</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.autoBroadcastJoinThreshold</code></td>
+ <td>10MB</td>
+ <td>1.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.avro.compression.codec</code></td>
+ <td>snappy</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.avro.deflate.level</code></td>
+ <td>-1</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.avro.filterPushdown.enabled</code></td>
+ <td>true</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.avro.xz.level</code></td>
+ <td>6</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.avro.zstandard.bufferPool.enabled</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.avro.zstandard.level</code></td>
+ <td>3</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.binaryOutputStyle</code></td>
+ <td>(none)</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.broadcastTimeout</code></td>
+ <td>300</td>
+ <td>1.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.bucketing.coalesceBucketsInJoin.enabled</code></td>
+ <td>false</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.bucketing.coalesceBucketsInJoin.maxBucketRatio</code></td>
+ <td>4</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.catalog.spark_catalog</code></td>
+ <td>builtin</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.cbo.enabled</code></td>
+ <td>false</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.cbo.joinReorder.dp.star.filter</code></td>
+ <td>false</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.cbo.joinReorder.dp.threshold</code></td>
+ <td>12</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.cbo.joinReorder.enabled</code></td>
+ <td>false</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.cbo.planStats.enabled</code></td>
+ <td>false</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.cbo.starSchemaDetection</code></td>
+ <td>false</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.charAsVarchar</code></td>
+ <td>false</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.chunkBase64String.enabled</code></td>
+ <td>true</td>
+ <td>3.5.2</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.cli.print.header</code></td>
+ <td>false</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.columnNameOfCorruptRecord</code></td>
+ <td>_corrupt_record</td>
+ <td>1.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.csv.filterPushdown.enabled</code></td>
+ <td>true</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.datetime.java8API.enabled</code></td>
+ <td>false</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.debug.maxToStringFields</code></td>
+ <td>25</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.defaultCacheStorageLevel</code></td>
+ <td>MEMORY_AND_DISK</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.defaultCatalog</code></td>
+ <td>spark_catalog</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.error.messageFormat</code></td>
+ <td>PRETTY</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.arrow.enabled</code></td>
+ <td>false</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.arrow.fallback.enabled</code></td>
+ <td>true</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.arrow.localRelationThreshold</code></td>
+ <td>48MB</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.arrow.maxRecordsPerBatch</code></td>
+ <td>10000</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.arrow.pyspark.enabled</code></td>
+ <td>(value of <code>spark.sql.execution.arrow.enabled</code>)</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.arrow.pyspark.fallback.enabled</code></td>
+ <td>(value of <code>spark.sql.execution.arrow.fallback.enabled</code>)</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.execution.arrow.pyspark.selfDestruct.enabled</code></td>
+ <td>false</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.arrow.sparkr.enabled</code></td>
+ <td>false</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.execution.arrow.transformWithStateInPandas.maxRecordsPerBatch</code></td>
+ <td>10000</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.arrow.useLargeVarTypes</code></td>
+ <td>false</td>
+ <td>3.5.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.interruptOnCancel</code></td>
+ <td>true</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.pandas.inferPandasDictAsMap</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.pandas.structHandlingMode</code></td>
+ <td>legacy</td>
+ <td>3.5.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.pandas.udf.buffer.size</code></td>
+ <td>(value of <code>spark.buffer.size</code>)</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.pyspark.udf.faulthandler.enabled</code></td>
+ <td>(value of <code>spark.python.worker.faulthandler.enabled</code>)</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.pyspark.udf.hideTraceback.enabled</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.pyspark.udf.idleTimeoutSeconds</code></td>
+ <td>(value of <code>spark.python.worker.idleTimeoutSeconds</code>)</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled</code></td>
+ <td>true</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.python.udf.buffer.size</code></td>
+ <td>(value of <code>spark.buffer.size</code>)</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.python.udf.maxRecordsPerBatch</code></td>
+ <td>100</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.pythonUDF.arrow.concurrency.level</code></td>
+ <td>(none)</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.pythonUDF.arrow.enabled</code></td>
+ <td>false</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.pythonUDTF.arrow.enabled</code></td>
+ <td>false</td>
+ <td>3.5.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.execution.topKSortFallbackThreshold</code></td>
+ <td>2147483632</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.extendedExplainProviders</code></td>
+ <td>(none)</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.files.ignoreCorruptFiles</code></td>
+ <td>false</td>
+ <td>2.1.1</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.files.ignoreInvalidPartitionPaths</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.files.ignoreMissingFiles</code></td>
+ <td>false</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.files.maxPartitionBytes</code></td>
+ <td>128MB</td>
+ <td>2.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.files.maxPartitionNum</code></td>
+ <td>(none)</td>
+ <td>3.5.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.files.maxRecordsPerFile</code></td>
+ <td>0</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.files.minPartitionNum</code></td>
+ <td>(none)</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.function.concatBinaryAsString</code></td>
+ <td>false</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.function.eltOutputAsString</code></td>
+ <td>false</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.groupByAliases</code></td>
+ <td>true</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.groupByOrdinal</code></td>
+ <td>true</td>
+ <td>2.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.convertInsertingPartitionedTable</code></td>
+ <td>true</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.convertInsertingUnpartitionedTable</code></td>
+ <td>true</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.convertMetastoreCtas</code></td>
+ <td>true</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.convertMetastoreInsertDir</code></td>
+ <td>true</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.convertMetastoreOrc</code></td>
+ <td>true</td>
+ <td>2.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.convertMetastoreParquet</code></td>
+ <td>true</td>
+ <td>1.1.1</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.convertMetastoreParquet.mergeSchema</code></td>
+ <td>false</td>
+ <td>1.3.1</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.dropPartitionByName.enabled</code></td>
+ <td>false</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.filesourcePartitionFileCacheSize</code></td>
+ <td>262144000</td>
+ <td>2.1.1</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.manageFilesourcePartitions</code></td>
+ <td>true</td>
+ <td>2.1.1</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.metastorePartitionPruning</code></td>
+ <td>true</td>
+ <td>1.5.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.hive.metastorePartitionPruningFallbackOnException</code></td>
+ <td>false</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.metastorePartitionPruningFastFallback</code></td>
+ <td>false</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.thriftServer.async</code></td>
+ <td>true</td>
+ <td>1.5.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.icu.caseMappings.enabled</code></td>
+ <td>true</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.inMemoryColumnarStorage.batchSize</code></td>
+ <td>10000</td>
+ <td>1.1.1</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.inMemoryColumnarStorage.compressed</code></td>
+ <td>true</td>
+ <td>1.0.1</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.inMemoryColumnarStorage.enableVectorizedReader</code></td>
+ <td>true</td>
+ <td>2.3.1</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.inMemoryColumnarStorage.hugeVectorReserveRatio</code></td>
+ <td>1.2</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.inMemoryColumnarStorage.hugeVectorThreshold</code></td>
+ <td>-1b</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.json.filterPushdown.enabled</code></td>
+ <td>true</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.json.useUnsafeRow</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.jsonGenerator.ignoreNullFields</code></td>
+ <td>true</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.leafNodeDefaultParallelism</code></td>
+ <td>(none)</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.mapKeyDedupPolicy</code></td>
+ <td>EXCEPTION</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.maven.additionalRemoteRepositories</code></td>
+ <td>https://maven-central.storage-download.googleapis.com/maven2/</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.maxMetadataStringLength</code></td>
+ <td>100</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.maxPlanStringLength</code></td>
+ <td>2147483632</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.maxSinglePartitionBytes</code></td>
+ <td>128m</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.operatorPipeSyntaxEnabled</code></td>
+ <td>true</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.optimizer.avoidCollapseUDFWithExpensiveExpr</code></td>
+ <td>true</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.optimizer.collapseProjectAlwaysInline</code></td>
+ <td>false</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.optimizer.dynamicPartitionPruning.enabled</code></td>
+ <td>true</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.optimizer.enableCsvExpressionOptimization</code></td>
+ <td>true</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.optimizer.enableJsonExpressionOptimization</code></td>
+ <td>true</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.optimizer.excludedRules</code></td>
+ <td>(none)</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold</code></td>
+ <td>10GB</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.optimizer.runtime.bloomFilter.creationSideThreshold</code></td>
+ <td>10MB</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.optimizer.runtime.bloomFilter.enabled</code></td>
+ <td>true</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.optimizer.runtime.bloomFilter.expectedNumItems</code></td>
+ <td>1000000</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.optimizer.runtime.bloomFilter.maxNumBits</code></td>
+ <td>67108864</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.optimizer.runtime.bloomFilter.maxNumItems</code></td>
+ <td>4000000</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.optimizer.runtime.bloomFilter.numBits</code></td>
+ <td>8388608</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.optimizer.runtime.rowLevelOperationGroupFilter.enabled</code></td>
+ <td>true</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.optimizer.runtimeFilter.number.threshold</code></td>
+ <td>10</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.orc.aggregatePushdown</code></td>
+ <td>false</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.orc.columnarReaderBatchSize</code></td>
+ <td>4096</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.orc.columnarWriterBatchSize</code></td>
+ <td>1024</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.orc.compression.codec</code></td>
+ <td>zstd</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.orc.enableNestedColumnVectorizedReader</code></td>
+ <td>true</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.orc.enableVectorizedReader</code></td>
+ <td>true</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.orc.filterPushdown</code></td>
+ <td>true</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.orc.mergeSchema</code></td>
+ <td>false</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.orderByOrdinal</code></td>
+ <td>true</td>
+ <td>2.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.aggregatePushdown</code></td>
+ <td>false</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.binaryAsString</code></td>
+ <td>false</td>
+ <td>1.1.1</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.columnarReaderBatchSize</code></td>
+ <td>4096</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.compression.codec</code></td>
+ <td>snappy</td>
+ <td>1.1.1</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.enableNestedColumnVectorizedReader</code></td>
+ <td>true</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.enableVectorizedReader</code></td>
+ <td>true</td>
+ <td>2.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.fieldId.read.enabled</code></td>
+ <td>false</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.fieldId.read.ignoreMissing</code></td>
+ <td>false</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.fieldId.write.enabled</code></td>
+ <td>true</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.filterPushdown</code></td>
+ <td>true</td>
+ <td>1.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.inferTimestampNTZ.enabled</code></td>
+ <td>true</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.int96AsTimestamp</code></td>
+ <td>true</td>
+ <td>1.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.int96TimestampConversion</code></td>
+ <td>false</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.mergeSchema</code></td>
+ <td>false</td>
+ <td>1.5.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.outputTimestampType</code></td>
+ <td>INT96</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.recordLevelFilter.enabled</code></td>
+ <td>false</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.respectSummaryFiles</code></td>
+ <td>false</td>
+ <td>1.5.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parquet.writeLegacyFormat</code></td>
+ <td>false</td>
+ <td>1.6.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.parser.quotedRegexColumnNames</code></td>
+ <td>false</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.pivotMaxValues</code></td>
+ <td>10000</td>
+ <td>1.6.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.planner.pythonExecution.memory</code></td>
+ <td>(none)</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.preserveCharVarcharTypeInfo</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.pyspark.inferNestedDictAsStruct.enabled</code></td>
+ <td>false</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.pyspark.jvmStacktrace.enabled</code></td>
+ <td>false</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.pyspark.plotting.max_rows</code></td>
+ <td>1000</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.pyspark.udf.profiler</code></td>
+ <td>(none)</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.readSideCharPadding</code></td>
+ <td>true</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.redaction.options.regex</code></td>
+ <td>(?i)url</td>
+ <td>2.2.2</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.redaction.string.regex</code></td>
+ <td>(value of <code>spark.redaction.string.regex</code>)</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.repl.eagerEval.enabled</code></td>
+ <td>false</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.repl.eagerEval.maxNumRows</code></td>
+ <td>20</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.repl.eagerEval.truncate</code></td>
+ <td>20</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.scripting.enabled</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.session.localRelationCacheThreshold</code></td>
+ <td>67108864</td>
+ <td>3.5.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.session.timeZone</code></td>
+ <td>(value of local timezone)</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.shuffle.partitions</code></td>
+ <td>200</td>
+ <td>1.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.shuffleDependency.fileCleanup.enabled</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.shuffleDependency.skipMigration.enabled</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.shuffledHashJoinFactor</code></td>
+ <td>3</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.sources.bucketing.autoBucketedScan.enabled</code></td>
+ <td>true</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.sources.bucketing.enabled</code></td>
+ <td>true</td>
+ <td>2.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.sources.bucketing.maxBuckets</code></td>
+ <td>100000</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.sources.default</code></td>
+ <td>parquet</td>
+ <td>1.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.sources.parallelPartitionDiscovery.threshold</code></td>
+ <td>32</td>
+ <td>1.5.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.sources.partitionColumnTypeInference.enabled</code></td>
+ <td>true</td>
+ <td>1.5.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.sources.partitionOverwriteMode</code></td>
+ <td>STATIC</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.sources.v2.bucketing.allowCompatibleTransforms.enabled</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.sources.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys.enabled</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.sources.v2.bucketing.enabled</code></td>
+ <td>false</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.sources.v2.bucketing.partiallyClusteredDistribution.enabled</code></td>
+ <td>false</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.sources.v2.bucketing.partition.filter.enabled</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.sources.v2.bucketing.pushPartValues.enabled</code></td>
+ <td>true</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.sources.v2.bucketing.shuffle.enabled</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.sources.v2.bucketing.sorting.enabled</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.stackTracesInDataFrameContext</code></td>
+ <td>1</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.statistics.fallBackToHdfs</code></td>
+ <td>false</td>
+ <td>2.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.statistics.histogram.enabled</code></td>
+ <td>false</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.statistics.size.autoUpdate.enabled</code></td>
+ <td>false</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.statistics.updatePartitionStatsInAnalyzeTable.enabled</code></td>
+ <td>false</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.storeAssignmentPolicy</code></td>
+ <td>ANSI</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.checkpointLocation</code></td>
+ <td>(none)</td>
+ <td>2.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.continuous.epochBacklogQueueSize</code></td>
+ <td>10000</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.disabledV2Writers</code></td>
+ <td></td>
+ <td>2.3.1</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.fileSource.cleaner.numThreads</code></td>
+ <td>1</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.forceDeleteTempCheckpointLocation</code></td>
+ <td>false</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.metricsEnabled</code></td>
+ <td>false</td>
+ <td>2.0.2</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.multipleWatermarkPolicy</code></td>
+ <td>min</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.noDataMicroBatches.enabled</code></td>
+ <td>true</td>
+ <td>2.4.1</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.numRecentProgressUpdates</code></td>
+ <td>100</td>
+ <td>2.1.1</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.streaming.sessionWindow.merge.sessions.in.local.partition</code></td>
+ <td>false</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.stateStore.encodingFormat</code></td>
+ <td>unsaferow</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.stateStore.stateSchemaCheck</code></td>
+ <td>true</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.stopActiveRunOnRestart</code></td>
+ <td>true</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.stopTimeout</code></td>
+ <td>0</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+
<td><code>spark.sql.streaming.transformWithState.stateSchemaVersion</code></td>
+ <td>3</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.thriftServer.interruptOnCancel</code></td>
+ <td>(value of <code>spark.sql.execution.interruptOnCancel</code>)</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.thriftServer.queryTimeout</code></td>
+ <td>0ms</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.thriftserver.scheduler.pool</code></td>
+ <td>(none)</td>
+ <td>1.1.1</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.thriftserver.ui.retainedSessions</code></td>
+ <td>200</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.thriftserver.ui.retainedStatements</code></td>
+ <td>200</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.timeTravelTimestampKey</code></td>
+ <td>timestampAsOf</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.timeTravelVersionKey</code></td>
+ <td>versionAsOf</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.timestampType</code></td>
+ <td>TIMESTAMP_LTZ</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.transposeMaxValues</code></td>
+ <td>500</td>
+ <td>4.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.tvf.allowMultipleTableArguments.enabled</code></td>
+ <td>false</td>
+ <td>3.5.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.ui.explainMode</code></td>
+ <td>formatted</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.variable.substitute</code></td>
+ <td>true</td>
+ <td>2.0.0</td>
+ <td></td>
+</tr>
+</table>
+
+
+## Static SQL Configuration
+
+<table class="spark-config">
+<thead><tr><th>Property Name</th><th>Default</th><th>Since
Version</th><th>Gluten Status</th></tr></thead>
+
+<tr>
+ <td><code>spark.sql.cache.serializer</code></td>
+
<td>org.apache.spark.sql.execution.columnar.DefaultCachedBatchSerializer</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.catalog.spark_catalog.defaultDatabase</code></td>
+ <td>default</td>
+ <td>3.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.event.truncate.length</code></td>
+ <td>2147483647</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.extensions</code></td>
+ <td>(none)</td>
+ <td>2.2.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.extensions.test.loadFromCp</code></td>
+ <td>true</td>
+ <td></td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.metastore.barrierPrefixes</code></td>
+ <td></td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.metastore.jars</code></td>
+ <td>builtin</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.metastore.jars.path</code></td>
+ <td></td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.metastore.sharedPrefixes</code></td>
+ <td>com.mysql.jdbc,org.postgresql,com.microsoft.sqlserver,oracle.jdbc</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.metastore.version</code></td>
+ <td>2.3.10</td>
+ <td>1.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.thriftServer.singleSession</code></td>
+ <td>false</td>
+ <td>1.6.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.hive.version</code></td>
+ <td>2.3.10</td>
+ <td>1.1.1</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.metadataCacheTTLSeconds</code></td>
+ <td>-1000ms</td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.queryExecutionListeners</code></td>
+ <td>(none)</td>
+ <td>2.3.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.sources.disabledJdbcConnProviderList</code></td>
+ <td></td>
+ <td>3.1.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.streamingQueryListeners</code></td>
+ <td>(none)</td>
+ <td>2.4.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.ui.enabled</code></td>
+ <td>true</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.ui.retainedProgressUpdates</code></td>
+ <td>100</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.streaming.ui.retainedQueries</code></td>
+ <td>100</td>
+ <td>3.0.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.ui.retainedExecutions</code></td>
+ <td>1000</td>
+ <td>1.5.0</td>
+ <td></td>
+</tr>
+
+<tr>
+ <td><code>spark.sql.warehouse.dir</code></td>
+ <td>(value of <code>$PWD/spark-warehouse</code>)</td>
+ <td>2.0.0</td>
+ <td></td>
+</tr>
+</table>
+
+# Cluster Managers
+These configurations are handled by Spark and do not affect Gluten’s behavior.
+
+# Push-based shuffle overview
+
+## External Shuffle service(server) side configuration options
+
+<table class="spark-config">
+<thead><tr><th>Property Name</th><th>Default</th><th>Since
Version</th><th>Gluten Status</th></tr></thead>
+<tr>
+ <td><code>spark.shuffle.push.server.mergedShuffleFileManagerImpl</code></td>
+ <td>
+ <code>org.apache.spark.network.shuffle.<br
/>NoOpMergedShuffleFileManager</code>
+ </td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+
<td><code>spark.shuffle.push.server.minChunkSizeInMergedShuffleFile</code></td>
+ <td><code>2m</code></td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.push.server.mergedIndexCacheSize</code></td>
+ <td><code>100m</code></td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+</table>
+
+### Client side configuration options
+
+<table class="spark-config">
+<thead><tr><th>Property Name</th><th>Default</th><th>Since
Version</th><th>Gluten Status</th></tr></thead>
+<tr>
+ <td><code>spark.shuffle.push.enabled</code></td>
+ <td><code>false</code></td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.push.finalize.timeout</code></td>
+ <td><code>10s</code></td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.push.maxRetainedMergerLocations</code></td>
+ <td><code>500</code></td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.push.mergersMinThresholdRatio</code></td>
+ <td><code>0.05</code></td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.push.mergersMinStaticThreshold</code></td>
+ <td><code>5</code></td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.push.numPushThreads</code></td>
+ <td>(none)</td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.push.maxBlockSizeToPush</code></td>
+ <td><code>1m</code></td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.push.maxBlockBatchSize</code></td>
+ <td><code>3m</code></td>
+ <td>3.2.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.push.merge.finalizeThreads</code></td>
+ <td>8</td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.push.minShuffleSizeToWait</code></td>
+ <td><code>500m</code></td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+<tr>
+ <td><code>spark.shuffle.push.minCompletedPushRatio</code></td>
+ <td><code>1.0</code></td>
+ <td>3.3.0</td>
+ <td></td>
+</tr>
+</table>
+
+
+
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
