nchammas commented on code in PR #44300:
URL: https://github.com/apache/spark/pull/44300#discussion_r1423139009
##########
docs/sql-performance-tuning.md:
##########
@@ -34,30 +34,9 @@ memory usage and GC pressure. You can call
`spark.catalog.uncacheTable("tableNam
Configuration of in-memory caching can be done using the `setConf` method on
`SparkSession` or by running
`SET key=value` commands using SQL.
-<table>
-<thead><tr><th>Property Name</th><th>Default</th><th>Meaning</th><th>Since
Version</th></tr></thead>
-<tr>
- <td><code>spark.sql.inMemoryColumnarStorage.compressed</code></td>
- <td>true</td>
- <td>
- When set to true, Spark SQL will automatically select a compression codec
for each column based
- on statistics of the data.
- </td>
- <td>1.0.1</td>
-</tr>
-<tr>
- <td><code>spark.sql.inMemoryColumnarStorage.batchSize</code></td>
- <td>10000</td>
- <td>
- Controls the size of batches for columnar caching. Larger batch sizes can
improve memory utilization
- and compression, but risk OOMs when caching data.
- </td>
- <td>1.1.1</td>
-</tr>
-
-</table>
+{% include_relative generated-sql-config-table-caching-data.html %}
Review Comment:
This diff demonstrates the main benefit of this PR. Instead of needing to
copy and maintain the full HTML table of some configs, we tags the ones we want
to group together in `SQLConf.scala` and then reference that group's table here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
