HyukjinKwon commented on a change in pull request #27459:
[SPARK-30510][SQL][DOCS] Publicly document Spark SQL configuration options
URL: https://github.com/apache/spark/pull/27459#discussion_r375663487
##########
File path: docs/configuration.md
##########
@@ -2399,47 +2399,10 @@ the driver or executor, or, in the absence of that
value, the number of cores av
Please refer to the [Security](security.html) page for available options on
how to secure different
Spark subsystems.
-### Spark SQL
-
-Running the <code>SET -v</code> command will show the entire list of the SQL
configuration.
-
-<div class="codetabs">
-<div data-lang="scala" markdown="1">
-{% highlight scala %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(numRows = 200, truncate = false)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="java" markdown="1">
-
-{% highlight java %}
-// spark is an existing SparkSession
-spark.sql("SET -v").show(200, false);
-{% endhighlight %}
-</div>
-
-<div data-lang="python" markdown="1">
-
-{% highlight python %}
-# spark is an existing SparkSession
-spark.sql("SET -v").show(n=200, truncate=False)
-{% endhighlight %}
-
-</div>
-
-<div data-lang="r" markdown="1">
-
-{% highlight r %}
-sparkR.session()
-properties <- sql("SET -v")
-showDF(properties, numRows = 200, truncate = FALSE)
-{% endhighlight %}
+### Spark SQL
-</div>
-</div>
+{% include sql-configs.html %}
Review comment:
This is actually a bit tricky. If I generate the documentation with skipping
API, it throws an error. I investigated this for a while .. but couldn't find a
good solution. This below was one way I could come up with ..
```diff
diff --git a/docs/_plugins/environment_variables.rb
b/docs/_plugins/environment_variables.rb
new file mode 100644
index 00000000000..482b7b9f741
--- /dev/null
+++ b/docs/_plugins/environment_variables.rb
@@ -0,0 +1,25 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+module Jekyll
+ class EnvironmentVariables < Generator
+ def generate(site)
+ site.config['env'] = {}
+ site.config['env']['SKIP_API'] = ENV['SKIP_API']
+ site.config['env']['SKIP_SQLDOC'] = ENV['SKIP_SQLDOC']
+ end
+ end
+end
diff --git a/docs/configuration.md b/docs/configuration.md
index af18a6169d5..ddd5b72e1f7 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -2402,8 +2402,9 @@ Spark subsystems.
### Spark SQL
-{% include sql-configs.html %}
-
+{% if site.env.SKIP_API != '1' and ste.env.SKIP_SQLDOC != '1' %}
+ {% include sql-configs.html %}
+{% endif %}
### Spark Streaming
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
