Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/22238#discussion_r213123711
--- Diff: docs/structured-streaming-programming-guide.md ---
@@ -2812,6 +2812,12 @@ See [Input Sources](#input-sources) and [Output
Sinks](#output-sinks) sections f
# Additional Information
+**Gotchas**
--- End diff --
I was going to add the explanation to `doc()` of
`spark.sql.shuffle.partitions`, but looks like what we explained in `doc()`
would not be published automatically. (Please correct me if I'm missing here.)
SQLConf is even not exposed to scaladoc. That's why I'm adding this to
structured streaming guide doc. Actually I think most of end users only take a
look at this doc for structured streaming, and we can't (and shouldn't) expect
end users to take a look at source code to find it.
But also actually I didn't notice that `spark.sql.shuffle.partitions` is
explained in `sql-programming-guide.md` but I also think we need to explain all
configs here if they work differently with batch query.
`spark.sql.shuffle.partitions` is the case.
Btw, `Gotchas` looks like funny though. Maybe having section would be
better. Maybe like `## Other Configuration Options` in
`sql-programming-guide.md`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
