Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/22238#discussion_r213267299
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -266,7 +266,9 @@ object SQLConf {
.createWithDefault(Long.MaxValue)
val SHUFFLE_PARTITIONS = buildConf("spark.sql.shuffle.partitions")
- .doc("The default number of partitions to use when shuffling data for
joins or aggregations.")
+ .doc("The default number of partitions to use when shuffling data for
joins or aggregations. " +
+ "Note: For structured streaming, this configuration cannot be
changed between query " +
--- End diff --
The sentence is borrowed from existing one:
https://github.com/apache/spark/blob/8198ea50192cad615071beb5510c73aa9e9178f4/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L978-L979
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
