Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21733#discussion_r206776398
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -871,6 +871,16 @@ object SQLConf {
.intConf
.createWithDefault(2)
+ val STREAMING_AGGREGATION_STATE_FORMAT_VERSION =
+
buildConf("spark.sql.streaming.streamingAggregation.stateFormatVersion")
+ .internal()
+ .doc("State format version used by streaming aggregation operations
triggered " +
+ "explicitly or implicitly via agg() in a streaming query. State
between versions are " +
--- End diff --
I was to explain that the option only applies to the operators which go
through StateStoreRestoreExec / StateStoreSaveExec (so max("field1") as well as
agg("field1" -> "max")), but now I feel it just gives confusion and I don't
think end users need to understand details behind of config. Will remove the
part `explicitly or implicitly via agg()`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
