HeartSaVioR commented on a change in pull request #35762:
URL: https://github.com/apache/spark/pull/35762#discussion_r822169436
##########
File path: docs/structured-streaming-programming-guide.md
##########
@@ -1225,6 +1225,12 @@ Note that there are some restrictions when you use
session window in streaming q
For batch query, global window (only having `session_window` in grouping key)
is supported.
+Due to implementation details of session window in streaming query, for each
input row in same session,
Review comment:
Why not explain the behavior of default and which case it helps? If we
don't explain the case default is better, we fail to let end users reason about
why this config exists, as well as why we disable this by default.
How about revising the content like this?
> By default, Spark does not perform partial aggregation for session window
aggregation, since it requires additional sort in local partitions before
grouping. It works better for the case there are only few number of input rows
in same group key for each local partition, but for the case there are numerous
input rows having same group key in local partition, doing partial aggregation
can still increase the performance significantly despite additional sort.
>
> You can enable
`spark.sql.streaming.sessionWindow.merge.sessions.in.local.partition` to
indicate Spark to perform partial aggregation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
