[
https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16622729#comment-16622729
]
Arun Mahadevan commented on SPARK-10816:
----------------------------------------
[~rxin], "map/flapMapGroupsWithState" looks to suited to mapping the grouped
values to custom state on the reduce side. Not sure its a general aggregation
mechanism that can merge the windows at the map side.
IMO, theres still value in providing a session window operator. For the more
advanced windowing it could be addressed by exposing Windowing with generic
"assignWindows" and "mergeWindows" constructs. It could then support different
types of custom windowing.
K,Values -> AssignWindows (produces [k, v, timestamp, window]) -> GroupByKey
(shuffle) -> MergeWindows (optional step) -> GroupWindows -> aggregate values.
The windows can also be merged and pre-aggregated at the map side for
optimization.
> EventTime based sessionization
> ------------------------------
>
> Key: SPARK-10816
> URL: https://issues.apache.org/jira/browse/SPARK-10816
> Project: Spark
> Issue Type: New Feature
> Components: Structured Streaming
> Reporter: Reynold Xin
> Priority: Major
> Attachments: SPARK-10816 Support session window natively.pdf
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
