[
https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631206#comment-16631206
]
Jungtaek Lim commented on SPARK-10816:
--------------------------------------
To avoid any concerns/wonders, I believe my proposal and
map/flatMapGroupsWithState can co-exist.
map/flatMapGroupsWithState target for general (arbitrary, not specific to
window) cases but can't be fully optimized for any specific cases by nature of
"generalization". Edge-case also came out from generalization, and if we tackle
the edge-case with map/flatMapGroupsWithState as supporting multiple values per
key, it would be non-trivial overhead for the cases which don't need to have
multiple values per key, as well as state function may be more complicated or
have couple of forms.
The point is whether simple gap session window is worth to be treated for first
class use cases. Spark supports tumble/slide window natively because we see
it's worth. I think we see worth of supporting session window since we have
example on sessionization, and I guess supporting it natively would give much
benefit over adding some complexities.
Same thing would apply if we would add some other API (function, or DSL) for
supporting custom window for follow-up issue (SPARK-25555 as Arun stated). If
we feel much convenient and see its worth to support it natively instead of let
end users play with map/flatMapGroupsWithState, it can be the thing to go.
> EventTime based sessionization
> ------------------------------
>
> Key: SPARK-10816
> URL: https://issues.apache.org/jira/browse/SPARK-10816
> Project: Spark
> Issue Type: New Feature
> Components: Structured Streaming
> Reporter: Reynold Xin
> Priority: Major
> Attachments: SPARK-10816 Support session window natively.pdf
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
