[
https://issues.apache.org/jira/browse/SPARK-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16658935#comment-16658935
]
Jungtaek Lim edited comment on SPARK-10816 at 10/22/18 12:29 PM:
-----------------------------------------------------------------
Have been thinking about [3] with no load on memory, but can't find good
approach which doesn't introduce outstanding overhead.
Let's assume plenty of sessions in given key (which loading sessions in memory
will be a problem), as well as sessions are sorted: when eviction happens,
removal always happen on earliest which means worst case of shift always
happens which is linear to the number of sessions.
We could put tombstone to workaround this, but then the list will become longer
and longer, and when we do actual removal on tombstone it should take huge
latency.
was (Author: kabhwan):
Have been thinking about [3] but can't find good approach which doesn't
introduce outstanding overhead.
Let's assume plenty of sessions in given key (which loading sessions in memory
will be a problem), as well as sessions are sorted: when eviction happens,
removal always happen on earliest which means worst case of shift always
happens which is linear to the number of sessions.
We could put tombstone to workaround this, but then the list will become longer
and longer, and when we do actual removal on tombstone it should take huge
latency.
> EventTime based sessionization
> ------------------------------
>
> Key: SPARK-10816
> URL: https://issues.apache.org/jira/browse/SPARK-10816
> Project: Spark
> Issue Type: New Feature
> Components: Structured Streaming
> Reporter: Reynold Xin
> Priority: Major
> Attachments: SPARK-10816 Support session window natively.pdf, Session
> Window Support For Structure Streaming.pdf
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
