[jira] [Commented] (BEAM-260) Know the getSideInputWindow upper bound so can gc side input state
[ https://issues.apache.org/jira/browse/BEAM-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498458#comment-15498458 ] Aljoscha Krettek commented on BEAM-260: --- The doc looks good, I'm not sure how many people will see this comment on a Jira issue so it might make sense to [POPOSE] it on the ML. > Know the getSideInputWindow upper bound so can gc side input state > -- > > Key: BEAM-260 > URL: https://issues.apache.org/jira/browse/BEAM-260 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Mark Shields >Assignee: Kenneth Knowles > > We currently have no static knowledge about the getSideInputWindow function, > and runners are thus forced to hold on to all side input state / elements in > case a future element reaches back into an earlier side input element. > Maybe we need an upper bound on lag from current to result of > getSideInputWindow so we can have a progressing gc horizon as we do for GKB > window state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-260) Know the getSideInputWindow upper bound so can gc side input state
[ https://issues.apache.org/jira/browse/BEAM-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15494237#comment-15494237 ] Kenneth Knowles commented on BEAM-260: -- This came up again on the dev list in [this thread|http://mail-archives.apache.org/mod_mbox/incubator-beam-dev/201609.mbox/%3CCA%2B5xAo0JXSufSTSpa1cQ-yUU%3DgKnTK84HjBqWWL8T3NpiRpbOA%40mail.gmail.com%3E] by [~thw]. I have a simple proposed solution at https://s.apache.org/beam-windowmappingfn-1-pager. It is small, but may just be large enough to merit a \[PROPOSAL\] thread on the mailing list or a "BIP" if we ever keep a separate catalog of those. > Know the getSideInputWindow upper bound so can gc side input state > -- > > Key: BEAM-260 > URL: https://issues.apache.org/jira/browse/BEAM-260 > Project: Beam > Issue Type: Bug > Components: beam-model >Reporter: Mark Shields >Assignee: Kenneth Knowles > > We currently have no static knowledge about the getSideInputWindow function, > and runners are thus forced to hold on to all side input state / elements in > case a future element reaches back into an earlier side input element. > Maybe we need an upper bound on lag from current to result of > getSideInputWindow so we can have a progressing gc horizon as we do for GKB > window state. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (BEAM-260) Know the getSideInputWindow upper bound so can gc side input state
[
https://issues.apache.org/jira/browse/BEAM-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275128#comment-15275128
]
Aljoscha Krettek commented on BEAM-260:
---
I thought about this as well while working on the Flink Streaming side input
support. Could it be enough to have something like
{{WindowFn.getSideInputCleanupTime(BoundedWindow)}} that tells you when you can
GC a side input window based on the main-input watermark. This would be called
on the WindowFn of the side input, since it knows how the main-input windows
are mapped to side inputs.
> Know the getSideInputWindow upper bound so can gc side input state
> --
>
> Key: BEAM-260
> URL: https://issues.apache.org/jira/browse/BEAM-260
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Reporter: Mark Shields
>Assignee: Frances Perry
>
> We currently have no static knowledge about the getSideInputWindow function,
> and runners are thus forced to hold on to all side input state / elements in
> case a future element reaches back into an earlier side input element.
> Maybe we need an upper bound on lag from current to result of
> getSideInputWindow so we can have a progressing gc horizon as we do for GKB
> window state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
