[
https://issues.apache.org/jira/browse/BEAM-986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668726#comment-15668726
]
ASF GitHub Bot commented on BEAM-986:
-------------------------------------
GitHub user scwhittle opened a pull request:
https://github.com/apache/incubator-beam/pull/1366
[BEAM-986] Improve ReduceFnRunner prefetching
Hi @kennknowles , can you please take a look?
This refactoring ensures that for batches of elements or timers we have at
most 2 fetches, one to prefetch the should fire bits and one to fetch the
actual firing state. The current behavior can end up with 2 fetches per window
triggered.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/scwhittle/incubator-beam reduce_fn_prefetching
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/1366.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1366
----
commit 597af025865c6ae2208fc94ea8864491cd84ff37
Author: Sam Whittle <[email protected]>
Date: 2016-11-10T20:59:49Z
Improvements to ReduceFnRunner prefetching:
- add prefetch* methods for prefetching state matching existing methods
- replace onTimer with batched onTimers method to allow prefetching
across timers
- prefetch triggers in processElements
commit 8f11421019e2acd603546071ae0d8383143a33be
Author: Sam Whittle <[email protected]>
Date: 2016-11-15T22:55:38Z
fix annotation
----
> ReduceFnRunner doesn't batch prefetching pane firings
> -----------------------------------------------------
>
> Key: BEAM-986
> URL: https://issues.apache.org/jira/browse/BEAM-986
> Project: Beam
> Issue Type: Bug
> Components: runner-core
> Reporter: Sam Whittle
> Assignee: Sam Whittle
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Specifically
> - in ProcessElements, if there are multiple windows to consider each is
> processed sequentially with sequential state fetches instead of a bulk
> prefetch
> - onTimer method doesn't evaluate multiple timers at a time meaning that if
> multiple timers are fired at once each is processed sequentially without
> batched prefetching
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
