[
https://issues.apache.org/jira/browse/BEAM-47?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Cwik closed BEAM-47.
-------------------------
Resolution: Won't Do
Fix Version/s: Not applicable
SplittableDoFn will replace this and has splitting designed within it for
unbounded restrictions.
> UnboundedSource should support dynamic work rebalancing
> -------------------------------------------------------
>
> Key: BEAM-47
> URL: https://issues.apache.org/jira/browse/BEAM-47
> Project: Beam
> Issue Type: New Feature
> Components: io-ideas
> Reporter: Daniel Halperin
> Priority: Major
> Fix For: Not applicable
>
>
> BoundedSource has splitAtFraction and related code to enable large work items
> to be split dynamically and moved to other workers.
> UnboundedSource does not have this support -- instead, runners are expected
> to allocate many splits initially and rebalance those between workers.
> We should add dynamic work rebalancing to UnboundedSource. This will almost
> certainly be done by checkpointing the source, and then the dynamically
> produced splits will both inherit the checkpoint (or a split-adapted version
> of it).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
