[
https://issues.apache.org/jira/browse/BEAM-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16696637#comment-16696637
]
Maximilian Michels commented on BEAM-5865:
------------------------------------------
Would be great if you want to pick it up! As per mailing list discussion, we
might want to remove the rebalancing logic from the source and move this to the
Runner translation where we can decide whether we just forward the outputs or
add a rebalancing/reshuffle.
[~robertwb] suggested to replace the existing rebalancing logic with a special
URN (transform) which would then enable each Runner to translate it accordingly.
Do you want to start drafting this and open a PR? We can help you out with the
implementation details.
> Auto sharding of streaming sinks in FlinkRunner
> -----------------------------------------------
>
> Key: BEAM-5865
> URL: https://issues.apache.org/jira/browse/BEAM-5865
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Reporter: Maximilian Michels
> Priority: Major
>
> The Flink Runner should do auto-sharding of streaming sinks, similar to
> BEAM-1438. That way, the user doesn't have to set shards manually which
> introduces additional shuffling and might cause skew in the distribution of
> data.
> As per discussion:
> https://lists.apache.org/thread.html/7b92145dd9ae68da1866f1047445479f51d31f103d6407316bb4114c@%3Cuser.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
