nielm commented on pull request #14811:
URL: https://github.com/apache/beam/pull/14811#issuecomment-890849612
> Are you sure that this splitting implementation is beneficial ?
>
> I think what it actually does is disabling (or reducing) partition-based
initial splitting and pushing partition based splitting to dynamic splitting.
I agree with Cham here,
From a high-level view, this PR replaces:
- a 1->many fan-out - GeneratePartitionsFn \
(where in practice "many" is only a few 10s of elements, not millions)
- followed by a Reshuffle
with a single splittable DoFn
In my opinion this will not increase parallelization, and makes the code
more confusing/verbose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
