Re: Does withkeys transform enforce a reshuffle?

Reshuffle is perfectly fine to use if the goal is just to redistribute work. It's only deprecated as a "checkpointing" mechanism. On Fri, Jan 19, 2024 at 9:44 AM Danny McCormick via user wrote: > > For runners that support Reshuffle, it should be safe to use. Its been > "deprecated" for 7

Re: Does withkeys transform enforce a reshuffle?

For runners that support Reshuffle, it should be safe to use. Its been "deprecated" for 7 years, but is still heavily used/often the recommended way to do things like this. I actually just added a PR to undeprecate it earlier today. Looks like you're

Re: Does withkeys transform enforce a reshuffle?

Also I looked at the code, reshuffle seems doing some groupby work internally. But I don't really need groupby On Fri, Jan 19, 2024 at 9:35 AM hsy...@gmail.com wrote: > ReShuffle is deprecated > > On Fri, Jan 19, 2024 at 8:25 AM XQ Hu via user > wrote: > >> I do not think it enforces a

Re: Does withkeys transform enforce a reshuffle?

ReShuffle is deprecated On Fri, Jan 19, 2024 at 8:25 AM XQ Hu via user wrote: > I do not think it enforces a reshuffle by just checking the doc here: > https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html?highlight=withkeys#apache_beam.transforms.util.WithKeys > >

Re: Does withkeys transform enforce a reshuffle?

I do not think it enforces a reshuffle by just checking the doc here: https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html?highlight=withkeys#apache_beam.transforms.util.WithKeys Have you tried to just add ReShuffle after PubsubLiteIO? On Thu, Jan 18, 2024 at 8:54 PM