Hi Olivier, Someone will correct me if I'm wrong, but I believe the max-parallelism limitation, where you cannot scale up past the previously defined max-parallelism, applies to all stateful jobs no matter which type of state you are using.
If you haven't seen it already, I think the Production Readiness checklist [1] offers a good explanation for this behavior/ best practices for setting it. Best, Austin [1]: https://ci.apache.org/projects/flink/flink-docs-master/docs/ops/production_ready/#set-an-explicit-max-parallelism On Tue, Apr 20, 2021 at 9:33 AM Olivier Nouguier <olivier.nougu...@teads.com> wrote: > Hi, thank you all for your reply, by limitation I meant the impossibility > to resume a job when scaling up because of this max-parallelism. > To be more precise, in our deployment, the operator (max) parallelism is > computed from the number of available slots ( ~~ task-manager * core ), > this approach is probably too naive, at least with keyed state. > > > > On Tue, Apr 20, 2021 at 10:46 AM Chesnay Schepler <ches...@apache.org> > wrote: > >> @Olivier Could you clarify which limitation you are referring to? >> >> On 4/20/2021 5:23 AM, Guowei Ma wrote: >> >> Hi, Olivier >> Yes. The introduction of this concept is to solve the problem of >> rescaling the keystate. >> Best, >> Guowei >> >> >> On Mon, Apr 19, 2021 at 8:56 PM Olivier Nouguier < >> olivier.nougu...@teads.com> wrote: >> >>> Hi, >>> May I have the confirmation that the max-parallelism limitation only >>> occurs when keyed states are used ? >>> >>> >>> -- >>> >>> Olivier Nouguier >>> >>> SSE >>> >>> e | olivier.nougu...@teads.com m | 0651383971 >>> >>> Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000 >>> Montpellier, France >>> [image: image] <https://www.teads.com> >>> The information in this email is confidential and intended only for the >>> addressee(s) named above. If you are not the intended recipient any >>> disclosure, copying, distribution or any action taken or omitted to be >>> taken in reliance on it is prohibited and may be unlawful. Teads does >>> not warrant that any attachment(s) are free from viruses or other defects >>> and accept no liability for any losses resulting from infected email >>> transmission. Please note that any views expressed in this email may be >>> those of the originator and do not necessarily reflect those of the >>> organization. >>> >> >> > > -- > > Olivier Nouguier > > SSE > > e | olivier.nougu...@teads.com m | 0651383971 > > Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000 > Montpellier, France > [image: image] <https://www.teads.com> > > The information in this email is confidential and intended only for the > addressee(s) named above. If you are not the intended recipient any > disclosure, copying, distribution or any action taken or omitted to be > taken in reliance on it is prohibited and may be unlawful. Teads does not > warrant that any attachment(s) are free from viruses or other defects and > accept no liability for any losses resulting from infected email > transmission. Please note that any views expressed in this email may be > those of the originator and do not necessarily reflect those of the > organization. >
