+1
On Tue, Apr 25, 2023 at 5:24 PM David Morávek <d...@apache.org> wrote: > > Hi Eric, > > this sounds reasonable, there are definitely cases where you need to limit > sink parallelism for example not to overload the storage or limit the > number of output files > > +1 > > Best, > D. > > On Sun, Apr 23, 2023 at 1:09 PM Weihua Hu <huweihua....@gmail.com> wrote: > > > Hi, Eric > > > > Thanks for bringing this discussion. > > I think it's reasonable to add ''setMaxParallelism" for DataStreamSink. > > > > +1 > > > > Best, > > Weihua > > > > > > On Sat, Apr 22, 2023 at 3:20 AM eric xiao <xiao.eric...@gmail.com> wrote: > > > > > Hi there devs, > > > > > > I would like to start a discussion thread for FLINK-31873[1]. > > > > > > We are in the processing of enabling Flink reactive mode as the default > > > scheduling mode. While reading configuration docs [2] (I believe it was > > > also mentioned during one of the training sessions during Flink Forward > > > 2022), one can/should replace all setParallelism calls with > > > setMaxParallelism when migrating to reactive mode. > > > > > > This currently isn't possible on a sink in a Flink pipeline as we do not > > > expose a setMaxParallelism on the DataStreamSink class [3]. The > > underlying > > > Transformation class does have both a setMaxParallelism and > > setParallelism > > > function defined [4], but only setParallelism is offered in the > > > DataStreamSink class. > > > > > > I believe adding setMaxParallelism would be beneficial for not just flink > > > reactive mode, both modes of running of a flink pipeline (non reactive > > > mode, flink auto scaling). > > > > > > Best, > > > > > > Eric Xiao > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-31873 > > > [2] > > > > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/elastic_scaling/#configuration > > > [3] > > > > > > > > https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStreamSink.java > > > [4] > > > > > > > > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/dag/Transformation.java#L248-L285 > > > > >
