Re: Question about parallelism

The client was not using a config file, it is a stand-alone java app using the flink-client jar file. Thanks for the clarification. Michael > On Apr 16, 2018, at 2:11 PM, Fabian Hueske wrote: > > The parallelism.default property that is configured in the flink-conf.yaml >

Re: Question about parallelism

The parallelism.default property that is configured in the flink-conf.yaml file is only considered if the config file belongs to the submitting client. If you configured the property in the config file of your cluster setup and used submitted from a client that used a different configuration file,

Re: Question about parallelism

1.4.2. I have since set the parallelism explicitly after creating the env and that is working. I also made the stream object serializable which may also be involved in this. I will retest without the explicit parallelism when I get a chance. Michael > On Apr 16, 2018, at 2:05 AM, Fabian

Re: Question about parallelism

(re-adding user mailing list) A non-serializable function object should cause the job to fail, but not to ignore a parallelism setting. This might be a bug. Most users specify the parallelism directly in the application code (via StreamExecutionEnvironment) or when submitting the application.

Question about parallelism

I am pretty new to flink. I have a flink job that has 10 transforms (mostly CoFlatMap with some simple filters and key extractrs as well. I have the config set for 6 slots and default parallelism of 6, but all my stages show paralellism of 1. Is that because there is only one task manager?

Re: Question about parallelism

18, 2017 at 2:08 PM, Jerry Peng <jerry.boyang.p...@gmail.com> > wrote: > > Hello all, > > > > I have a question about parallelism and partitioning in the > > DataStreams API. In Flink, a user can the parallelism of a data > > source as well as operators. So when I

Re: Question about parallelism

I guess my previous question is also asking if the parallelism is set for the operator or "data stream". Is there implied repartitioning when the parallelism changes? On Fri, Aug 18, 2017 at 2:08 PM, Jerry Peng <jerry.boyang.p...@gmail.com> wrote: > Hello all, > >

Question about parallelism

Hello all, I have a question about parallelism and partitioning in the DataStreams API. In Flink, a user can the parallelism of a data source as well as operators. So when I set the parallelism of a data source e.g. DataStream text = env.readTextFile(params.get("input")).setPar