Interesting. So if I understand correctly, basically you limited the parallelism of the sources in order to avoid running the job with constant backpressure, and then scaled up the windows to maximize throughput.
On Tue, May 4, 2021 at 11:23 PM vishalovercome <vis...@moengage.com> wrote: > In one of my jobs, windowing is the costliest operation while upstream and > downstream operators are not as resource intensive. There's another > operator > in this job that communicates with internal services. This has high > parallelism as well but not as much as that of the windowing operation. > Running all operators with the same parallelism as the windowing operation > would choke some of our internal services we'll be consuming from our > source > at a rate much higher than what our internal services can handle. Thus our > sources, sinks, validation, monitoring related operators have very low > parallelism while one has high parallelism and another has even higher > parallelism. > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >
