Re: Guidelines for setting task slots when running multiple jobs in a Flink cluster

2021-03-12 Thread Sush Bankapura
#task-chaining-and-resource-groups > > [2] https://issues.apache.org/jira/browse/FLINK-21267 > > [3] https://issues.apache.org/jira/browse/FLINK-10404 > > [4] https://issues.apache.org/jira/browse/FLINK-14187 > > > > Regards, > > Roman > > > > On F

Guidelines for setting task slots when running multiple jobs in a Flink cluster

2021-03-11 Thread Sush Bankapura
Hi, We have multiple jobs that need to be deployed to a Flink cluster. Parallelism for jobs vary and dependent on the type of work being done and so are the memory requirements. All jobs currently use the same state backend. Since the workloads handled by each job is different, the scaling

Re: Support for Event time clock specific to each stream in parallel streams

2020-08-04 Thread Sush Bankapura
rmark, causing > an idle device's events to become late. [SUSH] Yes. This is exactly what is needed - key based watermark support or at least support for subtask specific watermarks. The latter could be a possible option where there only a few keys and such an implementation does not have c

Support for Event time clock specific to each stream in parallel streams

2020-07-31 Thread Sush Bankapura
Hi, We have a single Flink job that works on data from multiple data sources. These data sources are not aligned in time and also have intermittent connectivity lasting for days, due to which data will arrive late We attempted to use the event time and watermarks with parallel streams using