Parallelism of Keyed Process Function

Hi, Here is a question related to parallelism of keyed-process-function that is applied to the KeyedStream. For some code that looks like this myStream.keyBy(...) .process(new MyKeyedProcessFunction()) .process().setParallelism(10) On a Flink cluster with 5 TM nodes each with 10 task

Re: Watermark generation issues with File sources in Flink 1.11.1

your file-based sources, > what API methods are you using? > > Best, > Aljoscha > > On 20.08.20 16:21, Arti Pande wrote: > > Hi Till, > > > > Thank you for your quick response. Both the > AssignerWithPeriodicWatermarks > > and WatermarkStrategy I am

Re: Watermark generation issues with File sources in Flink 1.11.1

> > Cheers, > Till > > On Wed, Aug 19, 2020 at 5:38 PM Arti Pande wrote: > >> Hi, >> >> When migrating Stream API based Flink application from 1.9.2 to 1.11.1 >> the watermark generation has issues with file source alone. It works we

Watermark generation issues with File sources in Flink 1.11.1

Hi, When migrating Stream API based Flink application from 1.9.2 to 1.11.1 the watermark generation has issues with file source alone. It works well with Kafka source. With 1.9.2 a custom watermark generator implementation of AssignerWithPeriodicWatermarks was to be used. Starting 1.11.1 it is

Absolute latency with Latency Tracking

Hi Given an SLA of some x milliseconds of end-to-end latency, is there a way to know the end-to-end latency at record level (time taken by a record to be processed by the entire pipeline and written to sink) for a flink job? This

Is State TTL possible with event-time characteristics ?

With Flink 1.9 is state TTL supported for event-time characteristics? This part of the documentation says that - Only TTLs in reference to *processing time* are currently

Re: Flink Async IO operator tuning / micro-benchmarks

uch more reliable > environment that is overall cheaper. > > We unfortunately still need more time to fully incorporate dynamic > scaling-in and out (ETA Flink 1.12 with Flink 1.11 currently being > finalized), then you would be able to react on slower traffic (during > nig

Re: Flink Async IO operator tuning / micro-benchmarks

that fully > satures your async library (so maxConnections+X, where X is a small > constant safety buffer). It also helps to determine if you need to scale > out or not. > > > On Thu, Jun 11, 2020 at 9:03 AM Arti Pande wrote: > >> Hi Arvid, >> >> Thanks for a qu

Re: Flink Async IO operator tuning / micro-benchmarks

taArtisans/flink-benchmarks/blob/master/src/main/java/org/apache/flink/benchmark/AsyncWaitOperatorBenchmark.java > [2] http://codespeed.dak8s.net:8000/timeline/?ben=asyncWait.ORDERED=2 > > On Wed, Jun 10, 2020 at 10:06 PM Arti Pande wrote: > >> As Flink Async IO operator is d

Flink Async IO operator tuning / micro-benchmarks

As Flink Async IO operator is designed for external API or DB calls, are there any specific guidelines / tips for scaling up this operator? Particularly for use-cases where incoming events are being ingested at a very high-speed and the Async IO operator with orderedWait mode can not keep up with