Re: BackPressure in RowTime Task of FlinkSql Job

2021-02-26 Thread Aeden Jameson
>>Is it possible that you are generating to many watermarks that need to be send to all downstream tasks? This was it basically. I had unexpected flooding on specific keys, which was guessing intermittently hot partitions that was back pressuring the rowtime task. I do have another question, how

Re: BackPressure in RowTime Task of FlinkSql Job

2021-02-26 Thread Timo Walther
Hi Aeden, the rowtime task is actually just a simple map function that extracts the event-time timestamp into a field of the row for the next operator. It should not be the problem. Can you share a screenshot of your pipeline? What is your watermarking strategy? Is it possible that you are

BackPressure in RowTime Task of FlinkSql Job

2021-02-24 Thread Aeden Jameson
I have a job made up of a few FlinkSQL statements using a statement set. In my job graph viewed through the Flink UI a few of the tasks/statements are preceded by this task rowtime field: (#11: event_time TIME ATTRIBUTE(ROWTIME)) that has an upstream Kafka source/sink task.