Hello, On a k8s cluster, I have the flink-k8s-operator running 1.8 with autoscaler = enabled (in-place) and a flinkDeployment (application mode) running 1.18.1.
The flinkDeployment i.e. the flink streaming application has a mock data producer as the source. The source generates data points every X milli to be processed (aggregated) by the downstream operators. The aggregated results are written to Iceberg. The pipeline starts with default-parallelism = 1 i..e all the job vertexes start with par = 1 and X = 0 so all data points are generated continuously. Due to the lag associated with the aggregation and sink, the source experiences backpressure and hence the autoscaler triggers a scale-up. I want to slow down the speed of data production by source after the first scale-up event. What are the ways I can detect the scale-up event so that the source can dynamically adjust (increase) X at run-time? I am wondering if there is a way to detect if the parallelism of any of the job-vertex in the flink execution graph has gone above 1 within the source operator at runtime. This is a test pipeline (flink app) and the goal is to test the scale-up and scale-down events thus I need to increase X in order to have a scale-down event get triggered afterwards. Thank you Chetas
