mxm opened a new pull request, #556: URL: https://github.com/apache/flink-kubernetes-operator/pull/556
In a prior change we set the processing rates and capacity (true processing rate) to near zero values to allow scaling down in case of idle tasks. This has worked well but caused issues for tasks which are fed via upstream selective outputs (side outputs). Those tests usually received none or only a few records of the upstream outputted records. Setting low values for the incoming records rate will yield a relatively low true processing rate which quickly triggers upscaling even if no records are being processed at all. We don't currently have per-output/per-input metrics, so the best we can do, is to assume infinite processing capacity if we don't receive any records or are completely idle. Once we received records, we can measure our true processing rate. In the case of selective outputs, we will still assume we receive all the data which is suboptimal but far better than scaling up all the way to the max parallelism as we have seen for the near zero true processing rates during idleness. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
