aplyusnin commented on PR #847: URL: https://github.com/apache/flink-kubernetes-operator/pull/847#issuecomment-2327364352
Hi, sorry for a big delay. I've tried to simplify the logic of the backpropagation. Every vertex has it's initial target data rate and the one limited by the donwstream and potential performance after scaling. Potential performance is how much data can an operator process when scaled by the maximum scale factor. Donwstream limitation is evaluated very close to how we discussed it earlier. There is a pitfall with negative limited data rate appearance during backpropagation (check ScalingExecutorBackpropagationTest#testDisconnectedJobMetricsUpdate) because of "even" backpropagation of data rate to upstream operators, so it is not still very clear how to resolve such situations. Also I decided to replace bottleneck-propagation.allow-scale-down option with processing.rate.backpropagation.impact, because I think it is very unclear how use the previous one properly. The new option represents a "weight" of backpropagation: 0 means backpropagation is ignored and 1.0 means new values will be used further. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
