Gyula Fora created FLINK-30571:
----------------------------------
Summary: Compute scale parallelism based on observed scalability
Key: FLINK-30571
URL: https://issues.apache.org/jira/browse/FLINK-30571
Project: Flink
Issue Type: New Feature
Components: Kubernetes Operator
Reporter: Gyula Fora
Assignee: Gyula Fora
When computing target parallelism for job vertices we currently assume linear
scaling with a fixed (1) coefficient.
This assumes that in order to double the capacity we simply double the
parallelism.
While linearity already might be violated by many real time workloads this form
of strong linearity rarely holds due to the overhead of increased network
traffic, coordination etc.
As we can access past (parallelism, processingRate) information based on the
scaling history we should estimate the scalability coefficient either using a
simple or weighted linear regression.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
