ryantaocer created FLINK-12002:
----------------------------------
Summary: Adaptive Parallelism of Job Vertex Execution
Key: FLINK-12002
URL: https://issues.apache.org/jira/browse/FLINK-12002
Project: Flink
Issue Type: Improvement
Reporter: ryantaocer
Assignee: BoWang
In Flink the parallelism of job is a pre-specified parameter, which is usually
an empirical value and thus might not be optimal for both performance and
resource depending on the amount of data processed in each task.
Furthermore, a fixed parallelism cannot scale to varying data size common in
production cluster where we may not often change configurations.
We propose to determine the job parallelism adaptive to the actual total input
data size and an ideal data size processed by each task. The ideal size is
pre-specified according to the properties of the operator such as the
preparation overhead compared with data processing time.
detailed design doc coming soon.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
