[
https://issues.apache.org/jira/browse/FLINK-29501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17613586#comment-17613586
]
Maximilian Michels commented on FLINK-29501:
--------------------------------------------
It's definitely an option to use the "rescaling" API:
[https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jobs-jobid-rescaling]
I think the rational behind this Jira is to have an immutable way to deploy a
rescaled version of a job. With the proposed changes, one simply alters the k8s
deployment spec with the updated parallelisms and the job redeploys. With the
adaptive scheduler, there is a whole level of indirection and complexity. The
upside may be shorter downtime. The downside clearly is the additional rest
call which is, as of now, is not reflected in the k8s deployment. I think it is
a bit hostile to k8s users to not allow rescale via some form of configuration.
Yes, the k8s operator could be modified to implement the rescale call. We would
have to allow additional jobvertex overrides as the endpoint currently only
supports setting the default parallelism. Currently this feature is intended
for streaming use cases only which work with the adaptive scheduler.
> Allow overriding JobVertex parallelisms during job submission
> -------------------------------------------------------------
>
> Key: FLINK-29501
> URL: https://issues.apache.org/jira/browse/FLINK-29501
> Project: Flink
> Issue Type: New Feature
> Components: Runtime / Configuration, Runtime / REST
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Minor
> Labels: pull-request-available
>
> It is a common scenario that users want to make changes to the parallelisms
> in the JobGraph. For example, because they discover that the job needs more
> or less resources. There is the option to do this globally via the job
> parallelism. However, for fine-tuned jobs jobs with potentially many
> branches, tuning on the job vertex level is required.
> This is to propose a way such that users can apply a mapping \{{jobVertexId
> => parallelism}} before the job is submitted without having to modify the
> JobGraph manually.
> One way to achieving this would be to add an optional map field to the Rest
> API jobs endpoint. However, in deployment modes like the application mode,
> this might not make sense because users do not have control the rest endpoint.
> Similarly to how other job parameters are passed in the application mode, we
> propose to add the overrides as a configuration parameter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
