[
https://issues.apache.org/jira/browse/FLINK-8656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364090#comment-16364090
]
ASF GitHub Bot commented on FLINK-8656:
---------------------------------------
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/5487
[FLINK-8656] [flip6] Add modify CLI command to rescale Flink jobs
## What is the purpose of the change
Jobs can now be rescaled by calling flink modify <JOB_ID> -p <PARALLELISM>.
Internally, the CliFrontend will send the corresponding REST call and poll
for status updates.
This PR is based on #5454.
## Brief change log
- Add `modify` call to `CliFrontend`
- Add `ClusterClient#rescaleJob` method with default implementation
- Implement `RestClusterClient#rescalJob` method to trigger asynchronous
rescale operation via REST and poll for its status updates
## Verifying this change
- Tested manually
- Added `CliFrontendModifyTest`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (JavaDocs + stdout help)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink rescaleCommand
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5487.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5487
----
commit d9159228091cae9ebbd1bb718b69e6cf452881e1
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-13T11:41:44Z
[FLINK-8643] [flip6] Use JobManagerOptions#SLOT_REQUEST_TIMEOUT in
ExecutionGraph
This commit changes the initialization of the ExecutionGraph to use the
JobManagerOptions#SLOT_REQUEST_TIMEOUT for the slot allocation. Furthermore,
it changes the behaviour of the SlotPool#ProviderAndOwner implementation
such
that the timeout is given to it via the SlotProvider#allocateSlot call.
commit 19780c9d284914ec51e92231536315299a3c2da3
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-13T12:18:01Z
[hotfix] [flip6] Remove unnecessary timeout from SlotPool
commit 9924776c92a378cef144c0767f1ff18b799d52e9
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-13T14:33:11Z
[FLINK-8647] [flip6] Introduce JobMasterConfiguration
This commit introduces a JobMasterConfiguration which contains JobMaster
specific
configuration settings.
commit fde75841de2e27cb7380f3a28066a99e2c1a690d
Author: zentol <chesnay@...>
Date: 2018-01-23T12:50:32Z
[FLINK-8475][config][docs] Integrate HA-ZK options
This closes #5462.
commit 788a17fdbd4aaf3429ead4491ede197fc775b1f0
Author: zentol <chesnay@...>
Date: 2018-01-23T13:04:36Z
[FLINK-8475][config][docs] Integrate YARN options
This closes #5463.
commit fcd783358c282e61bf12e0c18298c237c85a6695
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-13T15:08:38Z
[hotfix] [tests] Simplify JobMasterTest
commit 8206e6f13809c0b60bfaf776bc386088f535e723
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-13T15:10:09Z
[FLINK-8546] [flip6] Respect savepoints and restore from latest checkpoints
Let the JobMaster respect checkpoints and savepoints. The JobMaster will
always
try to restore the latest checkpoint if there is one available. Next it
will check
whether savepoint restore settings have been set. If so, then it will try
to restore
the savepoint. Only if these settings are not set, the job will be started
from
scratch.
commit 057a95b7328b1cca7b78bf1dd25e8d048df70410
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-13T15:11:37Z
[hotfix] Fix checkstyle violations in ExecutionGraph
commit 9930b0991320bcff268ca82db6378df8976560dc
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-13T15:12:41Z
[FLINK-8627] Introduce new JobStatus#SUSPENDING to ExecutionGraph
The new JobStatus#SUSPENDING says that an ExecutionGraph has been suspended
but its
clean up has not been done yet. Only after all Executions have been
canceled, the
ExecutionGraph will enter the SUSPENDED state and complete the termination
future
accordingly.
commit 6c51ad306c90464572353168ecafdb962794747e
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-13T15:14:41Z
[FLINK-8629] [flip6] Allow JobMaster to rescale jobs
This commit adds the functionality to rescale a job or parts of it to
the JobMaster. In order to rescale a job, the JobMaster does the following:
1. Take a savepoint
2. Create a rescaled ExecutionGraph from the JobGraph
3. Initialize it with the taken savepoint
4. Suspend the old ExecutionGraph
5. Restart the new ExecutionGraph once the old ExecutionGraph has been
suspended
commit 990aa95040264ffd3c3ab20638cb8212af68d155
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-13T15:34:31Z
[FLINK-8633] [flip6] Expose rescaling of jobs via the Dispatcher
This commit exposes the JobMaster#rescaleJob via the Dispatcher. This will
allow it to call this functionality from a REST handler.
commit ca85f392c0caea751bf7805beb4428eba5cf851f
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-03T16:41:51Z
[FLINK-8631] [rest] Add support for generic types to the RestClient
This commit allows the Restclient to receive generic response types. In
order
to do this, the MessageHeaders contain now information about the generic
type parameters of the response type.
commit 381099612b78ad1ab7bc5f239d2fd7c618c74222
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-02T10:06:35Z
[FLINK-8634] [rest] Introduce job rescaling REST handler
Add rescaling REST handler as a sub class of the
AbstractAsynchronousOperationHandlers.
commit f4e9f71b4821dabfadc47491e3b4a67ef5e3dab3
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-02T10:23:00Z
[FLINK-8632] [flip6] Introduce generalized asynchronous operation handlers
The asynchronous operation handlers are the generalization of the
SavepointHandlers.
They consist of a Trigger- and a StatusHandler. The TriggerHandler is used
to trigger
an asynchronous operation. The handler stores the operation future and
returns a
trigger id. The trigger id can be used to query the status of the operation
via the
StatusHandler. Once the operation has completed, the StatusHandler will
return the
result.
commit 79f20916a1366fed9caee27c6fcfe932d99c7482
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-11T18:50:46Z
[FLINK-8635] [rest] Register rescaling handlers at web endpoint
commit 7081cba4cd3ba377b64a4d165e1fc888be99873f
Author: Till Rohrmann <trohrmann@...>
Date: 2018-02-13T16:29:32Z
[FLINK-8656] [flip6] Add modify CLI command to rescale Flink jobs
Jobs can now be rescaled by calling flink modify <JOB_ID> -p <PARALLELISM>.
Internally, the CliFrontend will send the corresponding REST call and poll
for status updates.
----
> Add CLI command for rescaling
> -----------------------------
>
> Key: FLINK-8656
> URL: https://issues.apache.org/jira/browse/FLINK-8656
> Project: Flink
> Issue Type: New Feature
> Components: Client
> Affects Versions: 1.5.0
> Reporter: Till Rohrmann
> Assignee: Till Rohrmann
> Priority: Major
> Labels: flip-6
> Fix For: 1.5.0
>
>
> The REST rescaling calls should be made accessible via the {{CliFrontend}}.
> In order to do that I propose to add a {{modify}} command to the
> {{CliFrontend}} to which we can pass a new parallelism.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
