[
https://issues.apache.org/jira/browse/FLINK-7668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180895#comment-16180895
]
ASF GitHub Bot commented on FLINK-7668:
---------------------------------------
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/4728
[FLINK-7668] Add ExecutionGraphCache for ExecutionGraph based REST handlers
## What is the purpose of the change
The ExecutionGraphCache replaces the ExecutionGraphHolder. Unlike the
latter, the former
does not expect the AccessExecutionGraph to be the true ExecutionGraph.
Instead it assumes
it to be the ArchivedExecutionGraph. Therefore, it invalidates the cache
entries after
a given time to live period. This will trigger requesting the
AccessExecutionGraph again
and, thus, updating the ExecutionGraph information for the ExecutionGraph
based REST
handlers.
In order to avoid memory leaks, the WebRuntimeMonitor starts now a periodic
cleanup task
which triggers ExecutionGraphCache.cleanup. This methods releases all cache
entries which
have exceeded their time to live. Currently it is set to 20 *
refreshInterval of the
web gui.
## Brief change log
- Rename `ExecutionGraphHolder` into `ExecutionGraphCache`
- Make `ExecutionGraphCache` thread safe by using a `ConcurrentHashMap`
- Add `ExecutionGraph` request logic depending on the time to live for each
cache entry
- Add periodic cleanup task to `WebRuntimeMonitor` which triggers
`ExecutionGraphCache#cleanup`
## Verifying this change
This change added tests and can be verified as follows:
- `ExecutionGraphCacheTest` adds multiple tests
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink refreshExecutionGraphHolder
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4728.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4728
----
commit ed2d5482c315d64216e4a645b68e7b48dfb89456
Author: Till Rohrmann <[email protected]>
Date: 2017-09-21T08:53:24Z
[FLINK-7650] [flip6] Port JobCancellationHandler to new REST endpoint
Let the JobCancellationHandler implement the LegacyRestHandler interface.
Moreover,
this commit adds the DELETE method to HttpMethodWrapper and the
RestServerEndpoint#registerHandler method.
commit e83065fa8102c10a10cdc981f0c7608577c20c34
Author: Till Rohrmann <[email protected]>
Date: 2017-09-21T14:47:18Z
[FLINK-7649] [flip6] Extend JobTerminationHandler to support stop
Rename the JobCancellationHandler into JobTerminationHandler which is now
responsible
for terminating jobs. Moreover, this commits adds two termination modes,
cancel and stop,
which are specified by providing a query parameter.
commit 02e345b8f34890d67230d0a75a818a8e3b65cf2b
Author: Till Rohrmann <[email protected]>
Date: 2017-09-22T11:31:12Z
[FLINK-7667] [flip6] Use ArchivedExecutionGraph as serializable
AccessExecutionGraph
This commit removes AccessExecutionGraph#getCheckpointCoordinator and
changes the
AccessExecutionGraph#getJobCheckpointSettings into
#getJobCheckpointConfiguration.
The JobCheckpointConfiguration only contains the CheckpointCoordinator
relevant
configuration settings and excludes the serialized state backend and the
serialized master hooks. That way we don't send unnecessary information when
the ArchivedExecutionGraph is requested.
commit ab4b21fdf0c00d531d04de6c6272c2812cb61e51
Author: Till Rohrmann <[email protected]>
Date: 2017-09-25T13:29:59Z
[FLINK-7668] Add ExecutionGraphCache for ExecutionGraph based REST handlers
The ExecutionGraphCache replaces the ExecutionGraphHolder. Unlike the
latter, the former
does not expect the AccessExecutionGraph to be the true ExecutionGraph.
Instead it assumes
it to be the ArchivedExecutionGraph. Therefore, it invalidates the cache
entries after
a given time to live period. This will trigger requesting the
AccessExecutionGraph again
and, thus, updating the ExecutionGraph information for the ExecutionGraph
based REST
handlers.
In order to avoid memory leaks, the WebRuntimeMonitor starts now a periodic
cleanup task
which triggers ExecutionGraphCache.cleanup. This methods releases all cache
entries which
have exceeded their time to live. Currently it is set to 20 *
refreshInterval of the
web gui.
----
> Add AccessExecutionGraph refresh interval to ExecutionGraphHolder
> -----------------------------------------------------------------
>
> Key: FLINK-7668
> URL: https://issues.apache.org/jira/browse/FLINK-7668
> Project: Flink
> Issue Type: Sub-task
> Components: REST
> Affects Versions: 1.4.0
> Reporter: Till Rohrmann
> Assignee: Till Rohrmann
> Labels: flip-6
>
> Once we support offline {{AccessExecutionGraph}} implementation (see
> FLINK-7667) we should add a refresh interval to the {{ExecutionGraphHolder}}
> after which the {{AccessExecutionGraph}} is retrieved again from the
> {{JobMaster}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
