GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/5353
[FLINK-8503] [flip6] Display TaskExecutor logs and stdout files in web ui
## What is the purpose of the change
Introduce the AbstractHandler which takes a typed request and returns an
untyped
response. The AbstractRestHandler extends the AbstractHandler to add typed
reponses.
Introduce AbstractTaskManagerFileHandler which encapsulates the file
loading logic.
Upon request of a TaskManager file, the handler will trigger the file
upload via
the ResourceManager. The returned TransientBlobKey is then downloaded via
the
TransientBlobService. Once downloaded, the file is served to the client.
Each
transient blob key is cached for maximum duration after which it is purged
and has
to be reuploaded by the TaskExecutor.
This PR is based on #5341
## Brief change log
- Introduced untyped response handler `AbstractHandler`
- Added `AbstractTaskManagerFileHandler` which is responsible for serving
files from the `TaskExecutor`
- The `AbstractTaskManagerFileHandler` triggers the file upload via the
`ResourceManager` which knows the `TaskExecutors`, additionally it caches the
`TransientBlobKeys` in order to not always trigger a file upload
- Added `TaskManagerLogFileHandler` to serve the log file
- Added `TaskManagerStdoutFileHandler` to serve the stdout file
## Verifying this change
- Added `AbstractTaskManagerFileHandlerTest`
- Tested functionality manually
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink
enableTaskManagerLogRetrieval
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/5353.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5353
----
commit b9db2ab22c346ec64363b446ed23692af1365239
Author: Till Rohrmann <trohrmann@...>
Date: 2018-01-23T18:12:27Z
[FLINK-8501] [flip6] Use single BlobCacheService per TaskExecutor
Instead of creating for each new JobManagerConnection a dedicated
BlobCacheService
the TaskExecutor uses a single BlobCacheService which it shares between the
different JobManagerConnections. The initial BlobServer address is passed
by the
ResourceManager when the TaskExecutor registers at it. In order to avoid
the re-
creation of BlobCacheServices, this commit changes the behaviour such that
one can
update the BlobServer address.
commit 5bd4db619ff0e984477ead323345f5f7fa740626
Author: Till Rohrmann <trohrmann@...>
Date: 2018-01-24T12:41:53Z
[hotfix] [tests] Remove JobManagerRunnerMockTest
The JobManagerRunnerMockTest is completely ignored. Moreover, it tests
things with
heavy usage of Mockito which is hard to maintain.
commit c38adb65dd49150a03a2f5f7ea8d421b2ef34616
Author: Till Rohrmann <trohrmann@...>
Date: 2018-01-24T12:42:41Z
[FLINK-8502] [flip6] Remove LibraryCacheManager from JobMaster
This commit removes the LibraryCacheManager from the JobMaster since it is
no longer needed. The JobMaster is started with the correct user code class
loader and, thus, does not need the LibraryCacheManager.
This commit also corrects that the BlobServer is not closed by the
JobManagerServices#shutdown method.
commit eb3fed4ce96b4ce27f01d72f2dc76b151dbdd1ae
Author: Till Rohrmann <trohrmann@...>
Date: 2018-01-23T14:17:16Z
[FLINK-8495] [flip6] Enable main cluster component's log and stdout file
retrieval
This commit enables the log and stdout file retrieval of the cluster's main
component
via the web ui. This happens via the StaticFileServerHandler which serves
the log
and stdout file.
commit 23496241962c69ceb3ada9bd0d67c0344554cf99
Author: Till Rohrmann <trohrmann@...>
Date: 2018-01-23T17:27:28Z
[FLINK-8503] [flip6] Display TaskExecutor logs and stdout files in web ui
Introduce the AbstractHandler which takes a typed request and returns an
untyped
response. The AbstractRestHandler extends the AbstractHandler to add typed
reponses.
Introduce AbstractTaskManagerFileHandler which encapsulates the file
loading logic.
Upon request of a TaskManager file, the handler will trigger the file
upload via
the ResourceManager. The returned TransientBlobKey is then downloaded via
the
TransientBlobService. Once downloaded, the file is served to the client.
Each
transient blob key is cached for maximum duration after which it is purged
and has
to be reuploaded by the TaskExecutor.
----
---
