[
https://issues.apache.org/jira/browse/FLINK-9623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16518112#comment-16518112
]
ASF GitHub Bot commented on FLINK-9623:
---------------------------------------
GitHub user zentol opened a pull request:
https://github.com/apache/flink/pull/6187
FLINK-9623][runtime] Move zipping logic out of blobservice
## What is the purpose of the change
This PR moves the zipping logic for local user directory artifacts out of
the blobservice.
Instead, directories are explicitly zipped by the `JobGraphGenerators` when
compiling the graph.
The zipping code was generalized and moved from `FileCache` and
`BlobClient` to `FileUtils`.
Additionally, `JobGraphTest` now extends `TestLogger`.
## Brief change log
* move zip compression/expansion as general utility methods to `FileUtils`
* modify `[Streaming]JobGraphGenerator` to zip local user directory
artifacts before registering them with the graph
* adjust `FileCacheDirectoryTest` to not manually create a mock zip file
* extend documentation of `FileCache`
* extend documentation of `DistributedCache`
* remove setting of user-artifacts in `TestStreamEnvironment` since it is
redundant
* remove odd verification/normalization code in `Plan#registerCachedFile`,
changes are covered by JobGraphGenerators
## Verifying this change
* zip compression/expansion is covered by new tests in `FileUtils`
* handling of local user directory artifacts by JobGraphGenerators covered
by new tests in `JobGraphGeneratorTest`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zentol/flink 9280_alpha
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/6187.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #6187
----
commit a8552a5fc85c4a3122eecd7739f5a02065dc93cd
Author: zentol <chesnay@...>
Date: 2018-06-13T14:10:16Z
[hotfix][tests] JobGraphTest extends TestLogger
commit 483ff525a7c047360a5c044fbb3c6bffb08a10c5
Author: zentol <chesnay@...>
Date: 2018-06-04T11:50:44Z
FLINK-9623][runtime] Move zipping logic out of blobservice
----
> Move zipping logic out of blobservice
> -------------------------------------
>
> Key: FLINK-9623
> URL: https://issues.apache.org/jira/browse/FLINK-9623
> Project: Flink
> Issue Type: Improvement
> Components: Job-Submission
> Affects Versions: 1.6.0
> Reporter: Chesnay Schepler
> Assignee: Chesnay Schepler
> Priority: Major
> Fix For: 1.6.0
>
>
> Directories given to the blob-service (primarily a use-case for the
> distributed cache) are currently silently zipped, and later unzipped by the
> {{FileCache}}. This tightly coupled the zipping logic in the blob-service to
> the unzipping logic of the {{FileCache}}. The blob-service neither unzipped
> the directory if the blob was requested, nor did it provide any means of
> doing so manually, nor did it inform the user as to whether the requested
> blob is a zip or not.
> My conclusion in is that the blob-service should not support directories _for
> now_, and that instead directories for the {{distributed cache}} should be
> explicitly zipped beforehand, given that this is the only use-case we have at
> the moment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
