[
https://issues.apache.org/jira/browse/FLINK-9366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16509515#comment-16509515
]
ASF GitHub Bot commented on FLINK-9366:
---------------------------------------
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/6107#discussion_r194705137
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobGraph.java ---
@@ -584,17 +584,25 @@ public void uploadUserArtifacts(InetSocketAddress
blobServerAddress, Configurati
if (!userArtifacts.isEmpty()) {
try (BlobClient blobClient = new
BlobClient(blobServerAddress, clientConfig)) {
for (Map.Entry<String,
DistributedCache.DistributedCacheEntry> userArtifact :
userArtifacts.entrySet()) {
-
- final PermanentBlobKey key =
blobClient.uploadFile(jobID,
- new
Path(userArtifact.getValue().filePath));
-
- DistributedCache.writeFileInfoToConfig(
- userArtifact.getKey(),
- new
DistributedCache.DistributedCacheEntry(
-
userArtifact.getValue().filePath,
-
userArtifact.getValue().isExecutable,
-
InstantiationUtil.serializeObject(key)),
- jobConfiguration);
+ Path filePath = new
Path(userArtifact.getValue().filePath);
+
+ if
(filePath.getFileSystem().isDistributedFS()) {
--- End diff --
The file-system class may not be accessible on the client, so you have to
guard this with a try-catch block and write it into the config in both cases.
> Distribute Cache only works for client-accessible files
> -------------------------------------------------------
>
> Key: FLINK-9366
> URL: https://issues.apache.org/jira/browse/FLINK-9366
> Project: Flink
> Issue Type: Bug
> Components: Client, Local Runtime
> Affects Versions: 1.6.0
> Reporter: Chesnay Schepler
> Assignee: Dawid Wysakowicz
> Priority: Blocker
> Fix For: 1.6.0
>
>
> In FLINK-8620 the distributed cache was modified to the distribute files via
> the blob store, instead of downloading them from a distributed filesystem.
> Previously, taskmanagers would download requested files from the DFS. Now,
> they retrieve it form the blob store. This requires the client to
> preemptively upload all files used with distributed cache.
> As a result it is no longer possible to use the distributed cache for files
> that reside in a cluster-internal DFS, as the client cannot download it. This
> is a regression from the previous behavior and may break existing setups.
> [~aljoscha] [~dawidwys]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
