[
https://issues.apache.org/jira/browse/FLINK-19067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17266048#comment-17266048
]
Robert Metzger commented on FLINK-19067:
----------------------------------------
Thank you for your detailed analysis. I had a look at the code too, and I agree
that you can run into this situation.
Let's assume we have the following setup:
Node1
lead Dispatcher
BlobServer1
Node2
lead ResourceManager
BlobServer2
Node3
TaskManager
Job submissions will go to the Dispatcher on Node1. The JobSubmitHandler will
call DispatcherGateway.getBlobServerPort, which returns the address of
BlobServer1.
During job execution, the TaskManager on Node3 will execute a task from the
job. As part of the TaskExecutorRegistrationSuccess, we include the
ClusterInformation, which contains the BlobServer address. The
TaskExecutorRegistrationSuccess is coming from the ResourceManager, on Node2,
which returns it's local ClusterInformation, with the address of BlobServer2.
[~trohrmann] Could you take a quick look at this to verify or correct my
thinking?
> resource_manager and dispatcher register on different nodes in HA mode will
> cause FileNotFoundException
> -------------------------------------------------------------------------------------------------------
>
> Key: FLINK-19067
> URL: https://issues.apache.org/jira/browse/FLINK-19067
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.11.1
> Reporter: JieFang.He
> Priority: Major
> Attachments: flink-jobmanager-deployer-hejiefang01.log,
> flink-jobmanager-deployer-hejiefang02.log,
> flink-taskmanager-deployer-hejiefang01.log,
> flink-taskmanager-deployer-hejiefang02.log
>
>
> When run examples/batch/WordCount.jar，it will fail with the exception:
> {code:java}
> Caused by: java.io.FileNotFoundException:
> /data2/flink/storageDir/default/blob/job_d29414828f614d5466e239be4d3889ac/blob_p-a2ebe1c5aa160595f214b4bd0f39d80e42ee2e93-f458f1c12dc023e78d25f191de1d7c4b
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at
> org.apache.flink.core.fs.local.LocalDataInputStream.<init>(LocalDataInputStream.java:50)
> at
> org.apache.flink.core.fs.local.LocalFileSystem.open(LocalFileSystem.java:143)
> at
> org.apache.flink.runtime.blob.FileSystemBlobStore.get(FileSystemBlobStore.java:105)
> at
> org.apache.flink.runtime.blob.FileSystemBlobStore.get(FileSystemBlobStore.java:87)
> at
> org.apache.flink.runtime.blob.BlobServer.getFileInternal(BlobServer.java:501)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.get(BlobServerConnection.java:231)
> at
> org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:117)
> {code}
>
> I think the reason is that the jobFiles are upload to the dispatcher node，but
> the task get jobFiles from resource_manager node. So in HA mode, it need to
> ensure they are on one node
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
