[
https://issues.apache.org/jira/browse/FLINK-5129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17472435#comment-17472435
]
lupan commented on FLINK-5129:
------------------------------
Does blob.storage.directory already support AWS S3 ？
When I use the following configuration:
{code:java}
blob.storage.directory: s3://iceberg-bucket/flink/blob {code}
I get the following error：
{code:java}
taskmanager | 2022-01-11 02:41:11,460 ERROR
org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Terminating
TaskManagerRunner with exit code 1.
taskmanager | org.apache.flink.util.FlinkException: Failed to start the
TaskManagerRunner.
taskmanager | at
org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:374)
~[flink-dist_2.12-1.13.3.jar:1.13.3]
taskmanager | at
org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$runTaskManagerProcessSecurely$3(TaskManagerRunner.java:413)
~[flink-dist_2.12-1.13.3.jar:1.13.3]
taskmanager | at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
~[flink-dist_2.12-1.13.3.jar:1.13.3]
taskmanager | at
org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerProcessSecurely(TaskManagerRunner.java:413)
[flink-dist_2.12-1.13.3.jar:1.13.3]
taskmanager | at
org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerProcessSecurely(TaskManagerRunner.java:396)
[flink-dist_2.12-1.13.3.jar:1.13.3]
taskmanager | at
org.apache.flink.runtime.taskexecutor.TaskManagerRunner.main(TaskManagerRunner.java:354)
[flink-dist_2.12-1.13.3.jar:1.13.3]
taskmanager | Caused by: java.io.IOException: Could not create storage
directory for BLOB store in 's3:/iceberg-bucket/flink/blob'.
taskmanager | at
org.apache.flink.runtime.blob.BlobUtils.initLocalStorageDirectory(BlobUtils.java:139)
~[flink-dist_2.12-1.13.3.jar:1.13.3]
taskmanager | at
org.apache.flink.runtime.blob.AbstractBlobCache.<init>(AbstractBlobCache.java:89)
~[flink-dist_2.12-1.13.3.jar:1.13.3]
taskmanager | at
org.apache.flink.runtime.blob.PermanentBlobCache.<init>(PermanentBlobCache.java:93)
~[flink-dist_2.12-1.13.3.jar:1.13.3]
taskmanager | at
org.apache.flink.runtime.blob.BlobCacheService.<init>(BlobCacheService.java:55)
~[flink-dist_2.12-1.13.3.jar:1.13.3]
taskmanager | at
org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<init>(TaskManagerRunner.java:169)
~[flink-dist_2.12-1.13.3.jar:1.13.3]
taskmanager | at
org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:367)
~[flink-dist_2.12-1.13.3.jar:1.13.3]
taskmanager | ... 5 more{code}
> make the BlobServer use a distributed file system
> -------------------------------------------------
>
> Key: FLINK-5129
> URL: https://issues.apache.org/jira/browse/FLINK-5129
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Network
> Reporter: Nico Kruber
> Assignee: Nico Kruber
> Priority: Major
> Fix For: 1.3.0
>
> Attachments: image-2022-01-11-11-27-59-280.png
>
>
> Currently, the BlobServer uses a local storage and, in addition when the HA
> mode is set, a distributed file system, e.g. hdfs. This, however, is only
> used by the JobManager and all TaskManager instances request blobs from the
> JobManager. By using the distributed file system there as well, we would
> lower the load on the JobManager and increase scalability.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
