[ 
https://issues.apache.org/jira/browse/FLINK-6020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001004#comment-16001004
 ] 

ASF GitHub Bot commented on FLINK-6020:
---------------------------------------

Github user StephanEwen commented on the issue:

    https://github.com/apache/flink/pull/3525
  
    @WangTaoTheTonic I think we can solve that the following way:
    
      - The local upload uses `ATOMIC_MOVE` to rename the file
      - Only the thread that succeeds will store the blob in HDFS or S3
    
    What do you think?


> Blob Server cannot handle multiple job submits (with same content) parallelly
> -----------------------------------------------------------------------------
>
>                 Key: FLINK-6020
>                 URL: https://issues.apache.org/jira/browse/FLINK-6020
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Distributed Coordination
>            Reporter: Tao Wang
>            Assignee: Tao Wang
>            Priority: Critical
>
> In yarn-cluster mode, if we submit one same job multiple times parallelly, 
> the task will encounter class load problem and lease occuputation.
> Because blob server stores user jars in name with generated sha1sum of those, 
> first writes a temp file and move it to finalialize. For recovery it also 
> will put them to HDFS with same file name.
> In same time, when multiple clients sumit same job with same jar, the local 
> jar files in blob server and those file on hdfs will be handled in multiple 
> threads(BlobServerConnection), and impact each other.
> It's better to have a way to handle this, now two ideas comes up to my head:
> 1. lock the write operation, or
> 2. use some unique identifier as file name instead of ( or added up to) 
> sha1sum of the file contents.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Reply via email to