John Omernik created MESOS-1711:
-----------------------------------

             Summary: Create method for users to identify HDFS compatible 
protocols in fetcher.cpp
                 Key: MESOS-1711
                 URL: https://issues.apache.org/jira/browse/MESOS-1711
             Project: Mesos
          Issue Type: Improvement
          Components: general
    Affects Versions: 0.19.1
         Environment: All
            Reporter: John Omernik
            Priority: Minor
             Fix For: 0.21.0


In fetcher.cpp, the code to get the Mesos packages uses a hard coded list of 
protocols to determine if the Hadoop copytoLocal method is used or if another 
method (such as standard filecopy).  This limits the addition of new protocols 
that are HDFS compatible until the next release of Mesos. Tachyon Filesystem 
(tachyonfs://), MapR FS (maprfs://) and glusterfs:// are three examples that 
could make use of this. 

Instead of just adding those file systems in the hard coded list, I recommend 
following the lead of the Tachyon Project.  In tachyon-0.6.0-SNAPSHOT, they 
have added an environment variable of allowed hdfs compatible protocols. This 
comma-separated list allows the user/admin to specify which protocols are HDFS 
compatible, without hard coding it in the fetcher.cpp.   

I don't have access to the Tachyon issues list for linking, but the code is on 
line 75 of 

https://github.com/amplab/tachyon/blob/master/core/src/main/java/tachyon/UnderFileSystem.java





--
This message was sent by Atlassian JIRA
(v6.2#6252)

Reply via email to