John Omernik created MESOS-1711:
-----------------------------------
Summary: Create method for users to identify HDFS compatible
protocols in fetcher.cpp
Key: MESOS-1711
URL: https://issues.apache.org/jira/browse/MESOS-1711
Project: Mesos
Issue Type: Improvement
Components: general
Affects Versions: 0.19.1
Environment: All
Reporter: John Omernik
Priority: Minor
Fix For: 0.21.0
In fetcher.cpp, the code to get the Mesos packages uses a hard coded list of
protocols to determine if the Hadoop copytoLocal method is used or if another
method (such as standard filecopy). This limits the addition of new protocols
that are HDFS compatible until the next release of Mesos. Tachyon Filesystem
(tachyonfs://), MapR FS (maprfs://) and glusterfs:// are three examples that
could make use of this.
Instead of just adding those file systems in the hard coded list, I recommend
following the lead of the Tachyon Project. In tachyon-0.6.0-SNAPSHOT, they
have added an environment variable of allowed hdfs compatible protocols. This
comma-separated list allows the user/admin to specify which protocols are HDFS
compatible, without hard coding it in the fetcher.cpp.
I don't have access to the Tachyon issues list for linking, but the code is on
line 75 of
https://github.com/amplab/tachyon/blob/master/core/src/main/java/tachyon/UnderFileSystem.java
--
This message was sent by Atlassian JIRA
(v6.2#6252)