Haohui Mai created YARN-6556:
--------------------------------
Summary: Implement a FileSystem that reads from HTTP
Key: YARN-6556
URL: https://issues.apache.org/jira/browse/YARN-6556
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
We have a use case where YARN applications would like to localize resources
from Artifactory. Putting the resources on HDFS itself might not be ideal as we
would like to leverage Artifactory to manage different versions of the
resources.
It would be nice to have something like {{HttpFileSystem}} that implements the
Hadoop filesystem API and reads from a HTTP endpoint.
Note that Samza has implemented the proposal by themselves:
https://github.com/apache/samza/blob/master/samza-yarn/src/main/scala/org/apache/samza/util/hadoop/HttpFileSystem.scala
The downside of this approach is that it requires the YARN cluster to put the
Samza jar into the classpath for each NM.
It would be much nicer for Hadoop to have this feature built-in.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
