[ https://issues.apache.org/jira/browse/YARN-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412253#comment-16412253 ]
Vinod Kumar Vavilapalli commented on YARN-1151: ----------------------------------------------- >From the POC patch, it looks like NM will repeatedly download the tar again >and again on every restart, am I reading that right? We shouldn't be doing >that. May be we should get file-status, verify the checksum and skip if it is >the same. Exactly like dist-cache. These are some of the reasons why we should >simply reuse the core ResourceLocalizationService for localizing this. We should also figure out the right behavior if HDFS is down. What should NM do on a fresh start if HDFS is down? What it should do on a restart if HDFS is down? > Ability to configure auxiliary services from HDFS-based JAR files > ----------------------------------------------------------------- > > Key: YARN-1151 > URL: https://issues.apache.org/jira/browse/YARN-1151 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager > Affects Versions: 2.1.0-beta, 2.9.0 > Reporter: john lilley > Assignee: Xuan Gong > Priority: Major > Labels: auxiliary-service, yarn > Attachments: YARN-1151.1.patch, YARN-1151.branch-2.poc.patch, > [YARN-1151] [Design] Configure auxiliary services from HDFS-based JAR > files.pdf > > > I would like to install an auxiliary service in Hadoop YARN without actually > installing files/services on every node in the system. Discussions on the > user@ list indicate that this is not easily done. The reason we want an > auxiliary service is that our application has some persistent-data components > that are not appropriate for HDFS. In fact, they are somewhat analogous to > the mapper output of MapReduce's shuffle, which is what led me to > auxiliary-services in the first place. It would be much easier if we could > just place our service's JARs in HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org