[jira] [Commented] (YARN-2180) In-memory backing store for cache manager
[ https://issues.apache.org/jira/browse/YARN-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167039#comment-14167039 ] Sangjin Lee commented on YARN-2180: --- Thanks [~kasha]! > In-memory backing store for cache manager > - > > Key: YARN-2180 > URL: https://issues.apache.org/jira/browse/YARN-2180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Fix For: 2.7.0 > > Attachments: YARN-2180-trunk-v1.patch, YARN-2180-trunk-v2.patch, > YARN-2180-trunk-v3.patch, YARN-2180-trunk-v4.patch, YARN-2180-trunk-v5.patch, > YARN-2180-trunk-v6.patch, YARN-2180-trunk-v7.patch > > > Implement an in-memory backing store for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2180) In-memory backing store for cache manager
[ https://issues.apache.org/jira/browse/YARN-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166954#comment-14166954 ] Hudson commented on YARN-2180: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1922 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1922/]) YARN-2180. [YARN-1492] In-memory backing store for cache manager. (Chris Trezzo via kasha) (kasha: rev 4f426fe2232ed90d8fdf8619fbdeae28d788b5c8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/TestInMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/SharedCacheManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SharedCacheResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SharedCacheResourceReference.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/InMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/sharedcache/SharedCacheUtil.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/sharedcache/SharedCacheStructureUtil.java > In-memory backing store for cache manager > - > > Key: YARN-2180 > URL: https://issues.apache.org/jira/browse/YARN-2180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Fix For: 2.7.0 > > Attachments: YARN-2180-trunk-v1.patch, YARN-2180-trunk-v2.patch, > YARN-2180-trunk-v3.patch, YARN-2180-trunk-v4.patch, YARN-2180-trunk-v5.patch, > YARN-2180-trunk-v6.patch, YARN-2180-trunk-v7.patch > > > Implement an in-memory backing store for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2180) In-memory backing store for cache manager
[ https://issues.apache.org/jira/browse/YARN-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166865#comment-14166865 ] Hudson commented on YARN-2180: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1897 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1897/]) YARN-2180. [YARN-1492] In-memory backing store for cache manager. (Chris Trezzo via kasha) (kasha: rev 4f426fe2232ed90d8fdf8619fbdeae28d788b5c8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/sharedcache/SharedCacheStructureUtil.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/InMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/sharedcache/SharedCacheUtil.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/TestInMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SharedCacheResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SharedCacheResourceReference.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/SharedCacheManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml > In-memory backing store for cache manager > - > > Key: YARN-2180 > URL: https://issues.apache.org/jira/browse/YARN-2180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Fix For: 2.7.0 > > Attachments: YARN-2180-trunk-v1.patch, YARN-2180-trunk-v2.patch, > YARN-2180-trunk-v3.patch, YARN-2180-trunk-v4.patch, YARN-2180-trunk-v5.patch, > YARN-2180-trunk-v6.patch, YARN-2180-trunk-v7.patch > > > Implement an in-memory backing store for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2180) In-memory backing store for cache manager
[ https://issues.apache.org/jira/browse/YARN-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166672#comment-14166672 ] Hudson commented on YARN-2180: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #707 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/707/]) YARN-2180. [YARN-1492] In-memory backing store for cache manager. (Chris Trezzo via kasha) (kasha: rev 4f426fe2232ed90d8fdf8619fbdeae28d788b5c8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SharedCacheResourceReference.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/InMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SharedCacheResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/sharedcache/SharedCacheUtil.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/TestInMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/SharedCacheManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/sharedcache/SharedCacheStructureUtil.java > In-memory backing store for cache manager > - > > Key: YARN-2180 > URL: https://issues.apache.org/jira/browse/YARN-2180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Fix For: 2.7.0 > > Attachments: YARN-2180-trunk-v1.patch, YARN-2180-trunk-v2.patch, > YARN-2180-trunk-v3.patch, YARN-2180-trunk-v4.patch, YARN-2180-trunk-v5.patch, > YARN-2180-trunk-v6.patch, YARN-2180-trunk-v7.patch > > > Implement an in-memory backing store for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2180) In-memory backing store for cache manager
[ https://issues.apache.org/jira/browse/YARN-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166048#comment-14166048 ] Hudson commented on YARN-2180: -- FAILURE: Integrated in Hadoop-trunk-Commit #6229 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6229/]) YARN-2180. [YARN-1492] In-memory backing store for cache manager. (Chris Trezzo via kasha) (kasha: rev 4f426fe2232ed90d8fdf8619fbdeae28d788b5c8) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SharedCacheResource.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/InMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/sharedcache/SharedCacheStructureUtil.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/test/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/TestInMemorySCMStore.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/SharedCacheManager.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/pom.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/sharedcache/SharedCacheUtil.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager/src/main/java/org/apache/hadoop/yarn/server/sharedcachemanager/store/SharedCacheResourceReference.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml > In-memory backing store for cache manager > - > > Key: YARN-2180 > URL: https://issues.apache.org/jira/browse/YARN-2180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Fix For: 2.7.0 > > Attachments: YARN-2180-trunk-v1.patch, YARN-2180-trunk-v2.patch, > YARN-2180-trunk-v3.patch, YARN-2180-trunk-v4.patch, YARN-2180-trunk-v5.patch, > YARN-2180-trunk-v6.patch, YARN-2180-trunk-v7.patch > > > Implement an in-memory backing store for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2180) In-memory backing store for cache manager
[ https://issues.apache.org/jira/browse/YARN-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163456#comment-14163456 ] Karthik Kambatla commented on YARN-2180: +1 > In-memory backing store for cache manager > - > > Key: YARN-2180 > URL: https://issues.apache.org/jira/browse/YARN-2180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: YARN-2180-trunk-v1.patch, YARN-2180-trunk-v2.patch, > YARN-2180-trunk-v3.patch, YARN-2180-trunk-v4.patch, YARN-2180-trunk-v5.patch, > YARN-2180-trunk-v6.patch, YARN-2180-trunk-v7.patch > > > Implement an in-memory backing store for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2180) In-memory backing store for cache manager
[ https://issues.apache.org/jira/browse/YARN-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14163088#comment-14163088 ] Hadoop QA commented on YARN-2180: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12673521/YARN-2180-trunk-v7.patch against trunk revision 1efd9c9. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5321//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5321//console This message is automatically generated. > In-memory backing store for cache manager > - > > Key: YARN-2180 > URL: https://issues.apache.org/jira/browse/YARN-2180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: YARN-2180-trunk-v1.patch, YARN-2180-trunk-v2.patch, > YARN-2180-trunk-v3.patch, YARN-2180-trunk-v4.patch, YARN-2180-trunk-v5.patch, > YARN-2180-trunk-v6.patch, YARN-2180-trunk-v7.patch > > > Implement an in-memory backing store for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2180) In-memory backing store for cache manager
[ https://issues.apache.org/jira/browse/YARN-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156905#comment-14156905 ] Karthik Kambatla commented on YARN-2180: Looks mostly good, but for these minor comments: # App-checker and the store implementations aren't related: ## the app-checker config should be appended to SHARED_CACHE_PREFIX and IN_MEMORY_STORE ## the variable names should be updated accordingly. ## InMemorySCMStore#createAppCheckerService should move to a util class - how about changing SharedCacheStructureUtil to SharedCacheUtil and adding this method there? # Can we create a follow-up blocker sub-task to revisit all the config names before we include sharedcache work in a release? > In-memory backing store for cache manager > - > > Key: YARN-2180 > URL: https://issues.apache.org/jira/browse/YARN-2180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: YARN-2180-trunk-v1.patch, YARN-2180-trunk-v2.patch, > YARN-2180-trunk-v3.patch, YARN-2180-trunk-v4.patch, YARN-2180-trunk-v5.patch, > YARN-2180-trunk-v6.patch > > > Implement an in-memory backing store for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2180) In-memory backing store for cache manager
[ https://issues.apache.org/jira/browse/YARN-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154934#comment-14154934 ] Hadoop QA commented on YARN-2180: - {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12672206/YARN-2180-trunk-v6.patch against trunk revision 17d1202. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-sharedcachemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5195//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5195//console This message is automatically generated. > In-memory backing store for cache manager > - > > Key: YARN-2180 > URL: https://issues.apache.org/jira/browse/YARN-2180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: YARN-2180-trunk-v1.patch, YARN-2180-trunk-v2.patch, > YARN-2180-trunk-v3.patch, YARN-2180-trunk-v4.patch, YARN-2180-trunk-v5.patch, > YARN-2180-trunk-v6.patch > > > Implement an in-memory backing store for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2180) In-memory backing store for cache manager
[ https://issues.apache.org/jira/browse/YARN-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154060#comment-14154060 ] Chris Trezzo commented on YARN-2180: I also removed the clearCache() method from SCMStore and InMemorySCMStore. > In-memory backing store for cache manager > - > > Key: YARN-2180 > URL: https://issues.apache.org/jira/browse/YARN-2180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: YARN-2180-trunk-v1.patch, YARN-2180-trunk-v2.patch, > YARN-2180-trunk-v3.patch, YARN-2180-trunk-v4.patch, YARN-2180-trunk-v5.patch, > YARN-2180-trunk-v6.patch > > > Implement an in-memory backing store for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2180) In-memory backing store for cache manager
[ https://issues.apache.org/jira/browse/YARN-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14150328#comment-14150328 ] Karthik Kambatla commented on YARN-2180: Forgot to mention - we should annotate new classes as Private- Evolving|Unstable as appropriate. > In-memory backing store for cache manager > - > > Key: YARN-2180 > URL: https://issues.apache.org/jira/browse/YARN-2180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: YARN-2180-trunk-v1.patch, YARN-2180-trunk-v2.patch, > YARN-2180-trunk-v3.patch, YARN-2180-trunk-v4.patch, YARN-2180-trunk-v5.patch > > > Implement an in-memory backing store for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2180) In-memory backing store for cache manager
[ https://issues.apache.org/jira/browse/YARN-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14150324#comment-14150324 ] Karthik Kambatla commented on YARN-2180: Thanks for the updates, Chris. The overall approach looks good. Review comments: # SharedCacheManager#createSCMStoreService should use ReflectionUtils.newInstance. RMProxy is an example. # Thinking out loud here. YarnConfiguration (and yarn-default): I was wondering if we need a separate prefix for "manager". Do we have more configs coming later specific to manager? yarn.sharedcache.store is not ambiguous. # SCMStore ## A couple of lines longer than 80 chars. ## For resources that are not in the store, isn't the access time trivially zero? I am okay with returning -1 for those cases, but will returning zero help at call sites? ## Nit: Would keep the methods concerning references all together. # InMemorySCMStore configuration - do we need a separate configuration class for in-memory-store? Why not include it in YarnConfiguration similar to RMStore implementations? According to what we decide on, we might want to change the actual config names. # InMemorySCMStore ## Can we rename "map" to something more descriptive? cacheResources? ## Nit: Move bootstrapping code to a different method for readability? ## Isn't the following synchronized block prone to races when different threads lock on different objects? {code} synchronized (initialApps) { initialApps = getInitialApps(conf); } {code} ## We can leave it as is for now, but the implementation of AppChecker should come from some util method based on whether it is embedded or not. If we are open to it, we can add that method now an make it return RemoteAppChecker by default. ## Nit: The following should fit on two lines: {code} Map getInitialCachedResources(FileSystem fs, Configuration conf) throws IOException { {code} ## Use containsKey instead of the following? {code} String mapped = initialCachedEntries.get(key); if (mapped != null) { {code} ## clearCache() - we should annotate each TODO with a follow-up JIRA, so we don't forget. > In-memory backing store for cache manager > - > > Key: YARN-2180 > URL: https://issues.apache.org/jira/browse/YARN-2180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: YARN-2180-trunk-v1.patch, YARN-2180-trunk-v2.patch, > YARN-2180-trunk-v3.patch, YARN-2180-trunk-v4.patch, YARN-2180-trunk-v5.patch > > > Implement an in-memory backing store for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2180) In-memory backing store for cache manager
[ https://issues.apache.org/jira/browse/YARN-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142228#comment-14142228 ] Sangjin Lee commented on YARN-2180: --- bq. Synchronization is missing from the InMemoryStore operations? You are using a ConcurrentHashMap but to insert correctly (multiple apps adding a cache-entry to the same path) you'll need to use putIfAbsent? Surprised the test is presumably passing. I can answer this question as I wrote the in-memory store. :) Actually all operations (both access and mutation) on {{map}} are synchronized on the interned key. Since keys are unique, there can be no concurrent operations operating on the same key when synchronized on the key. Therefore, {{putIfAbsent()}} is not necessary (as it is needed if there are concurrent operations on the same key). There can be concurrent operations on the map on *different* keys, and that thread-safety is addressed by the {{ConcurrentHashMap}}. There are a couple of exceptions, and those are {{bootstrap()}} and {{clearCache()}}. The {{bootstrap()}} method is an exception because it operates on the map before it accepts any reads/writes. The {{clearCache()}} method is provided only for test purposes. Hope this helps. > In-memory backing store for cache manager > - > > Key: YARN-2180 > URL: https://issues.apache.org/jira/browse/YARN-2180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: YARN-2180-trunk-v1.patch, YARN-2180-trunk-v2.patch, > YARN-2180-trunk-v3.patch > > > Implement an in-memory backing store for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2180) In-memory backing store for cache manager
[ https://issues.apache.org/jira/browse/YARN-2180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142167#comment-14142167 ] Vinod Kumar Vavilapalli commented on YARN-2180: --- The patch looks fine overall, some comments - yarn.sharedcache.manager.store.impl -> yarn.sharedcache.store or store-class - YarnConfiguration.SCM_STORE_IMPL -> SCM_STORE/SCM_STORE_CLASS - We already have Resource, LocalResource. To avoid confuse, shall we use SharedCacheResource and hence ResourceReference -> SharedCacheResourceReference and so on everywhere? - InMemoryStore.bootstrap() can be done as part of serviceInit() - Synchronization is missing from the InMemoryStore operations? You are using a ConcurrentHashMap but to insert correctly (multiple apps adding a cache-entry to the same path) you'll need to use {{putIfAbsent}}? Surprised the test is presumably passing. > In-memory backing store for cache manager > - > > Key: YARN-2180 > URL: https://issues.apache.org/jira/browse/YARN-2180 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Chris Trezzo >Assignee: Chris Trezzo > Attachments: YARN-2180-trunk-v1.patch, YARN-2180-trunk-v2.patch, > YARN-2180-trunk-v3.patch > > > Implement an in-memory backing store for the cache manager. -- This message was sent by Atlassian JIRA (v6.3.4#6332)