[GitHub] [hadoop] jiwq commented on a change in pull request #1397: MAPREDUCE-7237. Supports config the shuffle's path cache related parameters

2020-03-04 Thread GitBox 
jiwq commented on a change in pull request #1397: MAPREDUCE-7237. Supports 
config the shuffle's path cache related parameters
URL: https://github.com/apache/hadoop/pull/1397#discussion_r387681223
 
 

 ##
 File path: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
 ##
 @@ -1266,6 +1266,29 @@
   
 
 
+
+  mapreduce.shuffle.pathcache.max-weight
+  10 * 1024 * 1024
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[GitHub] [hadoop] jiwq commented on a change in pull request #1397: MAPREDUCE-7237. Supports config the shuffle's path cache related parameters

2020-01-14 Thread GitBox 
jiwq commented on a change in pull request #1397: MAPREDUCE-7237. Supports 
config the shuffle's path cache related parameters
URL: https://github.com/apache/hadoop/pull/1397#discussion_r366278621
 
 

 ##
 File path: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
 ##
 @@ -836,63 +834,58 @@ public ChannelPipeline getPipeline() throws Exception {
   // TODO factor out encode/decode to permit binary shuffle
   // TODO factor out decode of index to permit alt. models
 }
-
   }
 
   class Shuffle extends SimpleChannelUpstreamHandler {
-private static final int MAX_WEIGHT = 10 * 1024 * 1024;
-private static final int EXPIRE_AFTER_ACCESS_MINUTES = 5;
-private static final int ALLOWED_CONCURRENCY = 16;
-private final Configuration conf;
+private static final String MAX_WEIGHT =
+"mapreduce.shuffle.pathcache.max-weight";
+private static final int DEFAULT_MAX_WEIGHT = 10 * 1024 * 1024;
+
+private static final String EXPIRE_AFTER_ACCESS_MINUTES =
+"mapreduce.shuffle.pathcache.expire-after-access-m";
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org