[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740733#comment-13740733 ] Bikas Saha commented on MAPREDUCE-5311: --- Let me understand this a bit. There is existing yarn.min config. This is used by schedulers to do their internal logic. MR AM uses this same config. This is not optimal but has the virtue of being simple and relatively error free since there is only 1 config to change. If we change yarn.min to yarn.sched-name.min and introduce mr.min for MR then who will keep both of these in sync. Also the sync value depends on the scheduler being currently used. This is also sub optimal but is less preferable than the former because 1) new config 2) manual synch of 2 configs based on scheduler being used. If YARN-1004 doesnt touch yarn.min then we are good. Currently both schedulers are using the min values and will probably continue to use it for the foreseeable future. IMO yarn.max is not scheduler specific. Its an admin config that must be enforced by any scheduler. Lets just leave these configs as they are because removing them is causing more grief than gain (including this jira). And let MR use these configs to do internal calculations. I wish we had more foresight earlier and done the right thing for slot_millis when we wrote the original code. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13739629#comment-13739629 ] Arun C Murthy commented on MAPREDUCE-5311: -- I'm glad we are coming around to understand we can break existing users of SLOT_MILLIS. I wasn't thrilled about using configs and I'm even more leery of adding new configs to track existing configs in RM. This can lead to all sorts of configuration-error hell with mismatch in configs b/w YarnConfiguration and MRConf. I'm against getting into that sort of situations. A much simpler approach is to re-introduce a LimitedPrivate RegisterApplicationMasterResponse.getMinimumResourceCapability as described here (http://bit.ly/18uA8zj). This would revert some part of YARN-787. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13739970#comment-13739970 ] Alejandro Abdelnur commented on MAPREDUCE-5311: --- [~acmurthy], I'm glad you are glad, *smile*. Yes, we agree we need to leave SLOT_MILLIS because Hadoop 0.23 users are relying on it. We also agreed before that we should remove 'minimum resource capability' from the protocol and the API because it is a scheduler implementation internal thing that should not be exposed to the users. The current code relies on using 'minimum resource capability' configuration property which is internal. We've also agreed on doing that until this JIRA is fixed. The proposed solution is a tweak to the current code just using a different configuration property, nothing else. Adding 'minimum resource capability' back to the protocol and API to support deprecated functionality that we all agree it should go away does not seem right. I prefer [~jlowe] suggestion to have a new ( deprecated) configuration property that users upgrading from 0.23 and wanting to preserve the SLOT_MILLIS counter information can use (and if they don't -default setting- SLOT_MILLIS is always zero). Also, doing the constant replacement is much simple that reintroducing the protocol and API minimum field. Lets move forward with this. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740037#comment-13740037 ] Bikas Saha commented on MAPREDUCE-5311: --- The uber problem seems that MR happens to be a special user that for historical reasons has its fingers poked into all kinds of YARN internals and as such is suffering when we close those gaps in YARN. Is there an issue with MR code using the existing (and possible in future deprecated) min resource YARN configuration for its internal calculations? That value was basically exposed by YARN via the min API which we removed since we believed it wasnt relevant to users. MR as a special user needs that info. How about we just get it from the config directly. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740074#comment-13740074 ] Alejandro Abdelnur commented on MAPREDUCE-5311: --- [~bikassaha], you are on the spot. The only difference being proposed is that instead of using the scheduler MIN property directly, to define a new one for this particular use case in the MRAM namepsace (and deprecate it). The reason for doing this is that the exiting scheduler MIN propery would go away per YARN-1004. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740390#comment-13740390 ] Arun C Murthy commented on MAPREDUCE-5311: -- I'll say this again, I'm -1 on adding a new config to track another one in the scheduler. Expecting admins to remember another config for the same value (one for YARN another for MR) is not kosher - our configuration is bad enough. We can use the same one in the scheduler (i.e. don't fix YARN-1004) or we can add back RegisterApplicationMasterResponse.getMinimumResourceCapability as LimitedPrivate(MapReduce). I'm ok with either of the above. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740397#comment-13740397 ] Alejandro Abdelnur commented on MAPREDUCE-5311: --- [~acmurthy], there are other 4 committers that are OK with the idea of a separate config to enable SLOT_MILLIS because it is a special usecase for 0.23 users. The changes are must less disruptive and (IMO) adequate given the usecase. Can you please reconsider your -1? Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740577#comment-13740577 ] Arun C Murthy commented on MAPREDUCE-5311: -- This isn't a special case for 0.23. It will break tools built atop both hadoop-1.x hadoop-0.23.*. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740594#comment-13740594 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5311: bq. Is there an issue with MR code using the existing (and possible in future deprecated) min resource YARN configuration for its internal calculations? Lets just do this and like I argued on YARN-1004, let's keep the min around as a RM property and use it for the SLOTS_MILLIS calculation. For Fifo and CS, things will just work. For FairScheduler, the new config hack is same as forcing operators to set the existing RM property. BTW, please edit back the title to just indicate the problem. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740598#comment-13740598 ] Arun C Murthy commented on MAPREDUCE-5311: -- I'm ok with Vinod's suggestion too. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740613#comment-13740613 ] Arun C Murthy commented on MAPREDUCE-5311: -- [~tucu00] Can you please decide on one of the 3 options available which are reasonable? I'm ready to roll 2.1.0-rc2 ATM, this is the only issue I can think of. Thanks. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740646#comment-13740646 ] Alejandro Abdelnur commented on MAPREDUCE-5311: --- Given than this is a legacy thing coming from Hadoop 1, I don't think we should use at all YARN constants properties. Why we don't use the Hadoop 1 JT properties for the same, in the mapred-site.xml documenting how they have to be set to the MIN of the scheduler for SLOT_MILLIS counter to kick in? To me this seem a much more correct way of doing it. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740671#comment-13740671 ] Arun C Murthy commented on MAPREDUCE-5311: -- I'll repeat that I'm against multiple configs for the same property which leads to operator errors (admins setting one but forgetting to set the other) and am willing to live with other alternatives which seem better from a technical standpoint. bq. Why we don't use the Hadoop 1 JT properties for the same, in the mapred-site.xml documenting how they have to be set to the MIN of the scheduler for SLOT_MILLIS counter to kick in? To me this seem a much more correct way of doing it. ATM, I don't see where this argument is going at all - frankly, we seem to be making YARN-1004 much bigger deal than it is. Several folks have expressed concerns about multiple configs and yet we don't seem to converge; this is in spite of, IMO, very reasonable alternatives. I'm inclined to go ahead with rc2 without this YARN-1004. We can revisit both for 2.1.1 via deprecation of properties in YARN-1004 etc. if necessary. My plan is to cut an rc2 by noon PST tmrw (9/15). Let's revisit this after if we can agree today. Thanks. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738236#comment-13738236 ] Arun C Murthy commented on MAPREDUCE-5311: -- I've repeatedly pointed out that SLOT_MILLIS is something which is very useful and is used by a lot of end-users. I know of a lot of users who have built rudimentary reporting/charge-back systems using SLOT_MILLIS. Breaking compatibility (semantic or api) is not an option. We've done a lot of work on MR to keep it compatible so far. The more I think about it, the more strongly I believe YARN-787 went too far by removing 'minimum' from the API. We can/should keep RegisterApplicationMasterResponse.getMinimumResourceCapability as a LimitedPrivate api and use it only for calculation of SLOT_MILLIS. We don't have to expose it to users or other applications. Makes sense? Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738516#comment-13738516 ] Alejandro Abdelnur commented on MAPREDUCE-5311: --- [~acmurthy], it seems we are talking pass each other here. * 1. SLOT_MILLIS does make sense in YARN. yes/no? * 2. We need to redefine what SLOT_MILLIS means/reports in YARN. yes/no? Are we in agreement that the answers to these questions is #1 NO and #2 YES. If we are in agreement, then we have to see how to address this in the least disruptive way. Sandy's latest proposal suggests we do the following: * Introduce the concept of CONTAINER_MILLIS (regardless of the container size) * Deprecate SLOT_MILLIS and map it to report CONTAINER_MILLIS And we could later augment this with additional counters: * Introduce the concept of MEMORY_MILLIS * Introduce the concept of CPU_MILLIS Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738528#comment-13738528 ] Arun C Murthy commented on MAPREDUCE-5311: -- Let's up-level. We agree we can't break compatibility for existing applications ? If so, we can figure ways around how to fix YARN-1004, MAPREDUCE-5310 etc. without breaking compatibility. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738620#comment-13738620 ] Sandy Ryza commented on MAPREDUCE-5311: --- As slots no longer exist in MR2, we cannot avoid breaking compatibility for some existing applications. Keeping SLOTS_MILLIS as it is will break compatibility for the majority of MR1 clusters. Defining SLOTS_MILLIS to be the same as CONTAINER_MILLIS will break compatibility for the minority of clusters that both used the mapred.cluster.map.memory.mb property in MR1 and chose to configure memory in MR2 in a way that exactly lines up with slots in MR1 (i.e. setting yarn.scheduler.minimum-allocation-mb to exactly the same as mapred.cluster.map.memory.mb, which is often not the right choice). Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738663#comment-13738663 ] Arun C Murthy commented on MAPREDUCE-5311: -- bq. As slots no longer exist in MR2, we cannot avoid breaking compatibility for some existing applications. The original intention was to model the old notion of 'slots' as a minimum container. So, we have 2 cases: # mapred.cluster.(map,reduce).memory.mb set to -1 for the job. In this case, we just divide the 'timing' by 1. # mapred.cluster.(map,reduce).memory.mb set to non-zero then we normalize to yarn.scheduler.minimum-allocation-mb and divide 'timing' by that number. Makes sense? Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738908#comment-13738908 ] Jason Lowe commented on MAPREDUCE-5311: --- bq. Defining SLOTS_MILLIS to be the same as CONTAINER_MILLIS will break compatibility for the minority of clusters that both used the mapred.cluster.map.memory.mb property in MR1 and chose to configure memory in MR2 in a way that exactly lines up with slots in MR1 (i.e. setting yarn.scheduler.minimum-allocation-mb to exactly the same as mapred.cluster.map.memory.mb, which is often not the right choice). It will also break compatibility for those on 0.23 where SLOT_MILLIS is currently based on the minimum allocation. Changing the units from slots to containers for those cases will significantly lower the reported value and could be misinterpreted as lower cluster utilization. That's why I'm not in favor of changing the SLOT_MILLIS units but still calling it SLOT_MILLIS. I'd rather the counter disappear than have it be a completely wrong value. As Arun mentioned, there are in-house tools to compute cluster utilization based on these metrics. I agree that supporting SLOT_MILLIS long-term is problematic due to the removal of the minimum or slot concept, hence my earlier suggestion for something more supportable like MEM_MILLIS or CPU_MILLIS. For backwards compatibility, we could consider doing something along the lines of what it's already doing today, i.e.: having the AM read a config and computing the slot size from there. That way admins can configure whatever is appropriate for a slot for their cluster to keep legacy tools reporting something appropriate when they try to look for SLOT_MILLIS. If the property is not configured then arguably the AM should not report a SLOT_MILLIS job counter since it doesn't know how big a slot should be. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[
https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13739043#comment-13739043
]
Alejandro Abdelnur commented on MAPREDUCE-5311:
---
[~jlowe], thanks for jumping in, I was no aware you guys have in-house tools
using this stuff with the current reported SLOT_MILLIS. With that in mind, and
along the lines of what Jason proposes would the following satisfy all parties?
This JIRA would then be repurposed to:
* Introduce and deprecate a {{MRConf.SLOT_MILLIS_MINIMUM_ALLOCATION_MB}} with
no default
* If set, use the {{MRConf.SLOT_MILLIS_MINIMUM_ALLOCATION_MB}} value to
compute, using today's logic, the SLOT_MILLIS counter values.
* If no set, SLOT_MILLIS counter should report 0.
This means than anybody relying on current SLOT_MILLIS reporting can continue
getting it until we decide to trash it (a few versions down the road).
A different JIRA would introduce MEM_MILLIS and CPU_MILLIS which have an
accurate meaning in Yarn's world.
YARN-1004 is then unblocked.
I believe this address the problem without breaking backwards compatibility as
[~acmurthy] asked.
Arun, Jason, [~sandyr], are you OK with this approach?
Replace SLOTS_MILLIS counters with MEM_MILLIS
-
Key: MAPREDUCE-5311
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster
Affects Versions: 2.0.4-alpha
Reporter: Alejandro Abdelnur
Assignee: Sandy Ryza
Priority: Blocker
Fix For: 2.1.0-beta
Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch,
MAPREDUCE-5311.patch
Per discussion in MAPREDUCE-5310 and comments in the code we should remove
all the related logic and just leave the counter constant for backwards
compatibility and deprecate the counter constants.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735003#comment-13735003 ] Arun C Murthy commented on MAPREDUCE-5311: -- We seem to understand that we can't remove SLOTS_MILLIS_MAPS - that will break BC. For now, let's punt on this for 2.1.0-beta since we can add new counters (CONTAINERS_MILLIS_MAPS) after a bit more discussion. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13735011#comment-13735011 ] Alejandro Abdelnur commented on MAPREDUCE-5311: --- [~acmurthy], I don't see how this will break BC, it is not being proposed removing the counter, but either making it zero or as [~sandyr] suggested introduced CONTAINER_MILLIS_MAP and map SLOT_MILLIS_MAP to it (approach that will make more sense than the current value). I don't want to punt because you are blocking YARN-1004 because of this one. YARN-1004 should go in 2.1.0-beta. Please ping me if you want to chat offline over the phone if you think will be easier to discussed it. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731067#comment-13731067 ] Alejandro Abdelnur commented on MAPREDUCE-5311: --- [~acmurthy], unless I got things wrong, we agreed to keep slot-millis around until we have memory-millis. And the latest patch here is doing that. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731444#comment-13731444 ] Arun C Murthy commented on MAPREDUCE-5311: -- This will break every single MR app. I'm -1 on this. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731445#comment-13731445 ] Arun C Murthy commented on MAPREDUCE-5311: -- To Sandy's point - several folks do use mapred.cluster.(map,reduce).memory.mb, so it will be a big BC break for them. Also, what does this mean for BC after we spent all the time fixing MAPREDUCE-5108. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Replace SLOTS_MILLIS counters with MEM_MILLIS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13731466#comment-13731466 ] Sandy Ryza commented on MAPREDUCE-5311: --- I am missing how the change would break every single MR app - we would leave the API intact. However, does this sound like a reasonable compromise: We keep SLOTS_MILLIS_MAPS, but instead of trying to map resources to slots (which will make even less sense when we introduce other resources), we compute it as the sum of the execution times for all map tasks. Essentially SLOTS_MILLIS_MAPS functions as CONTAINERS_MILLIS_MAPS would and is computed independently from container capabilities. This will break semantic compatibility in some cases for people using the mapred.cluster.(map,reduce).memory.mb properties, but it will restore semantic compatibility for people not using those properties (which I believe is a much larger number). It will also make the counter function in a more understandable way that is more in line with its description. Replace SLOTS_MILLIS counters with MEM_MILLIS - Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Sandy Ryza Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5311-1.patch, MAPREDUCE-5311.patch, MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
