[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733405#comment-15733405 ] Miklos Szegedi commented on YARN-5600: -- The unit test issue is YARN-5857 TestLogAggregationService.testFixedSizeThreadPool fails intermittently on trunk > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, > YARN-5600.011.patch, YARN-5600.012.patch, YARN-5600.013.patch, > YARN-5600.014.patch, YARN-5600.015.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15732961#comment-15732961 ] Miklos Szegedi commented on YARN-5600: -- The unit test issue is not related to the change. I cannot reproduce it locally. > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, > YARN-5600.011.patch, YARN-5600.012.patch, YARN-5600.013.patch, > YARN-5600.014.patch, YARN-5600.015.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730803#comment-15730803 ] Hadoop QA commented on YARN-5600: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 58s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 52s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 554 unchanged - 21 fixed = 554 total (was 575) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 37s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 4s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 60m 56s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12842253/YARN-5600.015.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux 38cae932a85f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / ea2895f | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | unit | https://builds.a
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15730449#comment-15730449 ] Miklos Szegedi commented on YARN-5600: -- Thank you, [~templedf]! I addressed most issues. Here are my replies on the others. {quote} In serviceInit(), you're missing the space before the brace on both your if statements. Also, do you think you can find a sane way to combine them or at least reuse the error message? {quote} I think it might even be more readable, when it is expanded like this. {quote} I'm still hung up on +999 / 1000 in ResourceLocalizationService. How about Math.ceil(... / 1000.0)? {quote} I think we are talking about personal preferences at this point. I think that converting to float and back is a bit overkill. This pattern is a known one for rounding an integer up to the nearest multiple of a number without casting to float. {quote} In TestContainerManager, I'm worried that the new tests are pretty heavily timing dependent. At the very least, the second wait should be for the full timeout period. Even better would be to validate the internal state is as expected, e.g. that the deletion thread is set to execute after the expected amount of time. {quote} The two waits add up to the total wait time. This is how it is designed to be consistent. Waiting half more is unnecessary, since {code}waitForApplicationDirDeleted{code} does the same. It verifies the internal state, whether the file was deleted. I see this pattern in other places in the tests as well. However, to make it more robust, I increased this timeout. The third test, {code}testDelayedKeep(){code} may cause false positives in case of a busy server, but any errors will be caught by "normal" test machines. {quote} In TestDeletionService.testCustomDisableDelete(), do you need to set DEBUG_NM_MAX_PER_APPLICATION_DELETE_DELAY_SEC? Also, you're missing an assert message in that test method further down. {quote} Are we looking at the same patch? I set it here: {code} 204 conf.setInt(YarnConfiguration.DEBUG_NM_DELETE_DELAY_SEC, almostForever); 205 conf.setInt(YarnConfiguration.DEBUG_NM_MAX_PER_APPLICATION_DELETE_DELAY_SEC, 206 Integer.MAX_VALUE); {code} {quote} testCustomRetentionPolicy() should probably also test what happens if no container override is given, if the container override is 0, and if it's 2. {quote} testEffectiveDelay() is testing these cases already. What do you think? > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, > YARN-5600.011.patch, YARN-5600.012.patch, YARN-5600.013.patch, > YARN-5600.014.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15729872#comment-15729872 ] Daniel Templeton commented on YARN-5600: Thanks, [~miklos.szeg...@cloudera.com] for all the updates. I'm going to stir the pot a little to see if we can get this one closed. :) Comments: * The blank lines in {{ApplicationConstants.DEBUG_DELETE_DELAY}} should have the leading asterisk * I think the javadoc for {{YarnConfiguration. DEBUG_NM_MAX_PER_APPLICATION_DELETE_DELAY_SEC}} is wrong. * In {{yarn-default.xml}}, let's not reference code if we can help it: {{refer to Environment.DEBUG_DELETE_DELAY}}. Better to just name the env var directly. * Language: {{prevent unreliable or malicious clients keep files arbitrarily.}} should be {{prevent unreliable or malicious clients from keeping files arbitrarily.}} * In {{DeletionService. getEffectiveDelaySec()}}, this code seems unnecessarily verbose: {code}int effectiveDelay; effectiveDelay = Math.max(debugDelayDefault, limitedDelay); return effectiveDelay;{code} It could just be a return. * The javadoc on this method seems off: {code} /** * Peek the beginning of the queue. * @return scheduled task count */ @VisibleForTesting ScheduledThreadPoolExecutor getSched() { return sched; }{code} * In {{serviceInit()}}, the indention looks wrong here: {code} ThreadFactory tf = new ThreadFactoryBuilder() .setNameFormat("DeletionService #%d") .build();{code} * In {{serviceInit()}}, you're missing the space before the brace on both your _if_ statements. Also, do you think you can find a sane way to combine them or at least reuse the error message? * In {{ApplicationImpl. getDelayedDeletionTime()}} javadoc, "container" is misspelled. * I'm still hung up on {{+999 / 1000}} in {{ResourceLocalizationService}}. How about {{Math.ceil(... / 1000.0)}}? * In {{TestContainerManager}}, I'm worried that the new tests are pretty heavily timing dependent. At the very least, the second wait should be for the full timeout period. Even better would be to validate the internal state is as expected, e.g. that the deletion thread is set to execute after the expected amount of time. * In {{TestDeletionService.testCustomDisableDelete()}}, do you need to set {{DEBUG_NM_MAX_PER_APPLICATION_DELETE_DELAY_SEC}}? Also, you're missing an assert message in that test method further down. * {{testCustomRetentionPolicy()}} should probably also test what happens if no container override is given, if the container override is 0, and if it's 2. > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, > YARN-5600.011.patch, YARN-5600.012.patch, YARN-5600.013.patch, > YARN-5600.014.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15675547#comment-15675547 ] Miklos Szegedi commented on YARN-5600: -- Thank you, [~vvasudev] for the comment! We will get 600 in all of the three cases above. The current rule is Math.max(CONF,Math.min(CUSTOM,MAX_CONF)), where CONF is yarn.nodemanager.delete.debug-delay-sec, custom is DEBUG_DELETE_DELAY and MAX_CONF is delete.max-per-application-debug-delay-sec. 1) We choose 600 over 200. This makes sense, since an administrator expressed that they want to keep any file longer than what the applications specified. 2) We chose 600 over 300. We did not let the client choose 400 with the maximum limit, but the maximum does not apply to the default, so that the administrator does not need to change two values for one configuration option. 3) We choose 600 over 300. A custom application request came in, so by using the maximum we fulfill both the administrator's and the client's request Thank you! > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, > YARN-5600.011.patch, YARN-5600.012.patch, YARN-5600.013.patch, > YARN-5600.014.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15674060#comment-15674060 ] Varun Vasudev commented on YARN-5600: - [~miklos.szeg...@cloudera.com] - can you explain when the container and application resources will be cleaned up in the following scenarios - 1) yarn.nodemanager.delete.debug-delay-sec=600 delete.max-per-application-debug-delay-sec=300 container sets DEBUG_DELETE_DELAY to 200 2) yarn.nodemanager.delete.debug-delay-sec=600 delete.max-per-application-debug-delay-sec=300 container sets DEBUG_DELETE_DELAY to 400 3) yarn.nodemanager.delete.debug-delay-sec=300 delete.max-per-application-debug-delay-sec=600 container sets DEBUG_DELETE_DELAY to 600 Thanks! Thanks! > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, > YARN-5600.011.patch, YARN-5600.012.patch, YARN-5600.013.patch, > YARN-5600.014.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15669156#comment-15669156 ] Hadoop QA commented on YARN-5600: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 37s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 0s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 51s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 557 unchanged - 21 fixed = 557 total (was 578) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 28s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m 48s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 60m 27s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12839093/YARN-5600.014.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux 20667dc03ebb 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 61c0bed | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13934/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoo
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668813#comment-15668813 ] Hadoop QA commented on YARN-5600: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 36s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m 13s{color} | {color:red} hadoop-yarn in trunk failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 35s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 34s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 57s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 558 unchanged - 21 fixed = 559 total (was 579) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 41s{color} | {color:red} hadoop-yarn-api in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 37s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 46s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 67m 16s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12839076/YARN-5600.013.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux c43609932365 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / f121d0b | | Default Java | 1.8.0_101 | | compile | https://builds.apache.org/job/PreCommit-YARN-Build/13931/artifact/pa
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668543#comment-15668543 ] Hadoop QA commented on YARN-5600: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color} | {color:red} YARN-5600 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12839056/YARN-5600.012.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/13929/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, > YARN-5600.011.patch, YARN-5600.012.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668474#comment-15668474 ] Miklos Szegedi commented on YARN-5600: -- Thank you, [~vvasudev] for the comments! I addressed all the issues that you raised. I will send out the new patch shortly. "Given that it was undocumented, I think it’s fine to break it in trunk. Maybe in branch-2 we leave as it was but undocumented as earlier?" We can do that. Since we are talking about a debug feature we may even be able to backport the whole change as it is. What do you think? > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, > YARN-5600.011.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15667245#comment-15667245 ] Varun Vasudev commented on YARN-5600: - Thanks for the updated patch [~miklos.szeg...@cloudera.com]! Some more comments - 1) {code} + * $LOG_DIRS {code} Wrong variable name 😃 2) {code} +DELETE_DELAY("DELETE_DELAY"); {code} How about DEBUG_DELETE_DELAY (for the variable and the environment variable) instead just to emphasize the debug part of it. 3) bq. I removed -1 as the default value. However, the more changes and administrator needs to do to enable the feature, the less likely it will be used. Because of this I suggest using 10 minutes as a default. This allows the user to modify an application launch and copy the results, when needed without a reboot. Is 10 minutes acceptable in a production cluster? No I think it’s fine for admins to be required to make the change. The reality is that 10 minutes may be acceptable to some and may not be acceptable to others. 4) bq. Note: this change will break backward compatibility, although -1 was not documented before. Given that it was undocumented, I think it’s fine to break it in trunk. Maybe in branch-2 we leave as it was but undocumented as earlier? 5) {code} + + Maximum mumber of seconds after an application finishes before the + nodemanager's DeletionService will delete the application's localized + file directory and log directory. This value is a security setting to + prevent unreliable or malicious clients keep files arbitrarily. + + This is a maximum limit on the following values +yarn.nodemanager.delete.debug-delay-sec +Environment.DELETE_DELAY + +yarn.nodemanager.delete.max-debug-delay-sec +600 + {code} This value shouldn’t affect yarn.nodemanager.delete.debug-delay-sec - it doesn’t make sense to have to modify a config file in two places if I want to enable debug delay for all applications. Instead of calling it "yarn.nodemanager.delete.max-debug-delay-sec”, how about calling it “yarn.nodemanager.delete.max-per-application-debug-delay-sec” that makes it clear that it’s related to the application debug feature and won’t conflict with "yarn.nodemanager.delete.max-debug-delay-sec" 6) {code} + private int debugDelayDefaultMax; {code} Rename debugDelayDefaultMax to maxDebugDelay 7) {code} + long getTaskCount() { {code} Annotate as VisibleForTesting? > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, > YARN-5600.011.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665680#comment-15665680 ] Hadoop QA commented on YARN-5600: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 50s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 51s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 563 unchanged - 21 fixed = 563 total (was 584) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 36s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 35s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 37s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 62m 30s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12838881/YARN-5600.011.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux 0c7597129377 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 99c2bbd | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13910/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-a
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665441#comment-15665441 ] Hadoop QA commented on YARN-5600: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 18s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 35s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 50s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 563 unchanged - 21 fixed = 564 total (was 584) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 39s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m 1s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 62m 27s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12838859/YARN-5600.010.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux 43ec835d9270 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 99c2bbd | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/13907/ar
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665283#comment-15665283 ] Miklos Szegedi commented on YARN-5600: -- Thank you, [~vvasudev] for the review! I will send a revised patch shortly. 1) 2) You are right, I moved the constant definition out from YarnConfiguration. 3) I removed -1 as the default value. However, the more changes and administrator needs to do to enable the feature, the less likely it will be used. Because of this I suggest using 10 minutes as a default. This allows the user to modify an application launch and copy the results, when needed without a reboot. Is 10 minutes acceptable in a production cluster? The administrator can change this to 0 with a reboot. What do you think? 4) I agree. Removing the forever option makes the code cleaner and easier to maintain. Note: this change will break backward compatibility, although -1 was not documented before. 5) I changed the logging to info as requested. 6) This was probably my toolset. I moved it back to the original location. 7) Since I changed significant portions, I decided to fix checkstyle violations in the file. checkstyle was the reason to rename these variables. > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15663130#comment-15663130 ] Varun Vasudev commented on YARN-5600: - Thanks for the patch [~miklos.szeg...@cloudera.com]! My apologies for the late comments - 1) {code} +String value = environment +.get(YarnConfiguration.DEBUG_NM_CUSTOM_DELETE_DELAY_SEC); {code} The environment variable name shouldn’t come from YarnConfiguration, which is meant for entries in yarn-site.xml. Please add the variable to the Environment class in ApplicationConstants.java 2) {code} + /** Delay before deleting resource to ease debugging of NM issues. + * This is a custom key for the environment map in + * container launch context*/ + public static final String DEBUG_NM_CUSTOM_DELETE_DELAY_SEC = + YarnConfiguration.NM_PREFIX + "delete.custom-debug-delay-sec"; + public static final int DEFAULT_DEBUG_NM_CUSTOM_DELETE_DELAY_SEC = 0; {code} Not required 3) {code} + /** Maximum delay before deleting the resource. + * This setting limits YarnConfiguration.DEFAULT_DEBUG_NM_DELETE_DELAY_SEC + * and any container specific delay as well for reliability and security*/ + public static final String DEBUG_NM_MAX_DELETE_DELAY_SEC = + YarnConfiguration.NM_PREFIX + "delete.max-debug-delay-sec"; + public static final int DEFAULT_DEBUG_NM_MAX_DELETE_DELAY_SEC = + -1; {code} {code} + + + This is a maximum cap on the following values +yarn.nodemanager.delete.debug-delay-sec +yarn.nodemanager.delete.custom-debug-delay-sec + + Maximum mumber of seconds after an application finishes before the nodemanager's + DeletionService will delete the application's localized file directory + and log directory. The value -1 means let the service keep the files forever, + if requested by any of the settings above. + +yarn.nodemanager.delete.max-debug-delay-sec +-1 + {code} The default value for this can’t be -1. This means that users can keep resources around forever without admins knowing about it. The default value should be 0. Admins should be required to explicitly set the value to a number they’re comfortable with. 4) To a larger point, I don’t think we should provide a keep forever option at all. Admins can set the value to a really high number if they desire. The idea behind the feature is to help debugging which implies a reasonable time window(a few days to a week?) - not sure why we need to keep artifacts around forever. 5) {code} +if (debugDelaySec == KEEP_FOREVER && +debugDelayDefaultMax != KEEP_FOREVER) { + LOG.warn(String.format("Limiting debug delay %d to %d", + debugDelaySec, debugDelayDefaultMax)); + limitedDelay = debugDelayDefaultMax; +} else if (debugDelaySec > debugDelayDefaultMax && +debugDelayDefaultMax != KEEP_FOREVER) { + LOG.warn(String.format("Limiting debug delay %d to %d", + debugDelaySec, debugDelayDefaultMax)); + limitedDelay = Math.min(debugDelaySec, debugDelayDefaultMax); +} {code} Not sure these should be logged as warnings - info might be fine? 6) {code} -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertFalse; -import static org.junit.Assert.assertNull; -import static org.junit.Assert.assertTrue; {code} Any particular reason for changing the location of the import statements? 7) {code} + private static final FileContext LFS = getLfs(); + private static FileContext getLfs() { {code} {code} - private static final Path base = -lfs.makeQualified(new Path("target", TestDeletionService.class.getName())); {code} Similar to above - any reason for renaming the variables? > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This me
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15657638#comment-15657638 ] Hadoop QA commented on YARN-5600: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 42s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m 5s{color} | {color:red} hadoop-yarn in trunk failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 14s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 58s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 58s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 54s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 538 unchanged - 21 fixed = 538 total (was 559) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 35s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 27s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 47s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 62m 5s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:e809691 | | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12838573/YARN-5600.009.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux 0f5199d57dff 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 503e73e | | Default Java | 1.8.0_101 | | compile | https://builds.apache.org/job/PreCommit-YARN-Build/1387
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656248#comment-15656248 ] Hadoop QA commented on YARN-5600: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 52s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 54s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 538 unchanged - 21 fixed = 538 total (was 559) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 34s{color} | {color:red} hadoop-yarn-api in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 44s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 39s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 57m 11s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:e809691 | | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12838500/YARN-5600.008.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 99fb6d6b2e4f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 8848a8a | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/13868/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13868/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-serve
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655828#comment-15655828 ] Miklos Szegedi commented on YARN-5600: -- Thank you for reviewing the patch [~wangda], and also thank you for looking into the build issue! I think it is a good idea to have the value in an environment variable. Speaking about a debug feature I would like the change to be as small and simple as possible. When talking about environment variables, are you referring to this pattern? {code} String containerImageName = container.getLaunchContext().getEnvironment() .get(YarnConfiguration.NM_DOCKER_CONTAINER_EXECUTOR_IMAGE_NAME); {code} I will write a new patch that includes your suggestions including yarn.nodemanager.delete.max-debug-delay-sec value to limit the maximum wait. > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655703#comment-15655703 ] Tan, Wangda commented on YARN-5600: --- This is a very useful feature without any doubt, thanks [~miklos.szeg...@cloudera.com] for working on this JIRA and thanks reviews from [~Naganarasimha] / [~templedf] for reviewing the patch. Apologize for my very late review, I only looked at API of the patch: Have you ever considered the other approach, which we can turn on debug-delay-sec by passing a pre-defined environment. The biggest benefit is, we don't need to update most applications to use this feature, for example, MR/Spark support specify environment. Making changes to all major applications to use this feature sounds like a big task. As an example LinuxDockerContainerExecutor uses the approach which specify configurations by passing env var. And in addition, it will be better to have a global max-debug-delay-sec in yarn-site (which could be MAX_INT by default), considering disk space and security, we may want application occupy disk beyond some specified time. + [~djp] > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655581#comment-15655581 ] Hadoop QA commented on YARN-5600: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 9s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m 22s{color} | {color:red} hadoop-yarn in trunk failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 17s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 4m 8s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:red}-1{color} | {color:red} cc {color} | {color:red} 4m 8s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 4m 8s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 54s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 577 unchanged - 7 fixed = 577 total (was 584) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 34s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 30s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 55s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 45s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 64m 33s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:e809691 | | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12838452/YARN-5600.007.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux 0805289b3944 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 93eeb13 | | Default Java | 1.8.0_101 | | compile | https://builds.apache.org/job/PreCommit-YARN-Build/13864/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt | | findbugs | v3.0.0 | | compi
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654874#comment-15654874 ] Miklos Szegedi commented on YARN-5600: -- The build error seems to be unrelated to the change: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (npm install) on project hadoop-yarn-ui > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654842#comment-15654842 ] Hadoop QA commented on YARN-5600: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 53s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 4s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m 17s{color} | {color:red} hadoop-yarn in trunk failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 42s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:red}-1{color} | {color:red} cc {color} | {color:red} 3m 42s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 42s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 53s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 577 unchanged - 7 fixed = 577 total (was 584) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 34s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 24s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 45s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 38s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 64m 21s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:e809691 | | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12838405/YARN-5600.006.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux 7c38b041e57b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 89354f0 | | Default Java | 1.8.0_111 | | compile | https://builds.apache.org/job/PreCommit-YARN-Build/13862/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt | | findbugs | v3.0.0 | | compi
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653290#comment-15653290 ] Hadoop QA commented on YARN-5600: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 14s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 55s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 577 unchanged - 7 fixed = 578 total (was 584) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 35s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 28s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 31s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 65m 55s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:e809691 | | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12838299/YARN-5600.005.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux 35c7d9d948ac 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / c202a10 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/13852/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt | | Test Results | https://bu
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15653151#comment-15653151 ] Miklos Szegedi commented on YARN-5600: -- Thank you, for the useful comments [~templedf]! I fixed most of the requests, I have just a few comments: "I'd rather see you divide by 1000 and add 1. It's a little less clever/more obvious." Unfortunately, it does not give the same results. It will add an unnecessary second, when the number is dividable by 1000 and that is not what I need. "In TestContainerManager, verifyContainerDir() and verifyAppDir() have a lot of common code. Maybe pull it out into another method?" There are sporadic differences that make it difficult to be one single function. They used to be one function and I changed it because I think this is still more readable this way. "In waitForApplicationDirDeleted() you might be better off with monotonic time." Can you be more specific? > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649299#comment-15649299 ] Daniel Templeton commented on YARN-5600: New comments: * Maybe define a constant for {{new Date(Long.MAX_VALUE)}} is make it a little more obvious. * In {{DeletionService.deleteWithDelay()}}, if the delay is -1, you're doing a bunch of work for no reason. * The javadoc description for the {{scheduleFileDeletionTask()}} methods should end with a period. * Is throwing the {{IllegalArgumentException}} the right thing to do? That information is not going to make it back to the end user who set the bogus value. I didn't verify what happens when the dispatcher gets the exception, but it may take the NM down. * Here: {code} debugDelayDefault = conf.getInt( YarnConfiguration.DEBUG_NM_DELETE_DELAY_SEC, 0);{code} it would be better to make the 0 a constant. * In {{ResourceLocalizationService.handleDestroyApplicationResources()}} you have {code}debugDelaySec = (int)(((applicationCleanupTime.getTime() - now.getTime()) + 999) / 1000);{code} I'd rather see you divide by 1000 and add 1. It's a little less clever/more obvious. * The javadoc description for {{ApplicationImpl.delayedDeletionTime}} should end with a period. * {{estimateRetention()}} should maybe be {{calculateRetention()}} or {{updateRetention()}} or {{recalculateRetention()}}. There's no estimation happening. * {{TestDeletionService.testCustomDisableDelete()}} has a spurious "remain" in the javadoc * In {{TestDeletionService.testCustomDisableDelete()}}, I'm concerned about the change from a 20 sec wait to a 1 sec wait. Are you sure that in all cases, even on slow ancient upstream infrastructure, that 1 sec is enough? Can you sensibly expose some metrics from the {{DeletionService}} so you can tell when a delete is ignored, rather than waiting around to see if the file is still there? * You have a {{println()}} in {{testCustomRetentionPolicy()}}. * I'm still worried about flakiness with the sleeps at the end of {{testCustomRetentionPolicy()}}. Can you sensibly expose some metrics from the {{DeletionService}} so you can be more assertive about what happened, rather than relying on timing across threads? * You have a {{println()}} in {{testAPIError()}}. * {{testCustomRetentionPolicy()}} and {{testAPIError()}} are almost identical. Maybe add a parameter and reuse the code? * In {{TestContainerManager}}, {{verifyContainerDir()}} and {{verifyAppDir()}} have a lot of common code. Maybe pull it out into another method? * In {{waitForApplicationDirDeleted()}} you might be better off with monotonic time. > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15637202#comment-15637202 ] Hadoop QA commented on YARN-5600: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 374 unchanged - 6 fixed = 374 total (was 380) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 21s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 20s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 56m 4s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12837184/YARN-5600.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux c47dece6fb99 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 0aafc12 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13788/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-ya
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634927#comment-15634927 ] Hadoop QA commented on YARN-5600: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 54s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 25s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 44s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 374 unchanged - 6 fixed = 378 total (was 380) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 19s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 17s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 22s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 56m 14s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12837006/YARN-5600.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux b451a2178d1b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 5cad93d | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/13778/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt | | Test Results | h
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634788#comment-15634788 ] Miklos Szegedi commented on YARN-5600: -- Thank you for the review [~templedf]! It seems to me that you're doing extra work to keep the delete time as a Date, not to mention adding potential time zone concerns. Millis since the epoch may be simpler. I do not necessarily agree with using a long of milliseconds here. The value is a time stamp and Date is the natural representation. I just use the standard Date functions. Date does not bring in timezone concerns, it is implemented with the same long inside. It is supposed to be timezone independent. A custom implementation may add review overhead. I could try MonotonicClock, but it uses System.nanoTime(). It is not safe to use System.nanoTime() across threads and getDelayedDeletionTime() may be called from multiple threads. The DeletionService.scheduleFileDeletionTask() methods can and probably should be private. Unfortunately ResourceLocalizationService#cleanUpFilesPerUserDir uses it, so I have to keep it public. I agree that this needs to be refactored. Feel free to file a JIRA. In your tests, instead of sleeping and asserting, sleep for short periods in a loop to minimize the test time. testDelayedDeletionContainerOnly(), testDelayedDeletion(): I need to wait as long as the specified retention time but I added the suggested pattern once that has elapsed. testDelayedKeep(): the condition is that nothing happens for X amount of time, so the wait is expected. I am happy to accept suggestions on the wait time, here. I think 200ms should be enough to catch any pending deletes. > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634787#comment-15634787 ] Miklos Szegedi commented on YARN-5600: -- Thank you for the review [~templedf]! It seems to me that you're doing extra work to keep the delete time as a Date, not to mention adding potential time zone concerns. Millis since the epoch may be simpler. I do not necessarily agree with using a long of milliseconds here. The value is a time stamp and Date is the natural representation. I just use the standard Date functions. Date does not bring in timezone concerns, it is implemented with the same long inside. It is supposed to be timezone independent. A custom implementation may add review overhead. I could try MonotonicClock, but it uses System.nanoTime(). It is not safe to use System.nanoTime() across threads and getDelayedDeletionTime() may be called from multiple threads. The DeletionService.scheduleFileDeletionTask() methods can and probably should be private. Unfortunately ResourceLocalizationService#cleanUpFilesPerUserDir uses it, so I have to keep it public. I agree that this needs to be refactored. Feel free to file a JIRA. In your tests, instead of sleeping and asserting, sleep for short periods in a loop to minimize the test time. testDelayedDeletionContainerOnly(), testDelayedDeletion(): I need to wait as long as the specified retention time but I added the suggested pattern once that has elapsed. testDelayedKeep(): the condition is that nothing happens for X amount of time, so the wait is expected. I am happy to accept suggestions on the wait time, here. I think 200ms should be enough to catch any pending deletes. > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15630951#comment-15630951 ] Daniel Templeton commented on YARN-5600: Thanks for the patch, [~miklos.szeg...@cloudera.com]. Some comments: * It seems to me that you're doing extra work to keep the delete time as a {{Date}}, not to mention adding potential time zone concerns. Millis since the epoch may be simpler. * Ignoring the {{IOException}} in {{ResourceLocalizationService.submitDirForDeletion()}} seems bad. While you're in there, it might be good to do something more useful. * In your javadoc, the param text should start with a lower case letter, e.g. {{DeletionService#deleteWithDelay()}} * The {{DeletionService.scheduleFileDeletionTask()}} methods can and probably should be private. * In your tests, instead of sleeping and asserting, sleep for short periods in a loop to minimize the test time. * In {{TestContainerManager}} you have {code}-for (File f : new File[] { containerDir, containerSysDir }) { +for (File f : new File[] {containerDir, containerSysDir }) {{code} You may as well remove the trailing space as well. * In {{TestContainerManager.verifyContainerDir()}}, your if-if-else-else-if-else would be cleaner as if-elseif-elseif-else. Also, the messages could be a little more descriptive so that someone reading it without the source code has some clue what's happening. And I don't think we need the exclamation points. :) Otherwise, the general approach looks fine. > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569739#comment-15569739 ] Hadoop QA commented on YARN-5600: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 6 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 4s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 25s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 41s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 38s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 33s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s {color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 378 unchanged - 6 fixed = 378 total (was 384) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 51s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s {color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 16s {color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 7s {color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 47m 15s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12832964/YARN-5600.002.patch | | JIRA Issue | YARN-5600 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux 56f439bb743f 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 6476934 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/13365/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569608#comment-15569608 ] Miklos Szegedi commented on YARN-5600: -- TestQueuingContainerManager.testKillMultipleOpportunisticContainers: It passes locally most of the time but I have seen instances when it failed with this error. > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Attachments: YARN-5600.000.patch, YARN-5600.001.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567250#comment-15567250 ] Hadoop QA commented on YARN-5600: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 6 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 2s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 37s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 36s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 9s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 14s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 41s {color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 378 unchanged - 6 fixed = 380 total (was 384) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 49s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s {color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 16s {color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 7s {color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 44m 50s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12832804/YARN-5600.001.patch | | JIRA Issue | YARN-5600 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux 78ea9e15b24b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / b84c489 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/13
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563897#comment-15563897 ] Hadoop QA commented on YARN-5600: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 6 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 57s {color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 50s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 51s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 51s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 36s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 41s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 59s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s {color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s {color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 46s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 46s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 44s {color} | {color:red} hadoop-yarn-project/hadoop-yarn: The patch generated 40 new + 381 unchanged - 2 fixed = 421 total (was 383) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 31s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 48s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s {color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 17s {color} | {color:red} hadoop-yarn-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 5s {color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 47m 52s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | | org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.getDelayedDeletionTime() may expose internal representation by returning ApplicationImpl.delayedDeletionTime At ApplicationImpl.java:by returning ApplicationImpl.delayedDeletionTime At ApplicationImpl.java:[line 696] | | Failed junit tests | hadoop.yarn.api.TestPBImplRecords | | | hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12832558/YARN-5600.000.patch | | JIRA Issue | YARN-5600 | | Optional Tests | asflicense compile javac
[jira] [Commented] (YARN-5600) Add a parameter to ContainerLaunchContext to emulate yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15457573#comment-15457573 ] Naganarasimha G R commented on YARN-5600: - +1 for the approach. > Add a parameter to ContainerLaunchContext to emulate > yarn.nodemanager.delete.debug-delay-sec on a per-application basis > --- > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Daniel Templeton >Assignee: Daniel Templeton > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org