[jira] [Commented] (YARN-5600) Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045342#comment-16045342 ] Hadoop QA commented on YARN-5600: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color} | {color:red} YARN-5600 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12843484/YARN-5600.017.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16176/console | | Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application > basis > > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, > YARN-5600.011.patch, YARN-5600.012.patch, YARN-5600.013.patch, > YARN-5600.014.patch, YARN-5600.015.patch, YARN-5600.016.patch, > YARN-5600.017.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935231#comment-15935231 ] Hadoop QA commented on YARN-5600: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 50s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 9s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 522 unchanged - 21 fixed = 522 total (was 543) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 37s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 56s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 28s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 42s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 97m 18s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12843484/YARN-5600.017.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux fc83739b5187 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / c01d15a | | Default Java | 1.8.0_121 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/15346/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
[jira] [Commented] (YARN-5600) Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935084#comment-15935084 ] Miklos Szegedi commented on YARN-5600: -- [~vvasudev], please review the latest patch. Do you have any other comments on this jira? > Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application > basis > > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, > YARN-5600.011.patch, YARN-5600.012.patch, YARN-5600.013.patch, > YARN-5600.014.patch, YARN-5600.015.patch, YARN-5600.016.patch, > YARN-5600.017.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752808#comment-15752808 ] Hadoop QA commented on YARN-5600: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 59s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 555 unchanged - 21 fixed = 555 total (was 576) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 38s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 54s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 42s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 76m 33s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:a9ad5d6 | | JIRA Issue | YARN-5600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12843484/YARN-5600.017.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux a70a8d73949d 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 631f1da | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/14334/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
[jira] [Commented] (YARN-5600) Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752626#comment-15752626 ] Miklos Szegedi commented on YARN-5600: -- I attached a new patch addressing the comments. > Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application > basis > > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, > YARN-5600.011.patch, YARN-5600.012.patch, YARN-5600.013.patch, > YARN-5600.014.patch, YARN-5600.015.patch, YARN-5600.016.patch, > YARN-5600.017.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5600) Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15752624#comment-15752624 ] Miklos Szegedi commented on YARN-5600: -- Thank you, [~kasha] for the review! In general about the design: delete.debug-delay-sec is there for backward compatibility, I think. I do not expect many people to use it after this new functionality, since it affects the whole cluster. Because of this I would not mix it's value much with delete.max-per-application-debug-delay-sec. I suggest filing a separate jira to deprecate delete.debug-delay-sec for v3. This would make the design of this feature simpler. {quote} ApplicationConstants: typo in the javadoc. {quote} Fixed. {quote} YarnConfiguration: max-per-application-debug-delay-sec "per-application" is redundant and makes the config long. Can we just have max-debug-delay-sec or d debug-delay-sec.max. {quote} [~vvasudev], You suggested originally this value along with differentiating the default and the maximum. I am okay with changing it back, what is your opinion? {quote} I don't see the need to have a separate constant for the corresponding default value. Can we use the value of delete.debug-delay-sec or a multiple of it (say, 2) as default instead? {quote} I do not think it is a good idea to set a default to a variable value. I could not even write anything to yarn-default.xml in this case. Moreover the default is 0 in 99%+ of the cases. I would expect people will use the new feature more often in the future, since it does not affect the whole cluster. delete.debug-delay-sec is there for backward compatibility. Can we keep it separate than delete.debug-delay-sec? {quote} Also, looks like max-debug-delay-sec could be smaller than debug-delay-sec. That is confusing. Can we make sure max is actually max? {quote} I have to defer this to [~vvasudev], since it was his suggestion originally. {quote} Move the javadoc from ApplicationImpl to Application. Also, can we linkify Application.NEVER. {quote} I added it to both. {quote} If the admin sets debug-delay-sec to a positive value (say, 5 mins) and the user explicitly sets it to a smaller value (even 0), shouldn't the user's setting override for all containers of that app? {quote} The idea is that the standard value is 0 in 99%+ of cases. Sometimes the user or the admin wants to delay that by specifying a delay value. Using the max fulfills both requests, there is no priority, no override. I thought this is the simplest what I can do to make both of them happy, if I need to specify both. Limiting the delay would not make much sense, since any smaller number what requested prevents them to collect the necessary debug information for the other party. There is a feature to disable the delay functionality in sensitive applications and that is the cluster wide max delay. {quote} It appears we allow setting this value on a per-container basis through ContainerLaunchContext, but consider only the largest value. Is there value in allowing specifying this on a per-container basis. Why not specify it on a per-app basis? Say, through ApplicationSubmissionContext? {quote} We consider all values specified in each CLC. However, if an application has multiple containers on the same node, we delete the application dir only after the latest CLC is deleted. The largest logic just makes sure that the application dir is not deleted before the containers in it are deleted. Is not it more flexible to specify a container specific value? {quote} Any specific reason for capturing the deletion time in Date instead of long? Isn't Date more expensive? {quote} Date is just a wrapper above a long value. I feel the code is more readable with Date than it would be otherwise. You can for example get the real time value in the debugger without any additional calculations. {quote} yarn-default: s/mumber/number {quote} Fixed. {quote} DeletionServe scheduleFileDeletionTask takes a delay, and then adjusts it based on configuration. These seems error-prone. Callers of this method would assume the file will be deleted at the provided time. Can the caller take care of the adjustment? {quote} It depends. If this is true, it was already error prone before my change, since we requested a delete that happened later overridden by the default delay. All I do is adding a custom delay. The contract is that the task will be delayed for at least the amount of time specified. I was thinking about changing the function name or creating a separate function that reflects this design better that the caller would call. Honestly, scheduleFileDeletionTask should be refactored anyways to be private, so I just changed the documentation for now. What do you think? (This problem would also go away, if we deprecate default delay.) {quote} getEffectiveDelay: if the limitedDelay > maxDebugDelay, one could set limitedDelay to maxDebugDelay
[jira] [Commented] (YARN-5600) Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application basis
[ https://issues.apache.org/jira/browse/YARN-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15749754#comment-15749754 ] Karthik Kambatla commented on YARN-5600: This is quite handy. Thanks for working on this Miklos. Comments: # ApplicationConstants: typo in the javadoc. # YarnConfiguration: max-per-application-debug-delay-sec ## "per-application" is redundant and makes the config long. Can we just have {{max-debug-delay-sec}} or {{debug-delay-sec.max}}. ## I don't see the need to have a separate constant for the corresponding default value. Can we use the value of {{delete.debug-delay-sec}} or a multiple of it (say, 2) as default instead? ## Also, looks like max-debug-delay-sec could be smaller than debug-delay-sec. That is confusing. Can we make sure max is actually max? # Move the javadoc from ApplicationImpl to Application. Also, can we linkify {{Application.NEVER}}. # If the admin sets debug-delay-sec to a positive value (say, 5 mins) and the user explicitly sets it to a smaller value (even 0), shouldn't the user's setting override for all containers of that app? # It appears we allow setting this value on a per-container basis through ContainerLaunchContext, but consider only the largest value. Is there value in allowing specifying this on a per-container basis. Why not specify it on a per-app basis? Say, through ApplicationSubmissionContext? # Any specific reason for capturing the deletion time in Date instead of long? Isn't Date more expensive? # yarn-default: s/mumber/number # DeletionServe ## scheduleFileDeletionTask takes a delay, and then adjusts it based on configuration. These seems error-prone. Callers of this method would assume the file will be deleted at the provided time. Can the caller take care of the adjustment? ## getEffectiveDelay: if the limitedDelay > maxDebugDelay, one could set limitedDelay to maxDebugDelay without computing Math.min. I haven't looked at the tests yet. Will take a look in my next round. > Allow setting yarn.nodemanager.delete.debug-delay-sec on a per-application > basis > > > Key: YARN-5600 > URL: https://issues.apache.org/jira/browse/YARN-5600 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Daniel Templeton >Assignee: Miklos Szegedi > Labels: oct16-medium > Attachments: YARN-5600.000.patch, YARN-5600.001.patch, > YARN-5600.002.patch, YARN-5600.003.patch, YARN-5600.004.patch, > YARN-5600.005.patch, YARN-5600.006.patch, YARN-5600.007.patch, > YARN-5600.008.patch, YARN-5600.009.patch, YARN-5600.010.patch, > YARN-5600.011.patch, YARN-5600.012.patch, YARN-5600.013.patch, > YARN-5600.014.patch, YARN-5600.015.patch, YARN-5600.016.patch > > > To make debugging application launch failures simpler, I'd like to add a > parameter to the CLC to allow an application owner to request delayed > deletion of the application's launch artifacts. > This JIRA solves largely the same problem as YARN-5599, but for cases where > ATS is not in use, e.g. branch-2. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org