[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16215456#comment-16215456 ] Miklos Szegedi commented on YARN-6757: -- Yes, we can modify the type. > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6757.000.patch, YARN-6757.001.patch, > YARN-6757.002.patch, YARN-6757.003.patch, YARN-6757.004.patch, > YARN-6757.005.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16214897#comment-16214897 ] Naganarasimha G R commented on YARN-6757: - [~dan...@cloudera.com] & [~miklos.szeg...@cloudera.com], This jira looks more like a improvement rather than bug fix, shall we modify the type ? > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6757.000.patch, YARN-6757.001.patch, > YARN-6757.002.patch, YARN-6757.003.patch, YARN-6757.004.patch, > YARN-6757.005.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118781#comment-16118781 ] Daniel Templeton commented on YARN-6757: Also committed to branch-2. > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6757.000.patch, YARN-6757.001.patch, > YARN-6757.002.patch, YARN-6757.003.patch, YARN-6757.004.patch, > YARN-6757.005.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118767#comment-16118767 ] Miklos Szegedi commented on YARN-6757: -- Yes please. > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Fix For: 3.0.0-beta1 > > Attachments: YARN-6757.000.patch, YARN-6757.001.patch, > YARN-6757.002.patch, YARN-6757.003.patch, YARN-6757.004.patch, > YARN-6757.005.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113835#comment-16113835 ] Hadoop QA commented on YARN-6757: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 41s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 38s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 51s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 56s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 33s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 28s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 16s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 68m 48s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6757 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12880306/YARN-6757.005.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle | | uname | Linux 6e22641
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113525#comment-16113525 ] Daniel Templeton commented on YARN-6757: LGTM. +1 > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6757.000.patch, YARN-6757.001.patch, > YARN-6757.002.patch, YARN-6757.003.patch, YARN-6757.004.patch, > YARN-6757.005.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113506#comment-16113506 ] Miklos Szegedi commented on YARN-6757: -- I attached the patch. > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6757.000.patch, YARN-6757.001.patch, > YARN-6757.002.patch, YARN-6757.003.patch, YARN-6757.004.patch, > YARN-6757.005.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113485#comment-16113485 ] Daniel Templeton commented on YARN-6757: Fine, leave Yarn in there and file a follow up JIRA to make it YARN in all the docs. > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6757.000.patch, YARN-6757.001.patch, > YARN-6757.002.patch, YARN-6757.003.patch, YARN-6757.004.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113376#comment-16113376 ] Miklos Szegedi commented on YARN-6757: -- Hmm, I do not understand. If YARN is the official naming should not we fix them everywhere now and enforce from now on? If the rule is just for this change, I actually think Yarn is more readable. > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6757.000.patch, YARN-6757.001.patch, > YARN-6757.002.patch, YARN-6757.003.patch, YARN-6757.004.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111904#comment-16111904 ] Daniel Templeton commented on YARN-6757: I think the YARN replacement may have gone a bit too far. I don't think you should be making changes in the release notes for shipped releases. To keep this from being a huge patch, let's limit the changes to the md doc you're editing and the text you're adding to yarn-defaults.xml. > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6757.000.patch, YARN-6757.001.patch, > YARN-6757.002.patch, YARN-6757.003.patch, YARN-6757.004.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109640#comment-16109640 ] Hadoop QA commented on YARN-6757: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m 2s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 43s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 44s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 21s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 59s{color} | {color:green} root: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 7s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 25s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 25s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 39s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 22s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 8s{co
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108256#comment-16108256 ] Miklos Szegedi commented on YARN-6757: -- I changed only documentation in the recent patch, so the unit test failures are probably due to something else. Can we rerun? > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6757.000.patch, YARN-6757.001.patch, > YARN-6757.002.patch, YARN-6757.003.patch, YARN-6757.004.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108245#comment-16108245 ] Hadoop QA commented on YARN-6757: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m 45s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 52s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 57s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 21s{color} | {color:green} root: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 7m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 45s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 2s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}106m 58s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 49s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 7s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 16s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 27s{co
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107785#comment-16107785 ] Daniel Templeton commented on YARN-6757: Thanks for the update, [~miklos.szeg...@cloudera.com]. A few more comments, mostly language/grammar: * The {{ResourceHandlerModule}} methods ({{getTrafficControlBandwidthHandler}}, et al) where you fixed the indentation should now have the third line indented two more spaces. * In {{yarn-defaults.xml}}: ** "so use this setting only, if the discovery does not work" shouldn't have a comma. ** Yarn should be YARN. ** I really struggle with this whole text. What about something like: "This property sets the path from which YARN will read the CGroups configuration. YARN has built-in functionality to discover the system CGroup mount paths, so use this property only if YARN's automatic mount path discovery does not work. The path specified by this property must exist before the NodeManager is launched. If yarn.nodemanager.linux-container-executor.cgroups.mount is set to true, YARN will first try to mount the CGroups at the specified path before reading them. Please refer to NodeManagerCgroups.html in the documentation for further details. This property only applies when the LCE resources handler is set to CgroupsLCEResourcesHandler." * In {{NodeManagerCgroups.md}}: ** In the definition of yarn.nodemanager.linux-container-executor.cgroups.mount-path, I think you're better off just saying, "Optional. See _CGroups mount options_ below for details." ** Yarn should be YARN. ** "leave other settings default" => "leave other settings set to the defaults" ** "Mount CGroups by Yarn" => "CGroups mounted by YARN" ** "CGroups mounted or linked already but not in `/proc/mounts`" => "CGroups already mounted or linked but not in `/proc/mounts`" > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6757.000.patch, YARN-6757.001.patch, > YARN-6757.002.patch, YARN-6757.003.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105813#comment-16105813 ] Hadoop QA commented on YARN-6757: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 36s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 52s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 53s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 30s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 25s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 66m 11s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6757 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879435/YARN-6757.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle | | uname | Linux f29b42a
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105675#comment-16105675 ] Hadoop QA commented on YARN-6757: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 38s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 54s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 8 unchanged - 15 fixed = 8 total (was 23) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 27s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 11s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 15s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 68m 8s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6757 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879415/YARN-6757.002.patch | | Optional Tests | asflicense compile javac javadoc
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16105581#comment-16105581 ] Miklos Szegedi commented on YARN-6757: -- Thanks. I updated the documentation including the html. I think that is a better way to describe the details than the xml which is not visible for everyone. I left a pointer in the xml. I did not cache {{getValidCGroups()}}, since it is just used one. Caching it would waste memory, although not much. > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6757.000.patch, YARN-6757.001.patch, > YARN-6757.002.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102430#comment-16102430 ] Daniel Templeton commented on YARN-6757: Thanks. A few more comments: * {{CGroupsHandler.getValidCGroups()}} might profit from caching the result; it never changes. * In {{CGroupsHandlerImpl.testManualCgroupSetting()}} and {{TestCgroupsLCEResourcesHandler.testManualCgroupSetting()}}, it would be nice to add a message to the {{assertEquals()}}. Otherwise, looks good. I still don't really understand what it's doing, though, so lets work on the property description, and hopefully once that's clear it'll all make sense. > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Yufei Gu >Priority: Minor > Attachments: YARN-6757.000.patch, YARN-6757.001.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077408#comment-16077408 ] Hadoop QA commented on YARN-6757: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | | {color:green} 1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green} 1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 13m 7s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} compile {color} | {color:green} 8m 26s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} checkstyle {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | | {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 49s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. {color} | | {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} compile {color} | {color:green} 5m 38s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} javac {color} | {color:green} 5m 38s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} checkstyle {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green} 1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green} 1{color} | {color:green} findbugs {color} | {color:green} 2m 20s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:green} 1{color} | {color:green} unit {color} | {color:green} 2m 32s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green} 1{color} | {color:green} unit {color} | {color:green} 13m 17s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green} 1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 62m 55s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6757 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12875999/YARN-6757.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle | | uname | Linux a0a30a72ee99 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 7576a68 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/16318/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/16318/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/1631
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077330#comment-16077330 ] Yufei Gu commented on YARN-6757: Uploaded a new patch. 1. fixed. 2. {{CGroupsHandler.getValidCGroups()}} should be OK since EnumSet keep the enum instead of its name string. It's easier for callers to use name string HashSet. 3. 4. 5. 6. fixed. 7. Yes, it's weird, but we cannot do anything about it. 8. Fixed partially. Need the offline discussion, then I can post a new patch. > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6757.000.patch, YARN-6757.001.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075708#comment-16075708 ] Yufei Gu commented on YARN-6757: Thanks [~templedf]'s review. I will take this as the offline discussion with [~miklos.szeg...@cloudera.com]. Will post the new patch soon. > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6757.000.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075459#comment-16075459 ] Daniel Templeton commented on YARN-6757: Thanks for the patch, [~miklos.szeg...@cloudera.com]. A few comments: # The javadoc description for {{CGroupsHandler.getValidCGroups()}} should end with a period. # {{CGroupsHandler.getValidCGroups()}} seems altogether unnecessary. It's really just doing {{new HashSet<>(Arrays.asList(CGroupsController.values()))}} long hand. Maybe even better would be to use an {{EnumSet}} instead of a {{HashSet}}. # In {checkConfiguredCGroupPath()}}, {{cGroupMountPathSpecified}} is a little long-winded. {{cGroupMountPath}} is plenty. # {{checkConfiguredCGroupPath()}} isn't a great name. Maybe {{loadConfiguredCGroupPath()}} or {{parseConfiguredCGroupPath}}? # Why does {{CgroupsLCEResourcesHandler}} have an identical copy of {{checkConfiguredCGroupPath()}}? Seems like there should be some code sharing... # Theres now a dead import in {{CgroupsLCEResourcesHandler}}. # What's with the mount path containing directories that contain files whose names are comma-separated lists or cgroups? Is that a normal cgroups thing? Sounds weird to me... # I love that the text in {{yarn-defaults.xml}} is detailed and prescriptive, but it needs some work. There are grammar issues, but first let's tackle the clarity issues. I don't know what it's trying to say. I get that the property sets a path that we'll use to resolve cgroups under certain circumstances, but it's not clear what those circumstances are and what will happen when this property is set. It also says nothing about the file names that are comma-separated lists of cgroups. If it helps, I'm happy to have an offline conversation and help craft the docs. > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6757.000.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of {{linux-container-executor.cgroups.mount-path}} would look > like: > {noformat} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6757) Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
[ https://issues.apache.org/jira/browse/YARN-6757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070635#comment-16070635 ] Miklos Szegedi commented on YARN-6757: -- See also the discussion in YARN-6515, why we need this. > Refactor the usage of > yarn.nodemanager.linux-container-executor.cgroups.mount-path > -- > > Key: YARN-6757 > URL: https://issues.apache.org/jira/browse/YARN-6757 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha4 >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Attachments: YARN-6757.000.patch > > > We should add the ability to specify a custom cgroup path. This is how the > documentation of inux-container-executor.cgroups.mount-path would look like: > {code} > Requested cgroup mount path. Yarn has built in functionality to discover > the system cgroup mount paths, so use this setting only, if the discovery > does not work. > This path must exist before the NodeManager is launched. > The location can vary depending on the Linux distribution in use. > Common locations include /sys/fs/cgroup and /cgroup. > If cgroups are not mounted, set > yarn.nodemanager.linux-container-executor.cgroups.mount > to true. In this case it specifies, where the LCE should attempt to mount > cgroups if not found. > If cgroups is accessible through lxcfs or some other file system, > then set this path and > yarn.nodemanager.linux-container-executor.cgroups.mount to false. > Yarn tries to use this path first, before any cgroup mount point > discovery. > If it cannot find this directory, it falls back to searching for cgroup > mount points in the system. > Only used when the LCE resources handler is set to the > CgroupsLCEResourcesHandler > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org