[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1450#comment-1450 ] Karthik Kambatla commented on YARN-2194: RHEL7 recommends using systemd, but doesn't require us to do so. As Wei said, we thought it might be better to fix the immediate issue (which Wei claims is a simple fix) instead of switching over to use systemd. We could follow-up with a fix that uses systemd. > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: YARN-2194-1.patch > > >In previous versions of RedHat, we can build custom cgroup hierarchies > with use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533256#comment-14533256 ] Wei Yan commented on YARN-2194: --- [~vinodkv], discussed about this with Karthik offline. We thought it would be better just let the existing cgroups code work well in RedHat7. And after that, we can provide another systemd solution. I can update a patch later. > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: YARN-2194-1.patch > > >In previous versions of RedHat, we can build custom cgroup hierarchies > with use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533089#comment-14533089 ] Vinod Kumar Vavilapalli commented on YARN-2194: --- Also, it will be really useful if we could find a way to support existing code on RHEL7, even if libcgroups is deprecated and the approach ends up becoming heavy-handed, perhaps with some manual steps. This [page|https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Resource_Management_Guide/chap-Using_libcgroup_Tools.html] talks about ways of doing this. > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: YARN-2194-1.patch > > >In previous versions of RedHat, we can build custom cgroup hierarchies > with use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533075#comment-14533075 ] Vinod Kumar Vavilapalli commented on YARN-2194: --- Is there no API that we can use instead of spawning shells to set this up? We should have some auto-detection to chose the right plugin for the right OS. IAC, YARN-3443 changed the way resource isolation code is organized in the ResourceManager. YARN-3542 is migrating existing cgroups+cpu support to the new layout. We need to relook at this patch in light of those changes. > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: YARN-2194-1.patch > > >In previous versions of RedHat, we can build custom cgroup hierarchies > with use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287645#comment-14287645 ] Wei Yan commented on YARN-2194: --- Sure, will update a new patch by combing [~bcwalrus] comments. > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: YARN-2194-1.patch > > >In previous versions of RedHat, we can build custom cgroup hierarchies > with use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287644#comment-14287644 ] Wei Yan commented on YARN-2194: --- Sure, will update a new patch by combing [~bcwalrus] comments. > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: YARN-2194-1.patch > > >In previous versions of RedHat, we can build custom cgroup hierarchies > with use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287446#comment-14287446 ] Karthik Kambatla commented on YARN-2194: container-executor.c - the new method significantly duplicates the existing one. Can we have separate methods to capture the differences and leave the original method as is. > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: YARN-2194-1.patch > > >In previous versions of RedHat, we can build custom cgroup hierarchies > with use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14185368#comment-14185368 ] bc Wong commented on YARN-2194: --- container-executor.c * L1188: If initialize_user() fails, do you not need to cleanup? * L1194: Same for create_log_dirs(). Seems that goto cleanup is still warranted. * L1207: Missing space before S_IRWXU. * L1243: Nit. Hardcoding 55 here is error-prone. You could allocate a 4K buffer here, and use snprintf. * L1244: You need to check the return value from malloc(). Since you're running as root here, everything has to be extra careful. * L1255: On failure, would log the command being executed. > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: YARN-2194-1.patch > > > In previous versions of RedHat, we can build custom cgroup hierarchies with > use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178586#comment-14178586 ] Wei Yan commented on YARN-2194: --- Thanks your comments, [~beckham007]. bq. startSystemdSlice/stopSystemdSlice needs root privilege? Yes, "systemctl start/stop slice" needs root privilege. bq. Let container-executor to run "sudo systemctl start "? You mean adding start/stop slice function in the container-executor, and let SystemdLCEResourceHandler invokes these functions? > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: YARN-2194-1.patch > > > In previous versions of RedHat, we can build custom cgroup hierarchies with > use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14177823#comment-14177823 ] Beckham007 commented on YARN-2194: -- startSystemdSlice/stopSystemdSlice needs root privilege? Let container-executor to run "sudo systemctl start " ? > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > Attachments: YARN-2194-1.patch > > > In previous versions of RedHat, we can build custom cgroup hierarchies with > use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14049593#comment-14049593 ] Beckham007 commented on YARN-2194: -- +1. A new LCEResourceHandler is needed. To support more resource isolation, we also need to have init(), preExecute() and postExecute() for different resource. Adding an abstract CgroupsResourceManager and its implement CPUResourceManager\MemResourceManager is good. > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > > In previous versions of RedHat, we can build custom cgroup hierarchies with > use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045110#comment-14045110 ] Wei Yan commented on YARN-2194: --- oh, yes, adding a new LCEResourceHandler to handle the preExecute and afterExecute is enough. > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > > In previous versions of RedHat, we can build custom cgroup hierarchies with > use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045105#comment-14045105 ] Alejandro Abdelnur commented on YARN-2194: -- i was not meaning autodetection, i was meaning that a new resource handler may be enough to deal with cgroups in RH7, without having to write a new LCE. > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > > In previous versions of RedHat, we can build custom cgroup hierarchies with > use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045101#comment-14045101 ] Wei Yan commented on YARN-2194: --- [~tucu00], you mean just detect whether the system is RedHat 7 inside the CgroupsLCEResourceHandler? If so, use systemd to execute? > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > > In previous versions of RedHat, we can build custom cgroup hierarchies with > use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[
https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14045097#comment-14045097
]
Alejandro Abdelnur commented on YARN-2194:
--
do we need to have a special container executor? Or a specialized
{{LCEResourcesHandler}} would do the trick?
> Add Cgroup support for RedHat 7
> ---
>
> Key: YARN-2194
> URL: https://issues.apache.org/jira/browse/YARN-2194
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Wei Yan
>Assignee: Wei Yan
>
> In previous versions of RedHat, we can build custom cgroup hierarchies with
> use of the cgconfig command from the libcgroup package. From RedHat 7,
> package libcgroup is deprecated and it is not recommended to use it since it
> can easily create conflicts with the default cgroup hierarchy. The systemd is
> provided and recommended for cgroup management. We need to add support for
> this.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041263#comment-14041263 ] Wei Yan commented on YARN-2194: --- SystemdNspawnContainerExectuor is a good idea. Just add one for systemd besides the standard CgroupsLCEHandler. > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > > In previous versions of RedHat, we can build custom cgroup hierarchies with > use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2194) Add Cgroup support for RedHat 7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14041255#comment-14041255 ] Jon Bringhurst commented on YARN-2194: -- It might also be useful to have a "SystemdNspawnContainerExectuor" for yarn.nodemanager.container-executor.class. I don't know how many people would be interesting it using it however. > Add Cgroup support for RedHat 7 > --- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan > > In previous versions of RedHat, we can build custom cgroup hierarchies with > use of the cgconfig command from the libcgroup package. From RedHat 7, > package libcgroup is deprecated and it is not recommended to use it since it > can easily create conflicts with the default cgroup hierarchy. The systemd is > provided and recommended for cgroup management. We need to add support for > this. -- This message was sent by Atlassian JIRA (v6.2#6252)
