Re: cgroups suggestions
It turns out everything works if you set permissions appropriately of $CGROUP_ROOT/mesos/$TASKID/ so the yarn user can write to the hierarchy. Then all works exactly as expected. I spent a while running through the container-executor code and when it mounts a cgroup subsystem it changes the ownership of the hierarchy to the yarn user, the original cgroups code of myriad attempted to do something similar by chmoding the directory but assumed the yarn user work be a member of group root, also when the code was written the chmod happened as root, currently that is ineffective as the standard framework user does not necessarily have permission to modify $CGROUP_ROOT/mesos/$TASKID. However, we have a mechanism for using a frameworksuperuser which can do this (my current hack). The current code also sets yarn.nodemanager.linux-container-executor.cgroups.mount-path=/sys/fs/cgroup and yarn.nodemanager.linux-container-executor.cgroups.mount=true, the documentation the requires edits to yarn-site.xml to get these passed through. Now that I've got things working, I'll start cleaning up the original code to provide an cleaner setup and adjust the documentation as necessary, I should have a PR soon.
Re: cgroups suggestions
Santosh, that is the behavior I'm seeing. On May 4, 2016 6:13 PM, "Santosh Marella"wrote: > > The second involves the cgroup hierarchy and the cgroup mount point. > Here > > the code attempts to create a hierarchy in $CGROUP_DIR/mesos/$TASK_ID. > > This is problematic as mesos will not unmount the hierarchy when the task > > finished (in this case the node manager) > > IIRC, when a task is launched by mesos, the agent creates > $CGROUP_DIR/mesos/$TASK_ID mount point to enforce cpu/mem for that task. > Once the task finishes, the agent should unmount the $TASK_ID. Are you > saying > that's not happening for NMs ? > > Santosh > > On Wed, May 4, 2016 at 10:30 AM, Darin Johnson > wrote: > > > I've been digging into groups support, there's a few things that are easy > > fixes but a few things become problematic so I'd like to discuss. > > > > First the code makes certain options dictated that can be placed in the > > yarn-site.xml - this should be done to remove code and provide > > flexibility. That's easy. > > > > The second involves the cgroup hierarchy and the cgroup mount point. > Here > > the code attempts to create a hierarchy in $CGROUP_DIR/mesos/$TASK_ID. > > This is problematic as mesos will not unmount the hierarchy when the task > > finished (in this case the node manager), it is also therefore unable to > > unmount it's own task hierarchy (This also creates the need to chmod a > > number of directories as a superuser). This leads to issues. An > > alternative approach would be to use the container-executor program > > (already suid w/ yarn's group) to create the hierarchy as > > $CGROUP_DIR/frameworkname if it doesn't exist, this may open another can > of > > worms as I haven't tested fully. > > > > Any thoughts or suggestions would be appreciated. > > > > Darin > > >
Re: cgroups suggestions
> The second involves the cgroup hierarchy and the cgroup mount point. Here > the code attempts to create a hierarchy in $CGROUP_DIR/mesos/$TASK_ID. > This is problematic as mesos will not unmount the hierarchy when the task > finished (in this case the node manager) IIRC, when a task is launched by mesos, the agent creates $CGROUP_DIR/mesos/$TASK_ID mount point to enforce cpu/mem for that task. Once the task finishes, the agent should unmount the $TASK_ID. Are you saying that's not happening for NMs ? Santosh On Wed, May 4, 2016 at 10:30 AM, Darin Johnsonwrote: > I've been digging into groups support, there's a few things that are easy > fixes but a few things become problematic so I'd like to discuss. > > First the code makes certain options dictated that can be placed in the > yarn-site.xml - this should be done to remove code and provide > flexibility. That's easy. > > The second involves the cgroup hierarchy and the cgroup mount point. Here > the code attempts to create a hierarchy in $CGROUP_DIR/mesos/$TASK_ID. > This is problematic as mesos will not unmount the hierarchy when the task > finished (in this case the node manager), it is also therefore unable to > unmount it's own task hierarchy (This also creates the need to chmod a > number of directories as a superuser). This leads to issues. An > alternative approach would be to use the container-executor program > (already suid w/ yarn's group) to create the hierarchy as > $CGROUP_DIR/frameworkname if it doesn't exist, this may open another can of > worms as I haven't tested fully. > > Any thoughts or suggestions would be appreciated. > > Darin >
cgroups suggestions
I've been digging into groups support, there's a few things that are easy fixes but a few things become problematic so I'd like to discuss. First the code makes certain options dictated that can be placed in the yarn-site.xml - this should be done to remove code and provide flexibility. That's easy. The second involves the cgroup hierarchy and the cgroup mount point. Here the code attempts to create a hierarchy in $CGROUP_DIR/mesos/$TASK_ID. This is problematic as mesos will not unmount the hierarchy when the task finished (in this case the node manager), it is also therefore unable to unmount it's own task hierarchy (This also creates the need to chmod a number of directories as a superuser). This leads to issues. An alternative approach would be to use the container-executor program (already suid w/ yarn's group) to create the hierarchy as $CGROUP_DIR/frameworkname if it doesn't exist, this may open another can of worms as I haven't tested fully. Any thoughts or suggestions would be appreciated. Darin