Re: [Lxc-users] lxc-start leaves temporary pivot dir behind
Daniel Lezcano daniel.lezc...@free.fr writes: Ferenc Wagner wrote: While playing with lxc-start, I noticed that /tmp is infested by empty lxc-r* directories: [...] Ok, this name comes from lxc-rootfs in conf.c:setup_rootfs. After setup_rootfs_pivot_root returns, the original /tmp is not available anymore, so rmdir(tmpname) at the bottom of setup_rootfs can't achieve much. Why is this temporary name needed anyway? Is pivoting impossible without it? That was put in place with chroot, before pivot_root, so the distro's scripts can remount their '/' without failing. Now we have pivot_root, I suppose we can change that to something cleaner... Like simply nuking it? Shall I send a patch? -- Feri. -- ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Resources sharing and limit
Yanick Emiliano wrote: Hi everybody, I have just started playing with lxc and having some difficulties to set cpu and memory on my guests. After my searches, seems that resources controlling is managing in cgroup files and I think that I missed something or I didn't understand how deal with cgroup. After reading cgroup documentation, I understand that: - *cpuset.cpus* indicate to a container the number of cpu available No exactly, it's a mask of usable cpus for the container. Let's imagine you have a 16 cpus machine. The content of cpuset.cpus will be, 0-15 which means cpu number 0 to cpu number 15 is used by the cgroup. If you want to assign the cpu 1 (second cpu) to the container, you have to set it by echo 1 /cgroup/name/cpuset.cpus. If you want to assign the cpu 1,2,3 to the container, echo 1,2,3 /cgroup/name/cpuset.cpus. If you want to assign cpu 0 up to 7 to the container, echo 0-7 /cgroup/name/cpuset.cpus. In the context of lxc. lxc-execute -n foo -s lxc.cgroup.cpuset.cpus=1,2,3 myforks etc ... -*cpuset.cpu_exclusive* limit the number of cpu which the container can use. Am I in good way? When you assigned the cpus to the container, the processes of the container will run on these cpus only but that does not prevent the other tasks of the system to run on these cpus. If you want the cpus to be used by the container *only*, set them 'exclusive'. This is what I understood. For example , can I tell to my container that there are 2 cpu available (* cpuset.cpus)*, but use one generally (*puset.cpu_exclusive)*, use the second one only when it's necessary (when there are a lot of application to run)? cpu on demand ? :) Hey, externally look at the cpu usage of the container, when it reach a threshold you define, assign another cpu to the container. What I want is manage QoS with my containers. Very likely, you are looking for the cgroup fair scheduler, it would be better than dynamically assign cpus to the container, IMHO. http://lwn.net/Articles/240474/ It's /cgroup/name/cpu.shares Create 2 containers, lxc-execute -n foo -s lxc.cgroup.cpu.shares=1 /bin/bash in another shell lxc-execute -n bar -s /bin/bash in both shell, do while $(true); do echo -n . ; done You will see foo displaying the dots vry slowly and bar being at the normal speed. As soon as bar exits or is frozen (via lxc-freeze), foo works at normal speed as it is no longer competing the cpu with bar. You can dynamically change the priority of the container with lxc-cgroup -n foo cpu.shares=1024 for example. And my last question is, Can I do the same thing with memory sharing? memory on demand :) I will let someone else to add comments here, as I am not very familiar with memory vs cgroup. Thanks -- Daniel -- ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxc-start leaves temporary pivot dir behind
Ferenc Wagner wrote: Daniel Lezcano daniel.lezc...@free.fr writes: Ferenc Wagner wrote: While playing with lxc-start, I noticed that /tmp is infested by empty lxc-r* directories: [...] Ok, this name comes from lxc-rootfs in conf.c:setup_rootfs. After setup_rootfs_pivot_root returns, the original /tmp is not available anymore, so rmdir(tmpname) at the bottom of setup_rootfs can't achieve much. Why is this temporary name needed anyway? Is pivoting impossible without it? That was put in place with chroot, before pivot_root, so the distro's scripts can remount their '/' without failing. Now we have pivot_root, I suppose we can change that to something cleaner... Like simply nuking it? Shall I send a patch? Sure, if we can kill it, I will be glad to take your patch :) -- ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxc-unshare woes and signal forwarding in lxc-start
Daniel Lezcano daniel.lezc...@free.fr writes: Ferenc Wagner wrote: I'd like to use lxc-start as a wrapper, invisible to the parent and the (jailed) child. Of course I could hack around this by not exec-ing lxc-start but keeping the shell around, trap all signals and lxc-killing them forward. But it's kind of ugly in my opinion. Ok, got it. I think that makes sense to forward the signals, especially for job management. What signals do you want to forward? Basically all of them. I couldn't find a definitive list of signals used for job control in SGE, but the following is probably a good approximation: SIGTTOU, SIGTTIN, SIGUSR1, SIGUSR2, SIGCONT, SIGWINCH and SIGTSTP. This is application specific, though, lxc-start shouldn't have this hard-coded. Looking at the source, the SIGCHLD mechanism could be mimicked, but LXC_TTY_ADD_HANDLER may get in the way. I'm also worried about signals sent to the whole process group: they may be impossible to distinguish from the targeted signals and thus can't propagate correctly. -- Thanks, Feri. -- ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxc-start leaves temporary pivot dir behind
Daniel Lezcano dlezc...@fr.ibm.com writes: Ferenc Wagner wrote: Daniel Lezcano daniel.lezc...@free.fr writes: Ferenc Wagner wrote: While playing with lxc-start, I noticed that /tmp is infested by empty lxc-r* directories: [...] Ok, this name comes from lxc-rootfs in conf.c:setup_rootfs. After setup_rootfs_pivot_root returns, the original /tmp is not available anymore, so rmdir(tmpname) at the bottom of setup_rootfs can't achieve much. Why is this temporary name needed anyway? Is pivoting impossible without it? That was put in place with chroot, before pivot_root, so the distro's scripts can remount their '/' without failing. Now we have pivot_root, I suppose we can change that to something cleaner... Like simply nuking it? Shall I send a patch? Sure, if we can kill it, I will be glad to take your patch :) I can't see any reason why lxc-start couldn't do without that temporary recursive bind mount of the original root. If neither do you, I'll patch it out and see if it still flies. -- Thanks, Feri. -- ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxc-unshare woes and signal forwarding in lxc-start
Ferenc Wagner wrote: Daniel Lezcano daniel.lezc...@free.fr writes: Ferenc Wagner wrote: I'd like to use lxc-start as a wrapper, invisible to the parent and the (jailed) child. Of course I could hack around this by not exec-ing lxc-start but keeping the shell around, trap all signals and lxc-killing them forward. But it's kind of ugly in my opinion. Ok, got it. I think that makes sense to forward the signals, especially for job management. What signals do you want to forward? Basically all of them. I couldn't find a definitive list of signals used for job control in SGE, but the following is probably a good approximation: SIGTTOU, SIGTTIN, SIGUSR1, SIGUSR2, SIGCONT, SIGWINCH and SIGTSTP. Yes, that could be a good starting point. I was wondering about SIGSTOP being sent to lxc-start which is not forwardable of course, is it a problem ? This is application specific, though, lxc-start shouldn't have this hard-coded. Ok, from the configuration then. Looking at the source, the SIGCHLD mechanism could be mimicked, but LXC_TTY_ADD_HANDLER may get in the way. We should remove LXC_TTY_ADD_HANDLER and do everything in the signal handler of SIGCHLD by extending the handler. I have a pending fix changing a bit the signal handler function. I'm also worried about signals sent to the whole process group: they may be impossible to distinguish from the targeted signals and thus can't propagate correctly. Good point. Maybe we can setpgrp the first process of the container ? -- ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxc-start leaves temporary pivot dir behind
Ferenc Wagner wf...@niif.hu writes:
Daniel Lezcano dlezc...@fr.ibm.com writes:
Ferenc Wagner wrote:
Daniel Lezcano daniel.lezc...@free.fr writes:
Ferenc Wagner wrote:
While playing with lxc-start, I noticed that /tmp is infested by
empty lxc-r* directories: [...] Ok, this name comes from lxc-rootfs
in conf.c:setup_rootfs. After setup_rootfs_pivot_root returns, the
original /tmp is not available anymore, so rmdir(tmpname) at the
bottom of setup_rootfs can't achieve much. Why is this temporary
name needed anyway? Is pivoting impossible without it?
That was put in place with chroot, before pivot_root, so the distro's
scripts can remount their '/' without failing.
Now we have pivot_root, I suppose we can change that to something
cleaner...
Like simply nuking it? Shall I send a patch?
Sure, if we can kill it, I will be glad to take your patch :)
I can't see any reason why lxc-start couldn't do without that temporary
recursive bind mount of the original root. If neither do you, I'll
patch it out and see if it still flies.
For my purposes the patch below works fine. I only run applications,
though, not full systems, so wider testing is definitely needed.
Thanks,
Feri.
From 98b24c13f809f18ab8969fb4d84defe6f812b25c Mon Sep 17 00:00:00 2001
From: Ferenc Wagner wf...@niif.hu
Date: Thu, 6 May 2010 14:47:39 +0200
Subject: [PATCH] no need to use a temporary directory for pivoting
That was put in place before lxc-start started using pivot_root, so
the distro scripts can remount / without problems.
Signed-off-by: Ferenc Wagner wf...@niif.hu
---
src/lxc/conf.c | 28 +++-
1 files changed, 3 insertions(+), 25 deletions(-)
diff --git a/src/lxc/conf.c b/src/lxc/conf.c
index b27a11d..4379a32 100644
--- a/src/lxc/conf.c
+++ b/src/lxc/conf.c
@@ -588,37 +588,15 @@ static int setup_rootfs_pivot_root(const char *rootfs,
const char *pivotdir)
static int setup_rootfs(const char *rootfs, const char *pivotdir)
{
- char *tmpname;
- int ret = -1;
-
if (!rootfs)
return 0;
- tmpname = tempnam(/tmp, lxc-rootfs);
- if (!tmpname) {
- SYSERROR(failed to generate temporary name);
- return -1;
- }
-
- if (mkdir(tmpname, 0700)) {
- SYSERROR(failed to create temporary directory '%s', tmpname);
- return -1;
- }
-
- if (mount(rootfs, tmpname, none, MS_BIND|MS_REC, NULL)) {
- SYSERROR(failed to mount '%s'-'%s', rootfs, tmpname);
- goto out;
- }
-
- if (setup_rootfs_pivot_root(tmpname, pivotdir)) {
+ if (setup_rootfs_pivot_root(rootfs, pivotdir)) {
ERROR(failed to pivot_root to '%s', rootfs);
- goto out;
+ return -1;
}
- ret = 0;
-out:
- rmdir(tmpname);
- return ret;
+ return 0;
}
static int setup_pts(int pts)
--
1.6.5
--
___
Lxc-users mailing list
Lxc-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxc-start leaves temporary pivot dir behind
Ferenc Wagner wrote:
Ferenc Wagner wf...@niif.hu writes:
Daniel Lezcano dlezc...@fr.ibm.com writes:
Ferenc Wagner wrote:
Daniel Lezcano daniel.lezc...@free.fr writes:
Ferenc Wagner wrote:
While playing with lxc-start, I noticed that /tmp is infested by
empty lxc-r* directories: [...] Ok, this name comes from lxc-rootfs
in conf.c:setup_rootfs. After setup_rootfs_pivot_root returns, the
original /tmp is not available anymore, so rmdir(tmpname) at the
bottom of setup_rootfs can't achieve much. Why is this temporary
name needed anyway? Is pivoting impossible without it?
That was put in place with chroot, before pivot_root, so the distro's
scripts can remount their '/' without failing.
Now we have pivot_root, I suppose we can change that to something
cleaner...
Like simply nuking it? Shall I send a patch?
Sure, if we can kill it, I will be glad to take your patch :)
I can't see any reason why lxc-start couldn't do without that temporary
recursive bind mount of the original root. If neither do you, I'll
patch it out and see if it still flies.
For my purposes the patch below works fine. I only run applications,
though, not full systems, so wider testing is definitely needed.
Thanks,
Feri.
From 98b24c13f809f18ab8969fb4d84defe6f812b25c Mon Sep 17 00:00:00 2001
From: Ferenc Wagner wf...@niif.hu
Date: Thu, 6 May 2010 14:47:39 +0200
Subject: [PATCH] no need to use a temporary directory for pivoting
That was put in place before lxc-start started using pivot_root, so
the distro scripts can remount / without problems.
Signed-off-by: Ferenc Wagner wf...@niif.hu
---
src/lxc/conf.c | 28 +++-
1 files changed, 3 insertions(+), 25 deletions(-)
diff --git a/src/lxc/conf.c b/src/lxc/conf.c
index b27a11d..4379a32 100644
--- a/src/lxc/conf.c
+++ b/src/lxc/conf.c
@@ -588,37 +588,15 @@ static int setup_rootfs_pivot_root(const char *rootfs,
const char *pivotdir)
static int setup_rootfs(const char *rootfs, const char *pivotdir)
{
- char *tmpname;
- int ret = -1;
-
if (!rootfs)
return 0;
- tmpname = tempnam(/tmp, lxc-rootfs);
- if (!tmpname) {
- SYSERROR(failed to generate temporary name);
- return -1;
- }
-
- if (mkdir(tmpname, 0700)) {
- SYSERROR(failed to create temporary directory '%s', tmpname);
- return -1;
- }
-
- if (mount(rootfs, tmpname, none, MS_BIND|MS_REC, NULL)) {
- SYSERROR(failed to mount '%s'-'%s', rootfs, tmpname);
- goto out;
- }
-
- if (setup_rootfs_pivot_root(tmpname, pivotdir)) {
+ if (setup_rootfs_pivot_root(rootfs, pivotdir)) {
ERROR(failed to pivot_root to '%s', rootfs);
- goto out;
+ return -1;
}
- ret = 0;
-out:
- rmdir(tmpname);
- return ret;
+ return 0;
}
static int setup_pts(int pts)
Thanks, I will test it with another patch I have in my backlog fixing
the pivot_root. I Cc'ed the lxc-devel mailing list which is more
adequate for this kind of discussion.
Thanks again.
-- Daniel
--
___
Lxc-users mailing list
Lxc-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] lxc-unshare woes and signal forwarding in lxc-start
Ferenc Wagner wrote: Daniel Lezcano daniel.lezc...@free.fr writes: Ferenc Wagner wrote: Daniel Lezcano daniel.lezc...@free.fr writes: Ferenc Wagner wrote: I'd like to use lxc-start as a wrapper, invisible to the parent and the (jailed) child. Of course I could hack around this by not exec-ing lxc-start but keeping the shell around, trap all signals and lxc-killing them forward. But it's kind of ugly in my opinion. Ok, got it. I think that makes sense to forward the signals, especially for job management. What signals do you want to forward? Basically all of them. I couldn't find a definitive list of signals used for job control in SGE, but the following is probably a good approximation: SIGTTOU, SIGTTIN, SIGUSR1, SIGUSR2, SIGCONT, SIGWINCH and SIGTSTP. Yes, that could be a good starting point. I was wondering about SIGSTOP being sent to lxc-start which is not forwardable of course, is it a problem ? I suppose not, SIGSTOP and SIGKILL are impossible to use in application- specific ways. On the other hand, SIGXCPU and SIGXFSZ should probably be forwarded, too. Naturally, this business can't be perfected, but a good enough solution could still be valuable. Agree. Looking at the source, the SIGCHLD mechanism could be mimicked, but LXC_TTY_ADD_HANDLER may get in the way. We should remove LXC_TTY_ADD_HANDLER and do everything in the signal handler of SIGCHLD by extending the handler. I have a pending fix changing a bit the signal handler function. Could you please send along your pending fix? I'd like to experiment with signal forwarding, but without stomping on that. Sure, no problem. I noticed something strange: # lxc-start -n jail -s lxc.mount.entry=/ /tmp/jail none bind 0 0 -s lxc.rootfs=/tmp/jail -s lxc.pivotdir=/mnt /bin/sleep 1000 (in another terminal) # lxc-ps --lxc CONTAINERPID TTY TIME CMD jail4173 pts/100:00:00 sleep # kill 4173 (this does not kill the sleep!) # strace -p 4173 Process 4173 attached - interrupt to quit restart_syscall(... resuming interrupted call ... = ? ERESTART_RESTARTBLOCK (To be restarted) --- SIGTERM (Terminated) @ 0 (0) --- Process 4173 detached # lxc-ps --lxc CONTAINERPID TTY TIME CMD jail4173 pts/100:00:00 sleep # fgrep -i sig /proc/4173/status SigQ: 1/16382 SigPnd: SigBlk: SigIgn: SigCgt: # kill -9 4173 That is, the jailed sleep process could be killed by SIGKILL only, even though (according to strace) SIGTERM was delivered and it isn't handled specially. Why does this happen? I sent a separate email for this problem in order to avoid confusion with the signal forwarding discussion. I'm also worried about signals sent to the whole process group: they may be impossible to distinguish from the targeted signals and thus can't propagate correctly. Good point. Maybe we can setpgrp the first process of the container? We've got three options: A) do nothing, as now B) forward to our child C) forward to our child's process group The signal could arrive because it was sent to 1) the PID of lxc-start 2) the process group of lxc-start If we don't put the first process of the container into a new process group (as now), this is what happens: AB C 1 swallowedOKothers also killed 2 OK child gets extraeverybody gets extra If we put the first process of the container into a new process group: AB C 1 swallowedOKothers also killed 2 swallowed only the child killed OK Neither is a clear winner, although the latter is somewhat more symmetrical. I'm not sure about wanting all this configurable... hmm ... Maybe Greg, (it's an expert with signals and processes), has an idea on how to deal with that. -- Daniel -- ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
