Re: [Lxc-users] Kernel 2.6.33-rc6, 3 bugs container specific.
Serge E. Hallyn wrote: > Quoting Jean-Marc Pigeon (j...@safe.ca): >> Hello, >> >> >>> I was wondering out loud about the best design to solve his problem. >>> >>> If we try to redirect kernel-generated messages to containers, we have >>> several problems, including whether we need to duplicate the messages >>> to the host container. So in one sense it seems more flexible to >>> 1. send everything to host syslog >> No, if we do that all CONTs message will reach >> the same bucket and it will be difficult to sort >> them out.. >> CONT sys_admin and HOST sys_admin could be different >> "entity", so you debug CONT config and critical >> needed information reach HOST (which you do not >> have access to). > > Yes, so a privileged task on HOST must pass that information back to > you on CONT. That is not a valid complaint imo. But how to sort the > msgs out is a valid question. > > We need some sort of identifier, unique system-wide, attached to.. something. > Is ifindex unique system-wide right now? Oh, IIRC it is, but we wnat it to > be containerized, so that would be a bad choice :) > >>> 2. clamp down on syslog use by processes not in the init_user_ns >> Could give me more detail??... > > Simplest choices would be to just refuse sys_syslog() and open(/proc/kmsg) > altogether from a container, or to only allow reading/writing messages > to own syslog. (I had hoped to find time to try out the second option but > simply haven't had the time, and it doesn't look like I will very soon. > So if anyone else wants to, pls jump at it...) > > Then /proc/kmsg can provide what I described above through a FUSE file, > and if, as you mentioned, the container unmounts the FUSE fs and gets > to real procfs, they just get nothing. > >>> 3. let the userspace on the host copy messages into a socket or >>>file so child container can pretend it has real syslog. >> So you trap printk message from CONT on the HOST and >> redirect them on CONT but on a standard syslog channel. >> Seem OK to me, as long /proc/kmsg is not existing >> (/dev/null) in the CONT file tree. We have: * Commands to sys_syslog: * * 0 -- Close the log. Currently a NOP. * 1 -- Open the log. Currently a NOP. * 2 -- Read from the log. * 3 -- Read all messages remaining in the ring buffer. * 4 -- Read and clear all messages remaining in the ring buffer * 5 -- Clear ring buffer. * 6 -- Disable printk to console * 7 -- Enable printk to console * 8 -- Set level of messages printed to console * 9 -- Return number of unread characters in the log buffer * 10 -- Return size of the log buffer And add: * 11 -- create a new ring buffer for the current process and its childs We have, let's say a global ring buffer keep untouched, used by syslog(2) and printk. When we create a new ring buffer, we allocate it and assign to the nsproxy (global ring buffer is the default in the nsproxy). The prink keeps writing in the global ring buffer and the syslog(2) writes to the "namespaced" ring buffer. Does it makes sense ? -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Kernel 2.6.33-rc6, 3 bugs container specific.
Hello [..] > > > We have: > * Commands to sys_syslog: > * > * 0 -- Close the log. Currently a NOP. > * 1 -- Open the log. Currently a NOP. > * 2 -- Read from the log. > * 3 -- Read all messages remaining in the ring buffer. > * 4 -- Read and clear all messages remaining in the ring buffer > * 5 -- Clear ring buffer. > * 6 -- Disable printk to console > * 7 -- Enable printk to console > * 8 -- Set level of messages printed to console > * 9 -- Return number of unread characters in the log buffer > * 10 -- Return size of the log buffer > > And add: >* 11 -- create a new ring buffer for the current process and > its childs > > > We have, let's say a global ring buffer keep untouched, used by > syslog(2) and printk. When we create a new ring buffer, we allocate it > and assign to the nsproxy (global ring buffer is the default in the > nsproxy). > > The prink keeps writing in the global ring buffer and the syslog(2) > writes to the "namespaced" ring buffer. > > Does it makes sense ? I like this idea, as it give us flexibility. Caution: iptable packet log is using printk (AFIK) to do tracing. iptable can be used within CONT: above the iptable within HOST: (ip filtering superset). So CONT:printk must be writing too in the "namespaced" ring buffer. -- A bientôt == Jean-Marc Pigeon Internet: j...@safe.ca SAFE Inc. Phone: (514) 493-4280 Fax: (514) 493-1946 Clement, 'a kiss solution' to get rid of SPAM (at last) Clement' Home base <"http://www.clement.safe.ca";> == -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Executing a command inside a running container?
Hey hey...
On Sun, 2010-01-31 at 13:38 -0600, Tony Risinger wrote:
> On Sat, Jan 30, 2010 at 12:11 PM, Michael H. Warfield
> wrote:
> > On Sat, 2010-01-30 at 14:20 +0100, Dominik Schulz wrote:
> >> Hi,
> >>
> >> I'm fairly new to LXC and I am looking for a way to execute a command
> >> inside a
> >> running container (a full blow one with its own rootfs and full isolation).
> >> lxc-execute doesn't seem to do the trick and lxc-console requires
> >> credential
> >> to login. I'm looking for a way to execute command w/o having to login via
> >> ssh
> >> or the lxc-console, so I can execute command directly inside the
> >> containers to
> >> shut them down properly.
> >
> > This is being worked on and looked at now through the use of some sort
> > of daemon in the running container but is not currently possible. More
> > over, cgroups in the kernel does not currently support joining an active
> > cgroup, and may never support it, so it would have to be done through
> > the mediation of some sort of daemon in the running container and
> > communicating with the host. There's been discussion on the -devel list
> > over the optimal way to do it and there are some patches running around
> > but AFAICT nothing has really been committed or committed to at this
> > time.
> if your using the standard init program, and you are only trying to
> control stutdown/reboot, i use something like this in my container
> inittab:
> p6::ctrlaltdel:/sbin/init 6
> p0::powerfail:/sbin/init 0
A lot of systems, like Fedora, are now using upstart. So, on fedora,
you've got /etc/event.d/control-alt-delete for the ctrlaltdel condition.
By default it's set to:
/sbin/shutdown -r now "Control-Alt-Delete pressed"
Not sure where it's handling powerfail.
Some of my older systems have this in inittab:
ca::ctrlaltdel:/sbin/shutdown -t3 -r now
pf::powerfail:/sbin/shutdown -f -h +2 "Power Failure; System Shutting Down"
That's a 2 minute delay in the shutdown. Yeah, I could modify those
lines to be the init commands, but that won't work in the upstart
systems.
My Ubuntu 9.04 container has this for powerfail related stuff:
# What to do when the power fails/returns.
pf::powerwait:/etc/init.d/powerfail start
pn::powerfailnow:/etc/init.d/powerfail now
po::powerokwait:/etc/init.d/powerfail stop
Ok... Not sure about that. At least the ctrlaltdel looks good.
> ctrlaltdel responds to a SIGINT, and powerfail responds to SIGPWR.
> this lets you send a:
> kill -INT
> to reboot, and:
> kill -PWR
Since this is all being run in the host environment, anyways... Another
trick, which I'm now incorporating into the monitory code described
below, is to set a file /var/lib/lxc/{VM}/halt_on_reboot if I want to
halt the container. THEN hit it with the -INT signal, which causes it
to start an immediate orderly shut down in either default case. We can
then take advantage of the hang at the end and the monitor code below
(or some variation there of) to decide to halt or reboot. Modify the
reboot case in the code below to include an if check on that file. If
it's there, remove the file but don't restart the container after
stopping it.
One gotcha in this is that we don't readily know the host pid of the
init process in the container. My startup scripts are now snarfing that
down right after starting the container and pawing through the tasks
file and /proc/{pid}/cmdline looking for init. That pid gets dumped
in /var/lib/lxc/{VM}/pid where we can reference it later. It really
shouldn't change over the life of the container.
Now I can have an lxc-shutdown command that snatches that pid, sets that
file, and issues the -INT and it should force the container into an
orderly shutdown. I like this.
> to shutdown. you will also need the help of a "monitor" process to
> overcome the fact that the init process will hang around, even though
> all other processes in the container are dead. an earlier thread
> detailed a nice trick using inotifywait and the utmp file in the
> container, see this thread for ideas:
>
> http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg00040.html
>
> specifically this post on:
>
> http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg00049.html
>
> you basically monitor the utmp file in the container for a change,
> then run "runlevel " to get the runlevel in the container,
> and act appropriately. here is an excerpt from my scripts (originally
> based off one provided in the previous thread), it has variables
> specific to my setup but it's fairly self explanatory:
>
> while true; do
> # time of 5 minutes on it JUST IN CASE...
> vps_utmp=${VPS_DOM}/${vps}/rootfs/var/run/utmp
> inotifywait -qqt 300 ${vps_utmp}
> if [ $(wc -l < ${VPS_VAR}/cgroup/${vps}/tasks) -eq 1 ]; then
>
> runlevel="$(runlevel ${vps_utmp})"
>
> case $runlevel in
> N*)
> # nothing for new boot state
> ;;
>
Re: [Lxc-users] Kernel 2.6.33-rc6, 3 bugs container specific.
Quoting Daniel Lezcano (daniel.lezc...@free.fr): > Serge E. Hallyn wrote: > >Quoting Jean-Marc Pigeon (j...@safe.ca): > >>Hello, > >> > >> > >>>I was wondering out loud about the best design to solve his problem. > >>> > >>>If we try to redirect kernel-generated messages to containers, we have > >>>several problems, including whether we need to duplicate the messages > >>>to the host container. So in one sense it seems more flexible to > >>> 1. send everything to host syslog > >>No, if we do that all CONTs message will reach > >>the same bucket and it will be difficult to sort > >>them out.. > >>CONT sys_admin and HOST sys_admin could be different > >>"entity", so you debug CONT config and critical > >>needed information reach HOST (which you do not > >> have access > >>to). > > > >Yes, so a privileged task on HOST must pass that information back to > >you on CONT. That is not a valid complaint imo. But how to sort the > >msgs out is a valid question. > > > >We need some sort of identifier, unique system-wide, attached to.. something. > >Is ifindex unique system-wide right now? Oh, IIRC it is, but we wnat it to > >be containerized, so that would be a bad choice :) > > > >>> 2. clamp down on syslog use by processes not in the init_user_ns > >>Could give me more detail??... > > > >Simplest choices would be to just refuse sys_syslog() and open(/proc/kmsg) > >altogether from a container, or to only allow reading/writing messages > >to own syslog. (I had hoped to find time to try out the second option but > >simply haven't had the time, and it doesn't look like I will very soon. > >So if anyone else wants to, pls jump at it...) > > > >Then /proc/kmsg can provide what I described above through a FUSE file, > >and if, as you mentioned, the container unmounts the FUSE fs and gets > >to real procfs, they just get nothing. > > > >>> 3. let the userspace on the host copy messages into a socket or > >>> file so child container can pretend it has real syslog. > >>So you trap printk message from CONT on the HOST and > >>redirect them on CONT but on a standard syslog channel. > >>Seem OK to me, as long /proc/kmsg is not existing > >>(/dev/null) in the CONT file tree. > > > We have: >* Commands to sys_syslog: >* >* 0 -- Close the log. Currently a NOP. >* 1 -- Open the log. Currently a NOP. >* 2 -- Read from the log. >* 3 -- Read all messages remaining in the ring buffer. >* 4 -- Read and clear all messages remaining in the ring buffer >* 5 -- Clear ring buffer. >* 6 -- Disable printk to console >* 7 -- Enable printk to console >* 8 -- Set level of messages printed to console >* 9 -- Return number of unread characters in the log buffer >* 10 -- Return size of the log buffer > > And add: > * 11 -- create a new ring buffer for the current process > and its childs > > > We have, let's say a global ring buffer keep untouched, used by > syslog(2) and printk. When we create a new ring buffer, we allocate > it and assign to the nsproxy (global ring buffer is the default in > the nsproxy). > > The prink keeps writing in the global ring buffer and the syslog(2) > writes to the "namespaced" ring buffer. > > Does it makes sense ? Yeah, it's a nice alternative. Though (1) there is something to be said for forcing a new ring buffer upon clone(CLONE_NEWUSER), and (2) assuming the new ring buffer is pointed to from nsproxy, it might be frowned upon to do an unshare/clone action in yet another way. I still think our first concern should be safety, and that we should consider just adding 'struct syslog_struct' to nsproxy, and making that NULL on a clone(CLONE_NEWUSER). any sys_syslog() or /proc/kmsg access returns -EINVAL after that. Then we can discuss whether and how to target printks to namespaces, and whether duplicates should be sent to parent namespaces. After we start getting flexible with syslog, the next request will be for audit flexibility. I don't even know how our netlink support suffices for that right now. (So, this all does turn into a big deal...) -serge -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Kernel 2.6.33-rc6, 3 bugs container specific.
Hello, [...] > > > > The prink keeps writing in the global ring buffer and the syslog(2) > > writes to the "namespaced" ring buffer. > > > > Does it makes sense ? > > Yeah, it's a nice alternative. Though (1) there is something to be said for > forcing a new ring buffer upon clone(CLONE_NEWUSER), and (2) assuming the > new ring buffer is pointed to from nsproxy, it might be frowned upon to do > an unshare/clone action in yet another way. > > I still think our first concern should be safety, and that we should consider > just adding 'struct syslog_struct' to nsproxy, and making that NULL on a > clone(CLONE_NEWUSER). any sys_syslog() or /proc/kmsg access returns -EINVAL > after that. Then we can discuss whether and how to target printks to > namespaces, and whether duplicates should be sent to parent namespaces. /proc/kmsg=-EINVAL will resolve the own HOST: ring buffer corruption not sure what sys_syslog()=-EINVAL mean???, rsyslog MUST be able to run within CONT: right? printk namespaces duplicate and sent to parent namespace is not a good idea (duplicating&forwarding is done by tools as rsyslogd). > > After we start getting flexible with syslog, the next request will be for > audit flexibility. I don't even know how our netlink support suffices for > that right now. > > (So, this all does turn into a big deal...) > > -serge -- A bientôt == Jean-Marc Pigeon Internet: j...@safe.ca SAFE Inc. Phone: (514) 493-4280 Fax: (514) 493-1946 Clement, 'a kiss solution' to get rid of SPAM (at last) Clement' Home base <"http://www.clement.safe.ca";> == -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Kernel 2.6.33-rc6, 3 bugs container specific.
Quoting Jean-Marc Pigeon (j...@safe.ca): > Hello, > > [...] > > > > > > The prink keeps writing in the global ring buffer and the syslog(2) > > > writes to the "namespaced" ring buffer. > > > > > > Does it makes sense ? > > > > Yeah, it's a nice alternative. Though (1) there is something to be said for > > forcing a new ring buffer upon clone(CLONE_NEWUSER), and (2) assuming the > > new ring buffer is pointed to from nsproxy, it might be frowned upon to do > > an unshare/clone action in yet another way. > > > > I still think our first concern should be safety, and that we should > > consider > > just adding 'struct syslog_struct' to nsproxy, and making that NULL on a > > clone(CLONE_NEWUSER). any sys_syslog() or /proc/kmsg access returns -EINVAL > > after that. Then we can discuss whether and how to target printks to > > namespaces, and whether duplicates should be sent to parent namespaces. > /proc/kmsg=-EINVAL will resolve the own HOST: ring buffer corruption > not sure what sys_syslog()=-EINVAL mean???, rsyslog MUST be able to > run within CONT: right? > > printk namespaces duplicate and sent to parent namespace > is not a good idea (duplicating&forwarding is done by tools as > rsyslogd). Heh, agreed, I just disagree that we should trust CONT to fwd stuff to HOST, rather we should have HOST fwd stuff to CONT. It comes down to whether HOST might need the info to determine what CONT or someone attacking CONT is up to. To the extend that HOST will never be fully safe from CONT (and if you think it is, even using kvm/vmware, then you you're deluding yourself) I think it's misguided to keep information from HOST. > > After we start getting flexible with syslog, the next request will be for > > audit flexibility. I don't even know how our netlink support suffices for > > that right now. > > > > (So, this all does turn into a big deal...) > > > > -serge > -- > A bientôt > == > Jean-Marc Pigeon Internet: j...@safe.ca > SAFE Inc. Phone: (514) 493-4280 >Fax: (514) 493-1946 > Clement, 'a kiss solution' to get rid of SPAM (at last) >Clement' Home base <"http://www.clement.safe.ca";> > == -- The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
[Lxc-users] [PATCH RFC][gross hack] containerized syslog
Provide each user namespace with its own syslog ringbuffer.
So you can do
ns_exec -cU /bin/bash
dmesg
and see nothing. Root in a container (with private user namespace)
cannot clear the host's ring buffer.
Since containers do not have a notion of consoles at present,
only the initial user namespace deals with console output or
with the console-related syslog commands.
This opens the door to targetting printk at certain syslog
namespaces. It's not safe to be applied - it's a quick-n-dirty
hack and won't even compile for CONFIG_PRINTK=n. Also I've not decided
what to do about duplication of printks to init_user_ns so for
now emit_one_char always duplicates to inti_user_ns. We probably
want to be smarter about this and output a prefix indicating the
target.
But I figured discussions about the API would be more meaningful
with a testable patch.
---
fs/proc/kmsg.c |5 +-
include/linux/user_namespace.h |2 +
kernel/printk.c| 225 ++--
kernel/user.c |4 +
kernel/user_namespace.c| 13 +++
5 files changed, 168 insertions(+), 81 deletions(-)
diff --git a/fs/proc/kmsg.c b/fs/proc/kmsg.c
index 7ca7834..2746b70 100644
--- a/fs/proc/kmsg.c
+++ b/fs/proc/kmsg.c
@@ -12,11 +12,12 @@
#include
#include
#include
+#include
#include
#include
-extern wait_queue_head_t log_wait;
+extern struct syslog_ns init_syslog_ns;
extern int do_syslog(int type, char __user *bug, int count);
@@ -41,7 +42,7 @@ static ssize_t kmsg_read(struct file *file, char __user *buf,
static unsigned int kmsg_poll(struct file *file, poll_table *wait)
{
- poll_wait(file, &log_wait, wait);
+ poll_wait(file, &init_syslog_ns.wait, wait);
if (do_syslog(9, NULL, 0))
return POLLIN | POLLRDNORM;
return 0;
diff --git a/include/linux/user_namespace.h b/include/linux/user_namespace.h
index cc4f453..3926c89 100644
--- a/include/linux/user_namespace.h
+++ b/include/linux/user_namespace.h
@@ -5,6 +5,7 @@
#include
#include
#include
+#include
#define UIDHASH_BITS (CONFIG_BASE_SMALL ? 3 : 8)
#define UIDHASH_SZ (1 << UIDHASH_BITS)
@@ -14,6 +15,7 @@ struct user_namespace {
struct hlist_head uidhash_table[UIDHASH_SZ];
struct user_struct *creator;
struct work_struct destroyer;
+ struct syslog_ns*syslog;
};
extern struct user_namespace init_user_ns;
diff --git a/kernel/printk.c b/kernel/printk.c
index 1751c45..5b93447 100644
--- a/kernel/printk.c
+++ b/kernel/printk.c
@@ -35,9 +35,18 @@
#include
#include
#include
+#include
#include
+struct syslog_ns init_syslog_ns;
+#define g_log_wait (init_syslog_ns.wait)
+#define g_log_start (init_syslog_ns.start)
+#define g_log_end (init_syslog_ns.end)
+#define g_log_buf_len (init_syslog_ns.buf_len)
+#define g_logged_chars (init_syslog_ns.logged_chars)
+#define g_log_buf (init_syslog_ns.buf)
+
/*
* for_each_console() allows you to iterate on each console
*/
@@ -52,6 +61,7 @@ void asmlinkage __attribute__((weak)) early_printk(const char
*fmt, ...)
}
#define __LOG_BUF_LEN (1 << CONFIG_LOG_BUF_SHIFT)
+#define CONTAINER_BUF_LEN 4096
/* printk's without a loglevel use this.. */
#define DEFAULT_MESSAGE_LOGLEVEL 4 /* KERN_WARNING */
@@ -60,8 +70,6 @@ void asmlinkage __attribute__((weak)) early_printk(const char
*fmt, ...)
#define MINIMUM_CONSOLE_LOGLEVEL 1 /* Minimum loglevel we let people use */
#define DEFAULT_CONSOLE_LOGLEVEL 7 /* anything MORE serious than KERN_DEBUG */
-DECLARE_WAIT_QUEUE_HEAD(log_wait);
-
int console_printk[4] = {
DEFAULT_CONSOLE_LOGLEVEL, /* console_loglevel */
DEFAULT_MESSAGE_LOGLEVEL, /* default_message_loglevel */
@@ -98,22 +106,20 @@ EXPORT_SYMBOL_GPL(console_drivers);
static int console_locked, console_suspended;
/*
- * logbuf_lock protects log_buf, log_start, log_end, con_start and logged_chars
+ * logbuf_lock protects g_log_buf, g_log_start, g_log_end, con_start and
g_logged_chars
* It is also used in interesting ways to provide interlocking in
* release_console_sem().
*/
static DEFINE_SPINLOCK(logbuf_lock);
-#define LOG_BUF_MASK (log_buf_len-1)
-#define LOG_BUF(idx) (log_buf[(idx) & LOG_BUF_MASK])
+#define LOG_BUF_MASK(ns) ((ns)->buf_len-1)
+#define LOG_BUF(ns, idx) ((ns)->buf[(idx) & LOG_BUF_MASK(ns)])
/*
- * The indices into log_buf are not constrained to log_buf_len - they
+ * The indices into g_log_buf are not constrained to g_log_buf_len - they
* must be masked before subscripting
*/
-static unsigned log_start; /* Index into log_buf: next char to be read by
syslog() */
-static unsigned con_start; /* Index into log_buf: next char to be sent to
consoles */
-static unsigned log_end; /* Index into log_buf:
most-recently-written-char + 1 */
+static unsigned con_start; /* Index into g_log_buf: next char to be sent
to consoles */
/*
*
