On 02/12/2013 08:11 PM, John Fastabend wrote:
> From: John Fastabend <john.fastab...@gmail.com>
>
> Add routine to scan rules.conf file and move matching running tasks
> in /proc/pid/* into configured control groups. Then at init time
> we can move running tasks into the correct control group. Also
> add tracking to get events when new directories are added to a
> control group so we can move running tasks into these new sub-control
> groups.
This can be easily achieved by executing something like cgclassify `cat
/sys/fs/cgroup/*/task` after starting the daemon. I don't like the code,
see below.
> To accomplish this track the watched directories in a list adding
> additional directories and removing them as needed. When inotify
> events are received scan the rules.conf file.
I don't follow here. Event of 'creating control group' does not indicate
that cgrules.conf has changed and should be reloaded. Shouldn't you
watch the cgrules.conf file instead?
> Without this adding control group after cgrulesengd has started
> does not work correctly and processes running before cgrulesengd
> is started are not managed.
>
> This allows for a couple use cases that I could not find a way to
> handle without this infrastructure. First processes that are used
> in initd time frame are run before cgrulesengd is started. Often
> these are daemons that we would like to manage via control groups.
>
> Another example is the net_{cls|prio} cgroups. In these cases
> we may be adding/removing groups somewhat dynamically based on
> control protocols (DCBX) or user input. In the DCB case DCBX
> negotiates what priority an application should use via
> link layer discovery protocol (LLDP). Then we use control
> groups to set this up but this usually happens after cgrulesengd
> has started so prior to this happening we don't know which apps
> and directories we need to support.
If you add/remove cgroups dynamically, you should also add rules to
cgrules.conf dynamically, so the daemon knows it should move something
to the new groups. And if you modify cgrules.conf, you can also send
SIGUSR2 to cgrulesengd to reload the rules.
(alternatively, I can imagine some socket/dbus/whatever in cgrulesengd
which would allow adding rules dynamically without fiddling with
cgrules.conf)
>
> Signed-off-by: John Fastabend <john.r.fastab...@intel.com>
> ---
> include/libcgroup/tasks.h | 1
> src/api.c | 83 +++++++++++++++++++
> src/daemon/cgrulesengd.c | 193
> ++++++++++++++++++++++++++++++++++++++++++++-
> src/libcgroup.map | 1
> 4 files changed, 273 insertions(+), 5 deletions(-)
>
> diff --git a/include/libcgroup/tasks.h b/include/libcgroup/tasks.h
> index 0f79220..fc0cb63 100644
> --- a/include/libcgroup/tasks.h
> +++ b/include/libcgroup/tasks.h
> @@ -109,6 +109,7 @@ int cgroup_reload_cached_rules(void);
> * @param fp Destination file, where the rules will be printed.
> */
> void cgroup_print_rules_config(FILE *fp);
> +void cgroup_search_rules_config(void);
>
> /**
> * @}
> diff --git a/src/api.c b/src/api.c
> index 11cd1b4..c9861a5 100644
> --- a/src/api.c
> +++ b/src/api.c
> @@ -3055,6 +3055,89 @@ int cgroup_change_cgroup_path(const char *dest, pid_t
> pid,
> return ret;
> }
>
> +static int cg_get_pid_from_flags(int uid, int gid, char *procname, int *pid)
> +{
> + DIR *dir;
> + struct dirent *pid_dir = NULL;
> + char *path = "/proc/";
> + char *pid_name;
> + char pid_path[PATH_MAX];
> + char buff[CGROUP_RULE_MAXLINE];
> + int sgid, suid, spid, err;
> + FILE *fd;
> +
> + dir = opendir(path);
> + if (!dir)
> + return -ECGOTHER;
> +
> + while ((pid_dir = readdir(dir)) != NULL) {
> + err = snprintf(pid_path, sizeof(pid_path),
> + "%s%s/status", path, pid_dir->d_name);
> + if (err < 0) {
> + cgroup_dbg("%s pid_path failure\n", __func__);
> + continue;
> + }
> +
> + fd = fopen(pid_path, "r");
> + if (!fd)
> + continue;
> + memset(pid_path, 0, sizeof(pid_path));
> + sgid = suid = spid = -1;
> + while (fgets(buff, sizeof(buff), fd) != NULL) {
> + sscanf(buff, "Name: %as", &pid_name);
> + sscanf(buff, "Gid: %i", &sgid);
> + sscanf(buff, "Uid: %i", &suid);
> + sscanf(buff, "Pid: %i", &spid);
> + }
> +
> + fclose(fd);
> + if ((!procname ||
> + (strncmp(procname, pid_name, strlen(procname)) == 0)) &&
> + (gid == (int) CGRULE_WILD || gid == sgid) &&
> + (uid == (int) CGRULE_WILD || uid == suid)) {
> + free(pid_name);
> + closedir(dir);
> + *pid = spid;
> + return 0;
> + }
> +
> + free(pid_name);
> + }
> +
> + closedir(dir);
> + return 1;
> +}
> +
> +void cgroup_search_rules_config(void)
> +{
> + struct cgroup_rule *itr;
> +
> + pthread_rwlock_rdlock(&rl_lock);
> +
> + if (!(rl.head)) {
> + pthread_rwlock_unlock(&rl_lock);
> + return;
> + }
> +
> + itr = rl.head;
> + while (itr) {
> + int err, pid = 0;
> +
> + err = cg_get_pid_from_flags(itr->uid, itr->gid,
> + itr->procname, &pid);
> +
> + if (!err) {
> + pthread_rwlock_unlock(&rl_lock);
> + cgroup_change_cgroup_flags(itr->uid, itr->gid,
> + itr->procname, pid, 0);
> + pthread_rwlock_rdlock(&rl_lock);
> + }
> + itr = itr->next;
> + }
> + pthread_rwlock_unlock(&rl_lock);
> +}
If I read the code correctly, for each rule in cgrules.conf it finds
*one* PID and moves it to the right group. Shouldn't it work like for
*every* PID find a matching rule and add it to its group? I think we
have API for the "find a matching rule and add it to its group", all you
need is to enumerate all PIDs.
> +
> +
> /**
> * Print the cached rules table. This function should be called only after
> * first calling cgroup_parse_config(), but it will work with an empty rule
> diff --git a/src/daemon/cgrulesengd.c b/src/daemon/cgrulesengd.c
> index f12db45..bf30a3b 100644
> --- a/src/daemon/cgrulesengd.c
> +++ b/src/daemon/cgrulesengd.c
> @@ -56,6 +56,10 @@
> #include <linux/un.h>
> #include <pwd.h>
> #include <grp.h>
> +#include <sys/inotify.h>
> +#include <sys/queue.h>
> +#include <dirent.h>
> +#include <mntent.h>
>
> #define NUM_PER_REALLOCATIOM (100)
>
> @@ -496,6 +500,150 @@ static int cgre_handle_msg(struct cn_msg *cn_hdr)
> return ret;
> }
>
> +/* List of directories to watch with inotify */
> +struct cgre_watch {
> + int fd;
> + char *dir;
> + LIST_ENTRY(cgre_watch) entries;
> +};
> +
> +LIST_HEAD(cgre_watch_head, cgre_watch);
> +struct cgre_watch_head cgre_watch_head;
> +
> +struct cgre_watch *cgre_watch_find(int fd, struct cgre_watch_head *head)
> +{
> + struct cgre_watch *tmp;
> +
> + LIST_FOREACH(tmp, head, entries) {
> + if (tmp->fd == fd)
> + return tmp;
> + }
> +
> + return NULL;
> +}
> +
> +static void cgre_watch_del(int sk_inot, struct cgre_watch *entry)
> +{
> + int err = inotify_rm_watch(sk_inot, entry->fd);
> +
> + if (err)
> + flog(LOG_WARNING, "Inotify delete watch failed: %s\n",
> + strerror(err));
> +
> + LIST_REMOVE(entry, entries);
> + close(entry->fd);
> + free(entry->dir);
> + free(entry);
> +}
> +
> +static int __cgre_watch_add(int sk_inot, char *dir)
> +{
> + DIR *test;
> + int fd;
> + struct cgre_watch *watch;
> + uint32_t mask = IN_CREATE | IN_DELETE;
> +
> + test = opendir(dir);
> + if (!test)
> + return -ENOTDIR;
> + closedir(test);
> +
> + fd = inotify_add_watch(sk_inot, dir, mask);
> + if (fd < 0) {
> + flog(LOG_WARNING, "Inotify add watch failed: %s",
> + strerror(fd));
> + return -EINVAL;
> + }
> +
> + watch = calloc(1, sizeof(*watch));
> + if (!watch)
> + return -ENOMEM;
> +
> + watch->fd = fd;
> + watch->dir = strdup(dir);
> + LIST_INSERT_HEAD(&cgre_watch_head, watch, entries);
> +
> + return 0;
> +}
> +
> +static int cgre_watch_add(int sk_inot, char *dir)
> +{
> + int i, err;
> + struct dirent **namelist;
> + int fd_cnt = scandir(dir, &namelist, 0, alphasort);
> +
> + err = __cgre_watch_add(sk_inot, dir);
> + if (err)
> + return err;
> +
> + for (i = 0; i < fd_cnt; i++) {
> + int vars;
> + char subdir[FILENAME_MAX];
> +
> + vars = snprintf(subdir, FILENAME_MAX, "%s/%s",
> + dir, namelist[i]->d_name);
> + if (vars < 0 ||
> + strcmp(namelist[i]->d_name, ".") == 0 ||
> + strcmp(namelist[i]->d_name, "..") == 0)
> + continue;
> +
> + err = __cgre_watch_add(sk_inot, subdir);
> + if (err && err != -ENOTDIR)
> + return err;
> + }
> +
> + return 0;
> +}
> +
> +static void cgre_watch_flush(int sk_inot, struct cgre_watch_head *head)
> +{
> + while (head->lh_first != NULL)
> + cgre_watch_del(sk_inot, head->lh_first);
> +}
> +
> +static int cgre_receive_inotify_msg(int sk_inot)
> +{
> + struct inotify_event *event;
> + char buf[BUFF_SIZE] = "";
> + char dir[FILENAME_MAX];
> + int ret = 0;
> +
> + ret = read(sk_inot, buf, sizeof(buf));
> + if (ret < 0)
> + return ret;
> +
> + for (event = (struct inotify_event *) buf; ret > 0; event++) {
> + struct cgre_watch *watch;
> +
> + ret -= (sizeof(*event) + event->len);
> +
> + if (event->len <= 0)
> + continue;
> +
> + watch = cgre_watch_find(event->wd, &cgre_watch_head);
> + if (!watch)
> + continue;
> +
> + if (event->mask & IN_DELETE) {
> + struct cgre_watch *tmp;
> +
> + LIST_FOREACH(tmp, &cgre_watch_head, entries) {
> + if (strcmp(tmp->dir, event->name) == 0)
> + cgre_watch_del(sk_inot, tmp);
> + }
> + } else if (event->mask & IN_CREATE) {
> + int vars = snprintf(dir, FILENAME_MAX, "%s/%s",
> + watch->dir, event->name);
> +
> + if (vars > 0)
> + cgre_watch_add(sk_inot, dir);
> + }
> + }
> +
> + cgroup_search_rules_config();
> + return ret;
> +}
> +
> static int cgre_receive_netlink_msg(int sk_nl)
> {
> char buff[BUFF_SIZE];
> @@ -590,7 +738,7 @@ close:
>
> static int cgre_create_netlink_socket_process_msg(void)
> {
> - int sk_nl = 0, sk_unix = 0, sk_max;
> + int sk_nl = 0, sk_unix = 0, sk_inot = 0, sk_max;
> struct sockaddr_nl my_nla;
> char buff[BUFF_SIZE];
> int rc = -1;
> @@ -598,6 +746,9 @@ static int cgre_create_netlink_socket_process_msg(void)
> struct cn_msg *cn_hdr;
> enum proc_cn_mcast_op *mcop_msg;
> struct sockaddr_un saddr;
> + struct mntent *ent, temp_ent;
> + char mntent_buffer[4 * FILENAME_MAX];
> + FILE *proc_mount;
> fd_set fds, readfds;
> sigset_t sigset;
>
> @@ -690,13 +841,37 @@ static int cgre_create_netlink_socket_process_msg(void)
> goto close_and_exit;
> }
>
> + /* inotify socket listen for controller change events */
> + sk_inot = inotify_init();
> + if (sk_inot < 0) {
> + cgroup_dbg("Error intializing inotify subsystem\n");
> + goto close_and_exit;
> + }
> +
> + proc_mount = fopen("/proc/mounts", "re");
> + if (!proc_mount) {
> + cgroup_dbg("Error proc mount open failed\n");
> + goto close_and_exit;
> + }
> +
> + while ((ent = getmntent_r(proc_mount, &temp_ent,
> + mntent_buffer,
> + sizeof(mntent_buffer))) != NULL) {
> + if (strcmp(ent->mnt_fsname, "cgroup_root") &&
> + strcmp(ent->mnt_type, "cgroup"))
> + continue;
> +
> + cgre_watch_add(sk_inot, ent->mnt_dir);
> + }
> + fclose(proc_mount);
> +
> FD_ZERO(&readfds);
> FD_SET(sk_nl, &readfds);
> FD_SET(sk_unix, &readfds);
> - if (sk_nl < sk_unix)
> - sk_max = sk_unix;
> - else
> - sk_max = sk_nl;
> + FD_SET(sk_inot, &readfds);
> +
> + sk_max = max(sk_nl, sk_unix);
> + sk_max = max(sk_max, sk_inot);
>
> sigemptyset(&sigset);
> sigaddset(&sigset, SIGUSR2);
> @@ -719,6 +894,8 @@ static int cgre_create_netlink_socket_process_msg(void)
> }
> if (FD_ISSET(sk_unix, &fds))
> cgre_receive_unix_domain_msg(sk_unix);
> + if (FD_ISSET(sk_inot, &fds))
> + cgre_receive_inotify_msg(sk_inot);
> }
>
> close_and_exit:
> @@ -726,6 +903,9 @@ close_and_exit:
> close(sk_nl);
> if (sk_unix > 0)
> close(sk_unix);
> + cgre_watch_flush(sk_inot, &cgre_watch_head);
> + if (sk_inot > 0)
> + close(sk_inot);
> return rc;
> }
>
> @@ -1171,6 +1351,9 @@ int main(int argc, char *argv[])
> if (logfile && loglevel >= LOG_INFO)
> cgroup_print_rules_config(logfile);
>
> + /* Scan for running applications with rules */
> + cgroup_search_rules_config();
> +
> flog(LOG_NOTICE, "Started the CGroup Rules Engine Daemon.");
>
> /* We loop endlesly in this function, unless we encounter an error. */
> diff --git a/src/libcgroup.map b/src/libcgroup.map
> index b550a58..e629c91 100644
> --- a/src/libcgroup.map
> +++ b/src/libcgroup.map
> @@ -33,6 +33,7 @@ global:
> cgroup_set_value_bool;
> cgroup_change_cgroup_uid_gid_flags;
> cgroup_print_rules_config;
> + cgroup_search_rules_config;
> cgroup_reload_cached_rules;
> cgroup_init_rules_cache;
> cgroup_get_current_controller_path;
>
>
> ------------------------------------------------------------------------------
> Free Next-Gen Firewall Hardware Offer
> Buy your Sophos next-gen firewall before the end March 2013
> and get the hardware for free! Learn more.
> http://p.sf.net/sfu/sophos-d2d-feb
> _______________________________________________
> Libcg-devel mailing list
> Libcg-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/libcg-devel
>
------------------------------------------------------------------------------
Free Next-Gen Firewall Hardware Offer
Buy your Sophos next-gen firewall before the end March 2013
and get the hardware for free! Learn more.
http://p.sf.net/sfu/sophos-d2d-feb
_______________________________________________
Libcg-devel mailing list
Libcg-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/libcg-devel
