I am wondering about the process of fetching the tgz files and running them on slaves. Basically, I am trying to run hadoop-mesos, but still use the LinuxTaskController (http://hadoop.apache.org/docs/r1.0.4/cluster_setup.html for details).
When I am using hadoop, I have to swich to the defaultTaskController because when Mesos untars the tgz, it loses the setuid bit on the binary. I've done a bit of testing around this, and I am unsure why it loses it (even if the running process is root) but it does. Basically, tar by itself works like this: If the user is a super user, tar maintain all permissions that are in the tgz. (I've tested this, when I manually untar with tar zxf myhadoop.tgz it untars properly, including permissions and setuid on the Linux Task Controller.) When I untar as a non-super user, the permissions all get moved to the user that untared it, and the setuid bit is lost. It makes sense from a security point of view. So how does this work in mesos and hadoop? Well, if I run the jobtracker as user hadoop, hadoop is not a super user, all the files in the untared hadooop folder are owned by hadoop:hadoop, and the setuid bit is lost. Ok, next test, well, let's run jobtracker as root, and see what happens. (remember, when I untared as root, the setuid and all permissions were preserved). So, when we run JT as root, all the files become root:root, and the setuid bit is lost. That's weird? What happened here? (This is where I get lost, perhaps the untar/gzipping isn't using the tar command thus permissions are not preserved like I would expect) Either way, when using the LinuxTaskController, tasktrackers WILL NOT RUN if the setuid bit is not set. That's a pain, the LinuxTaskController is really nice from an impersonation/security setup with hadoop jobs. I CAN run my hadoop framework as hadoop:hadoop, but then I am limited in how things are setup and I get strange permissions issues when trying to run certain jobs as other users. The fix? I am hoping we can have a discussion around this. As I see it, the slaves are running as root, they have the power to run however we need them to run. Ideally, I'd like to see the untarring happen with the preserve permissions bit. I.e. the archives for mesos, at the very least having the OPTION to preserve permissions in the tgz. If we could do this, as an option somehow, this would be a win. Also ideally, I don't want to run the framework as root, just untar the tgz as root, preserving permissions. There is a difference between the action of untarring, and the execution of the framework, and the security nerd in me would like to ensure while the slave COULD run the framework as root, we avoid it if possible. I am not sure how exactly mesos untars things, nor am I aware how hard it would be to do this, but I think from a security perspective, the flexibility that untarting/preserving permissions (especially the setuid bit) would bring Mesos would warrant the dev time. Thoughts?
