Ok all I now have a real answer and a solution to the needing to change /tmp locations in Mesos Slaves.
Basically, it wasn't a difference between Redhat and Ubuntu, it was a mistake I made in my yarn-site.xml for the resource manager. Basically I copied the yarn-site from the wiki but on one I had uncommented the cgroups section. My hypothesis: when running with Cgroups information, Yarn doesn't see the /tmp and it's permissions, therefore creating the tgz with the script I posted earlier works. The permissions are what Yarn requires. Once I set that on my Redhat cluster, all was well. I realized that in comparing configs. That addresses most of my concerns, and I recommend people use that where possible to avoid the Mesos Slave changes. We also may want to do some notes in the wiki around this, but I found it through hacking, and I'd welcome more "from a point of view of understanding" comments, I just made it work :) I'll send another email shortly on my last issue. On Wed, Nov 18, 2015 at 3:16 PM, John Omernik <[email protected]> wrote: > Baffling... so as I said, I fixed the mapr.host that works well (as long > as I have one NM per physical node) and I got my Ubuntu based Mesos cluster > to work with Myriad. At this point I am baffled on why Myriad/Yarn would > complain about permissions on Redhat but not Ubuntu, below I've pulled > files in /, /hadoop-2.7.0, /hadoop-2.7.0/bin and down the etc tree so you > can see how they are setup the same. I am running as the same users in > marathon etc. Same myriad configs... but I am getting the permissions issue > for Redhat, and Ubuntu is working fine (and more than just getting the NMs > running, I flexed up 6 NMs and was running hive queries with no issues!) > > The only differences I can see is that the java versions are different > (1.8.0_45 for Ubuntu and 1.8.0_65 for Redhat) and the apparently the builds > of hadoop-2.7.0 are different from MapR. (They are different in size as > well, but that may be the difference between a RPM Redhat install and a > Debian Ubuntu install, I guess I didn't expect different hadoop builds > between platforms though...) > > If you can see anything else, please let me know, I'd love to have this > working. > > John > > > Here are the dumps: Ubuntu... this is working: > > STARTUP_MSG: build = [email protected]:mapr/private-hadoop-common.git -r > fc95119f587541fb3a9af0dbeeed23c974178115; compiled by 'root' on > 2015-08-19T20:02Z > STARTUP_MSG: java = 1.8.0_45-internal > > No Error > Ubuntu > > marathon user: mapr > myriadframeuser: mapr > myriadsuperuser: mapr > > / > drwxr-xr-x 10 mapr mapr 4 KB Sep 10 14:40 hadoop-2.7.0 > -rw-r--r-- 1 mapr mapr 77 KB Nov 18 12:48 conf > -rw-r--r-- 1 mapr mapr 282 MB Nov 18 12:48 hadoop-2.7.0.tgz > -rw-r--r-- 1 mapr mapr 156 KB Nov 18 14:04 stderr > -rw-r--r-- 1 mapr mapr 743 B Nov 18 12:48 stdout > > /hadoop-2.7.0/ > > drwxrwxrwx 3 mapr root 4 KB Nov 18 12:48 logs > drwxr-xr-x 2 mapr root 4 KB Sep 10 14:40 bin > drwxr-xr-x 3 mapr root 4 KB Sep 10 14:40 etc > drwxr-xr-x 2 mapr root 4 KB Sep 10 14:40 include > drwxr-xr-x 3 mapr root 4 KB Sep 10 14:40 lib > drwxr-xr-x 2 mapr root 4 KB Sep 10 14:40 libexec > drwxr-xr-x 2 mapr root 4 KB Sep 10 14:40 sbin > drwxr-xr-x 4 mapr root 4 KB Sep 10 14:40 share > -rw-r--r-- 1 mapr root 15 KB Jul 09 04:36 LICENSE.txt > -rw-r--r-- 1 mapr root 101 B Jul 09 04:36 NOTICE.txt > -rw-r--r-- 1 mapr root 1 KB Jul 09 04:36 > > /hadoop-2.7.0/bin > > -rwxr-xr-x 1 mapr root 9 KB Jul 09 04:36 hadoop > -rwxr-xr-x 1 mapr root 10 KB Jul 09 04:36 hadoop.cmd > -rwxr-xr-x 1 mapr root 12 KB Jul 09 04:36 hdfs > -rwxr-xr-x 1 mapr root 7 KB Jul 09 04:36 hdfs.cmd > -rwxr-xr-x 1 mapr root 7 KB Jul 09 04:36 mapred > -rwxr-xr-x 1 mapr root 6 KB Jul 09 04:36 mapred.cmd > -rwxr-xr-x 1 mapr root 2 KB Jul 09 04:36 rcc > -rwxr-xr-x 1 mapr root 172 KB Jul 09 04:36 test-container-executor > -rwxr-xr-x 1 mapr root 14 KB Jul 09 04:36 yarn > -rwxr-xr-x 1 mapr root 11 KB Jul 09 04:36 yarn.cmd > ----r-x--- 1 root mapr 140 KB Jul 09 04:36 container-executor > > /hadoop-2.7.0/etc > > drwxr-xr-x 2 mapr root 4 KB Nov 18 12:48 hadoop > > /hadoop-2.7.0/etc/hadoop/ > > rw-r--r-- 1 mapr root 4 KB Jul 09 04:36 capacity-scheduler.xml > -rw-r--r-- 1 mapr root 1 KB Jul 09 04:36 configuration.xsl > -rw-r--r-- 1 root root 168 B Oct 06 08:37 container-executor.cfg > -rw-r--r-- 1 mapr root 775 B Oct 06 08:37 core-site.xml > -rw-r--r-- 1 mapr root 631 B Jul 09 04:36 fair-scheduler.xml > -rw-r--r-- 1 mapr root 4 KB Jul 09 04:36 hadoop-env.cmd > -rw-r--r-- 1 mapr root 4 KB Jul 09 04:36 hadoop-env.sh > -rw-r--r-- 1 mapr root 2 KB Jul 09 04:36 hadoop-metrics.properties > -rw-r--r-- 1 mapr root 3 KB Jul 09 04:36 hadoop-metrics2.properties > -rw-r--r-- 1 mapr root 9 KB Jul 09 04:36 hadoop-policy.xml > -rw-r--r-- 1 mapr root 775 B Jul 09 04:36 hdfs-site.xml > -rw-r--r-- 1 mapr root 1 KB Jul 09 04:36 httpfs-env.sh > -rw-r--r-- 1 mapr root 2 KB Jul 09 04:36 httpfs-log4j.properties > -rw-r--r-- 1 mapr root 21 B Jul 09 04:36 httpfs-signature.secret > -rw-r--r-- 1 mapr root 620 B Jul 09 04:36 httpfs-site.xml > -rw-r--r-- 1 mapr root 3 KB Jul 09 04:36 kms-acls.xml > -rw-r--r-- 1 mapr root 1 KB Jul 09 04:36 kms-env.sh > -rw-r--r-- 1 mapr root 2 KB Jul 09 04:36 kms-log4j.properties > -rw-r--r-- 1 mapr root 5 KB Jul 09 04:36 kms-site.xml > -rw-r--r-- 1 mapr root 11 KB Jul 09 04:36 log4j.properties > -rw-r--r-- 1 mapr root 951 B Jul 09 04:36 mapred-env.cmd > -rw-r--r-- 1 mapr root 1 KB Jul 09 04:36 mapred-env.sh > -rw-r--r-- 1 mapr root 4 KB Jul 09 04:36 mapred-queues.xml.template > -rw-r--r-- 1 mapr root 1 KB Oct 06 08:37 mapred-site.xml > -rw-r--r-- 1 mapr root 1 KB Jul 09 04:36 mapred-site.xml.template > -rw-r--r-- 1 root root 3 KB Nov 18 12:27 myriad-config-default.yml > -rw-r--r-- 1 mapr root 10 B Jul 09 04:36 slaves > -rw-r--r-- 1 mapr root 2 KB Oct 06 08:37 ssl-client.xml > -rw-r--r-- 1 mapr root 2 KB Jul 09 04:36 ssl-client.xml.example > -rw-r--r-- 1 mapr root 2 KB Oct 06 08:37 ssl-server.xml > -rw-r--r-- 1 mapr root 2 KB Jul 09 04:36 ssl-server.xml.example > -rw-r--r-- 1 mapr root 2 KB Jul 09 04:36 yarn-env.cmd > -rw-r--r-- 1 mapr root 5 KB Jul 09 04:36 yarn-env.sh > -rw-r--r-- 1 mapr root 2 KB Sep 14 10:07 yarn-site-2015-09-14.10-07.xml > -rw-r--r-- 1 mapr root 2 KB Oct 06 08:37 yarn-site-2015-10-06.08-37.xml > -rw-r--r-- 1 mapr mapr 77 KB Nov 18 12:48 yarn-site.xml > -rw-r--r-- 1 mapr root 4 KB Jul 09 04:36 yarn-site.xml.template > > > Redhat 7: This is not working with the error included: > > STARTUP_MSG: build = [email protected]:mapr/private-hadoop-common.git -r > 5264b1d5c5c2a849ee0eb09cfcbbed19fb0bfb53; compiled by 'root' on > 2015-07-07T19:19Z > STARTUP_MSG: java = 1.8.0_65 > > Error on Red hat > ExitCodeException exitCode=24: File > /tmp/mesos/slaves/76792ee5-3fcb-465c-8601-b2f0eb0dc95d-S4/frameworks/76792ee5-3fcb-465c-8601-b2f0eb0dc95d-0000/executors/myriad_executor76792ee5-3fcb-465c-8601-b2f0eb0dc95d-000076792ee5-3fcb-465c-8601-b2f0eb0dc95d-O51676792ee5-3fcb-465c-8601-b2f0eb0dc95d-S4/runs/0aa590af-9a94-42c7-b736-6cc7537fd226/hadoop-2.7.0/etc/hadoop > must be owned by root, but is owned by 700 > > marathon user: mapr > myriadframeuser: mapr > myriadsuperuser: mapr > > > > / > drwxr-xr-x 10 mapr mapr 4 KB Sep 25 09:15 > hadoop-2.7.0 > -rw-r--r-- 1 mapr mapr 76 KB Nov 18 > 15:59 conf > -rw-r--r-- 1 mapr mapr 306 MB > Nov 18 15:59 hadoop-2.7.0.tgz > -rw-r--r-- 1 mapr mapr 121 KB Nov 18 > 15:59 stderr > -rw-r--r-- > > > /hadoop-2.7.0/ > > drwxrwxrwx 2 mapr root 4 KB Nov 18 15:57 > logs > drwxr-xr-x 2 mapr root 4 KB Sep 25 > 09:16 bin > drwxr-xr-x 3 mapr root 4 KB Sep 25 > 09:15 etc > drwxr-xr-x 2 mapr root 4 KB Sep 25 > 09:15 include > drwxr-xr-x 3 mapr root 4 KB Sep 25 > 09:15 lib > drwxr-xr-x 2 mapr root 4 KB Sep 25 > 09:16 libexec > drwxr-xr-x 2 mapr root 4 KB Sep 25 > 09:16 sbin > drwxr-xr-x 4 mapr root 4 KB Sep 25 > 09:15 share > 1 > > /hadoop-2.7.0/bin > > -rwxr-xr-x 1 mapr root 9 KB Jul 09 > 05:38 hadoop > -rwxr-xr-x 1 mapr root 10 KB Jul 09 > 05:38 hadoop.cmd > -rwxr-xr-x 1 mapr root 12 KB Jul 09 > 05:38 hdfs > -rwxr-xr-x 1 mapr root 7 KB Jul 09 > 05:38 hdfs.cmd > -rwxr-xr-x 1 mapr root 7 KB Jul 09 > 05:37 mapred > -rwxr-xr-x 1 mapr root 6 KB Jul 09 > 05:37 mapred.cmd > -rwxr-xr-x 1 mapr root 2 KB Jul 09 > 05:38 rcc > -rwxr-xr-x 1 mapr root 128 KB Jul 09 > 05:38 test-container-executor > -rwxr-xr-x 1 mapr root 14 KB Jul 09 > 05:38 yarn > -rwxr-xr-x 1 mapr root 11 KB Jul 09 > 05:38 yarn.cmd > ----r-x--- 1 root mapr 111 KB Jul > 09 05:38 container-executor > > mapr mapr 786 B Nov 18 15:59 stdout > > > /hadoop-2.7.0/etc > mode nlink uid gid size mtime > > drwxr-xr-x 2 mapr root 4 KB Nov 18 > 15:59 hadoop > > /hadoop-2.7.0/etc/hadoop > -rw-r--r-- 1 mapr root 4 KB Jul > 09 05:38 capacity-scheduler.xml > -rw-r--r-- 1 mapr root 1 KB Jul > 09 05:38 configuration.xsl > -rw-r--r-- 1 root root 168 B Nov > 02 18:22 container-executor.cfg > -rw-r--r-- 1 mapr root 775 B Nov 02 > 18:22 core-site.xml > -rw-r--r-- 1 mapr root 631 B Jul 09 > 05:38 fair-scheduler.xml > -rw-r--r-- 1 mapr root 4 KB Jul > 09 05:38 hadoop-env.cmd > -rw-r--r-- 1 mapr root 4 KB Jul > 09 05:38 hadoop-env.sh > -rw-r--r-- 1 mapr root 2 KB Jul > 09 05:38 hadoop-metrics.properties > -rw-r--r-- 1 mapr root 3 KB Jul > 09 05:38 hadoop-metrics2.properties > -rw-r--r-- 1 mapr root 9 KB Jul > 09 05:38 hadoop-policy.xml > -rw-r--r-- 1 mapr root 775 B Jul 09 > 05:38 hdfs-site.xml > -rw-r--r-- 1 mapr root 1 KB Jul > 09 05:38 httpfs-env.sh > -rw-r--r-- 1 mapr root 2 KB Jul > 09 05:38 httpfs-log4j.properties > -rw-r--r-- 1 mapr root 21 B Jul > 09 05:38 httpfs-signature.secret > -rw-r--r-- 1 mapr root 620 B Jul 09 > 05:38 httpfs-site.xml > -rw-r--r-- 1 mapr root 3 KB Jul > 09 05:38 kms-acls.xml > -rw-r--r-- 1 mapr root 1 KB Jul > 09 05:38 kms-env.sh > -rw-r--r-- 1 mapr root 2 KB Jul > 09 05:38 kms-log4j.properties > -rw-r--r-- 1 mapr root 5 KB Jul > 09 05:38 kms-site.xml > -rw-r--r-- 1 mapr root 11 KB Jul 09 > 05:38 log4j.properties > -rw-r--r-- 1 mapr root 931 B Jul 09 > 05:38 mapred-env.cmd > -rw-r--r-- 1 mapr root 1 KB Jul > 09 05:38 mapred-env.sh > -rw-r--r-- 1 mapr root 4 KB Jul > 09 05:38 mapred-queues.xml.template > -rw-r--r-- 1 mapr root 1 KB Nov > 02 18:22 mapred-site.xml > -rw-r--r-- 1 mapr root 1 KB Jul > 09 05:38 mapred-site.xml.template > -rw-r--r-- 1 mapr root 3 KB Nov > 18 15:57 myriad-config-default.yml > -rw-r--r-- 1 mapr root 10 B Jul > 09 05:38 slaves > -rw-r--r-- 1 mapr root 2 KB Nov > 02 18:22 ssl-client.xml > -rw-r--r-- 1 mapr root 2 KB Jul > 09 05:38 ssl-client.xml.example > -rw-r--r-- 1 mapr root 2 KB Nov > 02 18:22 ssl-server.xml > -rw-r--r-- 1 mapr root 2 KB Jul > 09 05:38 ssl-server.xml.example > -rw-r--r-- 1 mapr root 2 KB Jul > 09 05:38 yarn-env.cmd > -rw-r--r-- 1 mapr root 5 KB Jul > 09 05:38 yarn-env.sh > -rw-r--r-- 1 mapr root 2 KB Oct > 01 16:56 yarn-site-2015-10-01.20-56.xml > -rw-r--r-- 1 mapr root 2 KB Oct > 08 06:32 yarn-site-2015-10-08.10-32.xml > -rw-r--r-- 1 mapr root 2 KB Oct > 16 15:33 yarn-site-2015-10-16.19-33.xml > -rw-r--r-- 1 mapr root 2 KB Nov > 02 18:22 yarn-site-2015-11-02.23-22.xml > -rw-r--r-- 1 mapr mapr 76 KB Nov 18 > 16:01 yarn-site.xml > -rw-r--r-- 1 mapr root 4 KB Jul > 09 05:38 yarn-site.xml.template > > On Wed, Nov 18, 2015 at 12:41 PM, John Omernik <[email protected]> wrote: > >> So there are two issues currently I am looking into. The first is the >> permissions of directories. I'd still like to get the feelings from the >> group on that because I've not managed to get Myriad/Yarn working on one >> cluster (based on Ubuntu 14.04) but can't get it to work on another cluster >> based on Red Hat 7. It's strange from what I can tell everything is the >> same, but the Redhat cluster complains about the the permissions >> /etc/hadoop not being owned by root (it's owned by mapr:root but on the >> ubuntu cluster that works fine with the same ownership!) I do notice that >> my build times reported by mapr are different.. but that may just be the >> build for Redhat vs the build for Ubuntu? Still digging into that one... >> >> As to the hostname / mapr.host issue. I found a neat hack that may work >> for folks >> >> By setting this in my myriad config: >> >> yarnEnvironment: >> >> YARN_HOME: hadoop-2.7.0 >> >> YARN_NODEMANAGER_OPTS: "-Dnodemanager.resource.io-spindles=4.0 >> -Dmapr.host=$(hostname -f)" >> >> >> I am able to get the mapr.host set back to be the correct hostname where >> the nodemanager is running, this helps with a number of issues. I thought >> about this, and realized it would be better if I could get the hostname to >> the createTTVolume script but use a unique name for the mount point (What >> if I have multiple NMs on a single physical node?) >> >> So, I tried: >> >> YARN_NODEMANAGER_OPTS: "-Dnodemanager.resource.io-spindles=4.0 >> -Dmapr.host=$(basename `pwd`)" >> >> >> Thinking that if I could get the directory name of the "run" in my >> sandbox, I should be reasonably assured of uniqueness. That seemed to work >> when the nodemanager kicked off the the command: >> >> /opt/mapr/server/createTTVolume.sh hadoopmapr2.brewingintel.com >> /var/mapr/local/48833481-0c7a-4728-8f93-bcf9b545ad81/mapred >> /var/mapr/local/48833481-0c7a-4728-8f93-bcf9b545ad81/mapred/nodeManager yarn >> >> >> However, the script never returned, and the task failed. So at this >> point, I think we can get good info passed to the MapR script, but I am not >> sure how the script or how mapr is creating the volume to work in >> conjunction with Yarn. >> >> To summarize: Right now I have Myriad working... but only on a Ubuntu >> Mesos cluster. I have NOT changed my /tmp location for the Mesos slaves, I >> can't get things working on my Redhat cluster, and I seemed to have found a >> hacky workaround for the mapr.host. Here is the script I am using to build >> my tgz file... >> >> >> #!/bin/bash >> >> >> BASEDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" >> >> >> echo "#### Base Location: $BASEDIR" >> >> >> ################## BUILD VARIABLES >> >> >> HADOOP_VER="hadoop-2.7.0" >> >> HADOOP_BASE="/opt/mapr/hadoop" >> >> HADOOP_HOME="${HADOOP_BASE}/${HADOOP_VER}" >> >> MYRIAD_BUILD="/opt/mapr/myriad/incubator-myriad" >> >> CONF_LOC="/mapr/brewpot/mesos/myriad/conf" >> >> URI_LOC="/mapr/brewpot/mesos/myriad" >> >> >> >> >> ################################# >> >> #Clean the working directory >> >> echo "#### Cleaning Build DIR and old tgz" >> >> sudo rm -rf ./${HADOOP_VER} >> >> sudo rm -rf ./${HADOOP_VER}.tgz >> >> >> #Copy a fresh copy of the hadoopz >> >> echo "#### Copying Clean Build" >> >> >> # I go here and tar with h and p. h is so all the symlinked items in the >> MapR get put into the tgz and p to preserver permissions >> >> cd $HADOOP_BASE >> >> sudo tar zcfhp ${BASEDIR}/${HADOOP_VER}.tgz $HADOOP_VER >> >> >> echo "#### Untaring New Build" >> >> # I untar things in a new location so I can play without affecting the >> "stock" install >> >> cd $BASEDIR >> >> sudo tar zxfp ${HADOOP_VER}.tgz >> >> >> echo "#### Now remove source tgz to get ready for build" >> >> sudo rm ${HADOOP_VER}.tgz >> >> >> >> >> # This permission combination seems to work. I go and grab the >> container-executor from the stock build so that I have the setuid version. >> >> echo "#### Cleaning Base Build Logs, yarn-site, and permissions" >> >> sudo rm $HADOOP_VER/etc/hadoop/yarn-site.xml >> >> sudo rm -rf $HADOOP_VER/logs/* >> >> sudo chown mapr:mapr ${HADOOP_VER} >> >> sudo chown -R mapr:root ${HADOOP_VER}/* >> >> sudo chown root:root ${HADOOP_VER}/etc/hadoop/container-executor.cfg >> >> sudo cp --preserve ${HADOOP_HOME}/bin/container-executor >> ${HADOOP_VER}/bin/ >> >> >> >> >> #Copy the jars from Myriad into the Hadoop libs folders (You will need to >> have Myriad build first with the root of your build being $MYRIAD_BUILD >> >> echo "#### Copying Myriad Jars" >> >> sudo cp $MYRIAD_BUILD/myriad-scheduler/build/libs/*.jar >> $HADOOP_VER/share/hadoop/yarn/lib/ >> >> sudo cp >> $MYRIAD_BUILD/myriad-executor/build/libs/myriad-executor-0.1.0.jar >> $HADOOP_VER/share/hadoop/yarn/lib/ >> >> >> #Address Configs >> >> >> # First take the myriad-config-default.yml and put it into the >> $HADOOP/etc/hadoop so it's in the tarball >> >> echo "#### Updating myriad-config-default.yml" >> >> sudo cp ${CONF_LOC}/myriad-config-default.yml ${HADOOP_VER}/etc/hadoop/ >> >> >> # Tar all the things with all the privs >> >> echo "#### Tarring all the things" >> >> sudo tar zcfhp ${HADOOP_VER}.tgz ${HADOOP_VER}/ >> >> >> # Copy to the URI location... note I am using MapR so I cp it directly to >> the MapFS location via NFS share, it would probably be good to use a hadoop >> copy command for interoperability >> >> echo "#### Copying to HDFS Location" >> >> cp ${HADOOP_VER}.tgz ${URI_LOC}/ >> >> >> # I do this because it worked... not sure if I remo >> >> #sudo chown mapr:mapr ${URI_LOC}/${HADOOP_VER}.tgz >> >> >> #echo "#### Cleaning unpacked location" >> >> sudo rm -rf ./${HADOOP_VER} >> >> sudo rm ./${HADOOP_VER}.tgz >> >> >> >> >> On Wed, Nov 18, 2015 at 9:40 AM, yuliya Feldman < >> [email protected]> wrote: >> >>> I would love to have that piece of code "configurable", but it is not at >>> the moment. >>> Will send you patch offline. >>> Thanks,Yuliya >>> From: John Omernik <[email protected]> >>> To: [email protected]; yuliya Feldman < >>> [email protected]> >>> Sent: Wednesday, November 18, 2015 6:02 AM >>> Subject: Re: Struggling with Permissions >>> >>> Yuliya, I would be interested in the patch for MapR, is that a patch for >>> Myriad or a patch for Hadoop on MapR? I wonder if there is a hadoop env >>> file I could modified in my TGZ to help address the issue on my nodes as >>> well. Can you describe what "mapr.host" is and if I can force overwrite >>> that in my ENV file or will MapR clobber that at a later point in >>> execution? I am thinking that with some simple sed, I could "fix" the >>> conf >>> file. >>> >>> Wait, I suppose there is no way for me edit the command used to run the >>> node manager... there's a thought. Could Myriad provide an ENV value or >>> something that would allow us to edit the command or insert something >>> into >>> the command that is used to run the NM? (below is the the command on my >>> cluster) Basically, if there was a way to template that and alter it in >>> the Myriad config, I could add commands to update the variables in the >>> conf >>> file before it's copied to yarn-site on every node... just spitballing >>> ideas here... >>> >>> >>> >>> sudo tar -zxpf hadoop-2.7.0-NM.tar.gz && sudo chown mapr . && cp conf >>> hadoop-2.7.0/etc/hadoop/yarn-site.xml; export YARN_HOME=hadoop-2.7.0; >>> sudo >>> -E -u mapr -H env YARN_HOME=hadoop-2.7.0 >>> YARN_NODEMANAGER_OPTS=-Dnodemanager.resource.io-spindles=4.0 >>> -Dyarn.resourcemanager.hostname=myriad.marathon.mesos >>> >>> -Dyarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor >>> -Dnodemanager.resource.cpu-vcores=2 -Dnodemanager.resource.memory-mb=8192 >>> -Dmyriad.yarn.nodemanager.address=0.0.0.0:31984 >>> -Dmyriad.yarn.nodemanager.localizer.address=0.0.0.0:31233 >>> -Dmyriad.yarn.nodemanager.webapp.address=0.0.0.0:31716 >>> -Dmyriad.mapreduce.shuffle.port=31786 /bin/yarn nodemanager >>> >>> >>> >>> On Tue, Nov 17, 2015 at 4:44 PM, yuliya Feldman >>> <[email protected] >>> > wrote: >>> >>> > Hadoop (not Mapr) requires whole path starting from "/" be owned by >>> root >>> > and writable only by root >>> > The second problem is exactly what I was talking about configuration >>> being >>> > taken from RM that overwrites local one >>> > I can give you a patch to mitigate the issue for Mapr if you are >>> building >>> > from source. >>> > Thanks,Yuliya >>> > From: John Omernik <[email protected]> >>> > To: [email protected] >>> > Sent: Tuesday, November 17, 2015 1:15 PM >>> > Subject: Re: Struggling with Permissions >>> > >>> > Well sure /tmp is world writeable but /tmp/mesos is not world writable >>> thus >>> > there is a sandbox to play in there... or am I missing something. Not >>> to >>> > mention my tmp is rwt which is world writable but only the creator or >>> root >>> > can modify (based on the googles). >>> > Yuliya: >>> > >>> > I am seeing a weird behavior with MapR as it relates to (I believe) the >>> > mapr_direct_shuffle. >>> > >>> > In the Node Manager logs, I see things starting and it saying >>> "Checking for >>> > local volume, if local volume is not present command will create and >>> mount >>> > it" >>> > >>> > Command invoked is : /opt/mapr/server/createTTVolume.sh >>> > hadoopmapr7.brewingintel.com /var/mapr/local/ >>> > hadoopmapr2.brewingintel.com/mapred /var/mapr/local/ >>> > hadoopmapr2.brewingintel.com/mapred/nodeManager yarn >>> > >>> > >>> > What is interesting here is hadoopmapr7 is the nodemanager it's trying >>> to >>> > start on, however the mount point it's trying to create is hadoopmapr2 >>> > which is the node the resource manager happened to fall on... I was >>> very >>> > confused by that because in no place should hadoopmapr2 be "known" to >>> the >>> > nodemanager, because it thinks the resource manager hostname is >>> > myriad.marathon.mesos. >>> > >>> > So why was it hard coding to the node the resource manager is running >>> on? >>> > >>> > Well if I look at the conf file in the sandbox (the file that gets >>> copied >>> > to be yarn-site.xml for node managers. There ARE four references the >>> > hadoopmapr2. Three of the four say "source programatically" and one is >>> just >>> > set... that's mapr.host. Could there be some down stream hinkyness >>> going >>> > on with how MapR is setting hostnames? All of these variables seem >>> "wrong" >>> > in that mapr.host (on the node manager) should be hadoopmapr7 in this >>> case, >>> > and the resource managers should all be myriad.marathon.mesos. I'd be >>> > interested in your thoughts here, because I am stumped at how these are >>> > getting set. >>> > >>> > >>> > >>> > >>> > >>> <property><name>yarn.resourcemanager.address</name><value>hadoopmapr2:8032</value><source>programatically</source></property> >>> > <property><name>mapr.host</name><value>hadoopmapr2.brewingintel.com >>> > </value></property> >>> > >>> > >>> <property><name>yarn.resourcemanager.resource-tracker.address</name><value>hadoopmapr2:8031</value><source>programatically</source></property> >>> > >>> > >>> <property><name>yarn.resourcemanager.admin.address</name><value>hadoopmapr2:8033</value><source>programatically</source></property> >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > On Tue, Nov 17, 2015 at 2:51 PM, Darin Johnson < >>> [email protected]> >>> > wrote: >>> > >>> > > Yuliya: Are you referencing yarn.nodemanager.hostname or a mapr >>> specific >>> > > option? >>> > > >>> > > I'm working right now on passing a >>> > > -Dyarn.nodemanager.hostname=offer.getHostName(). Useful if you've >>> got >>> > > extra ip's for a san or management network. >>> > > >>> > > John: Yeah the permissions on the tarball are a pain to get right. >>> I'm >>> > > working on Docker Support and a build script for the tarball, which >>> > should >>> > > make things easier. Also, to the point of using world writable >>> > directories >>> > > it's a little scary from the security side of things to allow >>> executables >>> > > to run there, especially things running as privileged users. Many >>> > distro's >>> > > of linux will mount /tmp noexec. >>> > > >>> > > Darin >>> > > >>> > > On Tue, Nov 17, 2015 at 2:53 PM, yuliya Feldman >>> > > <[email protected] >>> > > > wrote: >>> > > >>> > > > Please change workdir directory for mesos slave to one that is not >>> /tmp >>> > > > and make sure that dir is owned by root. >>> > > > There is one more caveat with binary distro and MapR - in Myriad >>> code >>> > for >>> > > > binary distro configuration is copied from RM to NMs - it doe snot >>> work >>> > > for >>> > > > MapR since we need hostname (yes for the sake of local volumes) to >>> be >>> > > > unique. >>> > > > MapR will have Myriad release to handle this situation. >>> > > > From: John Omernik <[email protected]> >>> > > > To: [email protected] >>> > > > Sent: Tuesday, November 17, 2015 11:37 AM >>> > > > Subject: Re: Struggling with Permissions >>> > > > >>> > > > Oh hey, I found a post by me back on Sept 9. I looked at the >>> Jiras and >>> > > > followed the instructions with the same errors. At this point do I >>> > still >>> > > > need to have a place where the entire path is owned by root? That >>> seems >>> > > > like a an odd requirement (a changed of each node to facilitate a >>> > > > framework) >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > On Tue, Nov 17, 2015 at 1:25 PM, John Omernik <[email protected]> >>> > wrote: >>> > > > >>> > > > > Hey all, I am struggling with permissions on myriad, trying to >>> get >>> > the >>> > > > > right permissions in the tgz as well as who to run as. I am >>> running >>> > in >>> > > > > MapR, which means I need to run as mapr or root (otherwise my >>> volume >>> > > > > creation scripts will fail on MapR, MapR folks, we should talk >>> more >>> > > about >>> > > > > those scripts) >>> > > > > >>> > > > > But back to the code, I've had lots issues. When I run the >>> > > Frameworkuser >>> > > > > and Superuser as mapr, it unpacks everything as MapR and I get a >>> > > > > "/bin/container-executor" must be owned by root but is owned by >>> 700 >>> > (my >>> > > > > mapr UID). >>> > > > > >>> > > > > So now I am running as root, and I am getting the error below as >>> it >>> > > > > relates to /tmp. I am not sure which /tmp this refers to. the >>> /tmp >>> > that >>> > > > my >>> > > > > slave is executing in? (i.e. my local mesos agent /tmp >>> directory) or >>> > my >>> > > > > MaprFS /tmp directory (both of which are world writable, as /tmp >>> > > > typically >>> > > > > is... or am I mistaken here?) >>> > > > > >>> > > > > Any thoughts on how to get this to resolve? This is when >>> nodemanager >>> > is >>> > > > > trying to start running as root and root for both of my Myriad >>> users. >>> > > > > >>> > > > > Thanks! >>> > > > > >>> > > > > >>> > > > > Caused by: ExitCodeException exitCode=24: File /tmp must not be >>> world >>> > > or >>> > > > group writable, but is 1777 >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > >>> > >>> > >>> > >>> > >>> >>> >>> >>> >> >> >
