Yes, but only in the yarn-site.xml, I have not set cgroups: true in the
myriad config, so I am not sure the relationship there...
On Thu, Nov 19, 2015 at 10:33 AM, yuliya Feldman <
yufeld...@yahoo.com.invalid> wrote:
> You mean that if you enable cgroups in yarn /tmp as a slave temp dir works
> just fine?
> From: John Omernik <j...@omernik.com>
> To: dev@myriad.incubator.apache.org; yuliya Feldman <yufeld...@yahoo.com>
> Sent: Thursday, November 19, 2015 8:02 AM
> Subject: Re: Struggling with Permissions
>
> Ok all I now have a real answer and a solution to the needing to change
> /tmp locations in Mesos Slaves.
>
> Basically, it wasn't a difference between Redhat and Ubuntu, it was a
> mistake I made in my yarn-site.xml for the resource manager. Basically I
> copied the yarn-site from the wiki but on one I had uncommented the cgroups
> section. My hypothesis: when running with Cgroups information, Yarn doesn't
> see the /tmp and it's permissions, therefore creating the tgz with the
> script I posted earlier works. The permissions are what Yarn requires.
> Once I set that on my Redhat cluster, all was well. I realized that in
> comparing configs. That addresses most of my concerns, and I recommend
> people use that where possible to avoid the Mesos Slave changes. We also
> may want to do some notes in the wiki around this, but I found it through
> hacking, and I'd welcome more "from a point of view of understanding"
> comments, I just made it work :)
>
> I'll send another email shortly on my last issue.
>
> On Wed, Nov 18, 2015 at 3:16 PM, John Omernik <j...@omernik.com> wrote:
>
> > Baffling... so as I said, I fixed the mapr.host that works well (as long
> > as I have one NM per physical node) and I got my Ubuntu based Mesos
> cluster
> > to work with Myriad. At this point I am baffled on why Myriad/Yarn would
> > complain about permissions on Redhat but not Ubuntu, below I've pulled
> > files in /, /hadoop-2.7.0, /hadoop-2.7.0/bin and down the etc tree so you
> > can see how they are setup the same. I am running as the same users in
> > marathon etc. Same myriad configs... but I am getting the permissions
> issue
> > for Redhat, and Ubuntu is working fine (and more than just getting the
> NMs
> > running, I flexed up 6 NMs and was running hive queries with no issues!)
> >
> > The only differences I can see is that the java versions are different
> > (1.8.0_45 for Ubuntu and 1.8.0_65 for Redhat) and the apparently the
> builds
> > of hadoop-2.7.0 are different from MapR. (They are different in size as
> > well, but that may be the difference between a RPM Redhat install and a
> > Debian Ubuntu install, I guess I didn't expect different hadoop builds
> > between platforms though...)
> >
> > If you can see anything else, please let me know, I'd love to have this
> > working.
> >
> > John
> >
> >
> > Here are the dumps: Ubuntu... this is working:
> >
> > STARTUP_MSG: build = g...@github.com:mapr/private-hadoop-common.git -r
> > fc95119f587541fb3a9af0dbeeed23c974178115; compiled by 'root' on
> > 2015-08-19T20:02Z
> > STARTUP_MSG: java = 1.8.0_45-internal
> >
> > No Error
> > Ubuntu
> >
> > marathon user: mapr
> > myriadframeuser: mapr
> > myriadsuperuser: mapr
> >
> > /
> > drwxr-xr-x 10 mapr mapr 4 KB Sep 10 14:40 hadoop-2.7.0
> > -rw-r--r-- 1 mapr mapr 77 KB Nov 18 12:48 conf
> > -rw-r--r-- 1 mapr mapr 282 MB Nov 18 12:48 hadoop-2.7.0.tgz
> > -rw-r--r-- 1 mapr mapr 156 KB Nov 18 14:04 stderr
> > -rw-r--r-- 1 mapr mapr 743 B Nov 18 12:48 stdout
> >
> > /hadoop-2.7.0/
> >
> > drwxrwxrwx 3 mapr root 4 KB Nov 18 12:48 logs
> > drwxr-xr-x 2 mapr root 4 KB Sep 10 14:40 bin
> > drwxr-xr-x 3 mapr root 4 KB Sep 10 14:40 etc
> > drwxr-xr-x 2 mapr root 4 KB Sep 10 14:40 include
> > drwxr-xr-x 3 mapr root 4 KB Sep 10 14:40 lib
> > drwxr-xr-x 2 mapr root 4 KB Sep 10 14:40 libexec
> > drwxr-xr-x 2 mapr root 4 KB Sep 10 14:40 sbin
> > drwxr-xr-x 4 mapr root 4 KB Sep 10 14:40 share
> > -rw-r--r-- 1 mapr root 15 KB Jul 09 04:36 LICENSE.txt
> > -rw-r--r-- 1 mapr root 101 B Jul 09 04:36 NOTICE.txt
> > -rw-r--r-- 1 mapr root 1 KB Jul 09 04:36
> >
> > /hadoop-2.7.0/bin
> >
> > -rwxr-xr-x 1 mapr root 9 KB Jul 09 04:36 hadoop
> > -rwxr-xr-x 1 mapr root 10 KB Jul 09 04:36 hadoop.cmd
> > -rwxr-xr-x 1 mapr root 12 KB Jul 09 04:36 hdfs
> > -rwxr-xr-x 1 mapr root 7 KB Jul 09 04:36 hdfs.cmd
> > -rwxr-xr-x 1 mapr root 7 KB Jul 09 04:36 mapred
> > -rwxr-xr-x 1 mapr root 6 KB Jul 09 04:36 mapred.cmd
> > -rwxr-xr-x 1 mapr root 2 KB Jul 09 04:36 rcc
> > -rwxr-xr-x 1 mapr root 172 KB Jul 09 04:36 test-container-executor
> > -rwxr-xr-x 1 mapr root 14 KB Jul 09 04:36 yarn
> > -rwxr-xr-x 1 mapr root 11 KB Jul 09 04:36 yarn.cmd
> > ----r-x--- 1 root mapr 140 KB Jul 09 04:36 container-executor
> >
> > /hadoop-2.7.0/etc
> >
> > drwxr-xr-x 2 mapr root 4 KB Nov 18 12:48 hadoop
> >
> > /hadoop-2.7.0/etc/hadoop/
> >
> > rw-r--r-- 1 mapr root 4 KB Jul 09 04:36 capacity-scheduler.xml
> > -rw-r--r-- 1 mapr root 1 KB Jul 09 04:36 configuration.xsl
> > -rw-r--r-- 1 root root 168 B Oct 06 08:37 container-executor.cfg
> > -rw-r--r-- 1 mapr root 775 B Oct 06 08:37 core-site.xml
> > -rw-r--r-- 1 mapr root 631 B Jul 09 04:36 fair-scheduler.xml
> > -rw-r--r-- 1 mapr root 4 KB Jul 09 04:36 hadoop-env.cmd
> > -rw-r--r-- 1 mapr root 4 KB Jul 09 04:36 hadoop-env.sh
> > -rw-r--r-- 1 mapr root 2 KB Jul 09 04:36 hadoop-metrics.properties
> > -rw-r--r-- 1 mapr root 3 KB Jul 09 04:36 hadoop-metrics2.properties
> > -rw-r--r-- 1 mapr root 9 KB Jul 09 04:36 hadoop-policy.xml
> > -rw-r--r-- 1 mapr root 775 B Jul 09 04:36 hdfs-site.xml
> > -rw-r--r-- 1 mapr root 1 KB Jul 09 04:36 httpfs-env.sh
> > -rw-r--r-- 1 mapr root 2 KB Jul 09 04:36 httpfs-log4j.properties
> > -rw-r--r-- 1 mapr root 21 B Jul 09 04:36 httpfs-signature.secret
> > -rw-r--r-- 1 mapr root 620 B Jul 09 04:36 httpfs-site.xml
> > -rw-r--r-- 1 mapr root 3 KB Jul 09 04:36 kms-acls.xml
> > -rw-r--r-- 1 mapr root 1 KB Jul 09 04:36 kms-env.sh
> > -rw-r--r-- 1 mapr root 2 KB Jul 09 04:36 kms-log4j.properties
> > -rw-r--r-- 1 mapr root 5 KB Jul 09 04:36 kms-site.xml
> > -rw-r--r-- 1 mapr root 11 KB Jul 09 04:36 log4j.properties
> > -rw-r--r-- 1 mapr root 951 B Jul 09 04:36 mapred-env.cmd
> > -rw-r--r-- 1 mapr root 1 KB Jul 09 04:36 mapred-env.sh
> > -rw-r--r-- 1 mapr root 4 KB Jul 09 04:36 mapred-queues.xml.template
> > -rw-r--r-- 1 mapr root 1 KB Oct 06 08:37 mapred-site.xml
> > -rw-r--r-- 1 mapr root 1 KB Jul 09 04:36 mapred-site.xml.template
> > -rw-r--r-- 1 root root 3 KB Nov 18 12:27 myriad-config-default.yml
> > -rw-r--r-- 1 mapr root 10 B Jul 09 04:36 slaves
> > -rw-r--r-- 1 mapr root 2 KB Oct 06 08:37 ssl-client.xml
> > -rw-r--r-- 1 mapr root 2 KB Jul 09 04:36 ssl-client.xml.example
> > -rw-r--r-- 1 mapr root 2 KB Oct 06 08:37 ssl-server.xml
> > -rw-r--r-- 1 mapr root 2 KB Jul 09 04:36 ssl-server.xml.example
> > -rw-r--r-- 1 mapr root 2 KB Jul 09 04:36 yarn-env.cmd
> > -rw-r--r-- 1 mapr root 5 KB Jul 09 04:36 yarn-env.sh
> > -rw-r--r-- 1 mapr root 2 KB Sep 14 10:07 yarn-site-2015-09-14.10-07.xml
> > -rw-r--r-- 1 mapr root 2 KB Oct 06 08:37 yarn-site-2015-10-06.08-37.xml
> > -rw-r--r-- 1 mapr mapr 77 KB Nov 18 12:48 yarn-site.xml
> > -rw-r--r-- 1 mapr root 4 KB Jul 09 04:36 yarn-site.xml.template
> >
> >
> > Redhat 7: This is not working with the error included:
> >
> > STARTUP_MSG: build = g...@github.com:mapr/private-hadoop-common.git -r
> > 5264b1d5c5c2a849ee0eb09cfcbbed19fb0bfb53; compiled by 'root' on
> > 2015-07-07T19:19Z
> > STARTUP_MSG: java = 1.8.0_65
> >
> > Error on Red hat
> > ExitCodeException exitCode=24: File
> >
> /tmp/mesos/slaves/76792ee5-3fcb-465c-8601-b2f0eb0dc95d-S4/frameworks/76792ee5-3fcb-465c-8601-b2f0eb0dc95d-0000/executors/myriad_executor76792ee5-3fcb-465c-8601-b2f0eb0dc95d-000076792ee5-3fcb-465c-8601-b2f0eb0dc95d-O51676792ee5-3fcb-465c-8601-b2f0eb0dc95d-S4/runs/0aa590af-9a94-42c7-b736-6cc7537fd226/hadoop-2.7.0/etc/hadoop
> > must be owned by root, but is owned by 700
> >
> > marathon user: mapr
> > myriadframeuser: mapr
> > myriadsuperuser: mapr
> >
> >
> >
> > /
> > drwxr-xr-x 10 mapr mapr 4 KB Sep 25 09:15
> > hadoop-2.7.0
> > -rw-r--r-- 1 mapr mapr 76 KB Nov 18
> > 15:59 conf
> > -rw-r--r-- 1 mapr mapr 306 MB
> > Nov 18 15:59 hadoop-2.7.0.tgz
> > -rw-r--r-- 1 mapr mapr 121 KB Nov 18
> > 15:59 stderr
> > -rw-r--r--
> >
> >
> > /hadoop-2.7.0/
> >
> > drwxrwxrwx 2 mapr root 4 KB Nov 18 15:57
> > logs
> > drwxr-xr-x 2 mapr root 4 KB Sep 25
> > 09:16 bin
> > drwxr-xr-x 3 mapr root 4 KB Sep 25
> > 09:15 etc
> > drwxr-xr-x 2 mapr root 4 KB Sep 25
> > 09:15 include
> > drwxr-xr-x 3 mapr root 4 KB Sep 25
> > 09:15 lib
> > drwxr-xr-x 2 mapr root 4 KB Sep 25
> > 09:16 libexec
> > drwxr-xr-x 2 mapr root 4 KB Sep 25
> > 09:16 sbin
> > drwxr-xr-x 4 mapr root 4 KB Sep 25
> > 09:15 share
> > 1
> >
> > /hadoop-2.7.0/bin
> >
> > -rwxr-xr-x 1 mapr root 9 KB Jul 09
> > 05:38 hadoop
> > -rwxr-xr-x 1 mapr root 10 KB Jul 09
> > 05:38 hadoop.cmd
> > -rwxr-xr-x 1 mapr root 12 KB Jul 09
> > 05:38 hdfs
> > -rwxr-xr-x 1 mapr root 7 KB Jul 09
> > 05:38 hdfs.cmd
> > -rwxr-xr-x 1 mapr root 7 KB Jul 09
> > 05:37 mapred
> > -rwxr-xr-x 1 mapr root 6 KB Jul 09
> > 05:37 mapred.cmd
> > -rwxr-xr-x 1 mapr root 2 KB Jul 09
> > 05:38 rcc
> > -rwxr-xr-x 1 mapr root 128 KB Jul 09
> > 05:38 test-container-executor
> > -rwxr-xr-x 1 mapr root 14 KB Jul 09
> > 05:38 yarn
> > -rwxr-xr-x 1 mapr root 11 KB Jul 09
> > 05:38 yarn.cmd
> > ----r-x--- 1 root mapr 111 KB Jul
> > 09 05:38 container-executor
> >
> > mapr mapr 786 B Nov 18 15:59 stdout
> >
> >
> > /hadoop-2.7.0/etc
> > mode nlink uid gid size mtime
> >
> > drwxr-xr-x 2 mapr root 4 KB Nov 18
> > 15:59 hadoop
> >
> > /hadoop-2.7.0/etc/hadoop
> > -rw-r--r-- 1 mapr root 4 KB Jul
> > 09 05:38 capacity-scheduler.xml
> > -rw-r--r-- 1 mapr root 1 KB Jul
> > 09 05:38 configuration.xsl
> > -rw-r--r-- 1 root root 168 B Nov
> > 02 18:22 container-executor.cfg
> > -rw-r--r-- 1 mapr root 775 B Nov 02
> > 18:22 core-site.xml
> > -rw-r--r-- 1 mapr root 631 B Jul 09
> > 05:38 fair-scheduler.xml
> > -rw-r--r-- 1 mapr root 4 KB Jul
> > 09 05:38 hadoop-env.cmd
> > -rw-r--r-- 1 mapr root 4 KB Jul
> > 09 05:38 hadoop-env.sh
> > -rw-r--r-- 1 mapr root 2 KB Jul
> > 09 05:38 hadoop-metrics.properties
> > -rw-r--r-- 1 mapr root 3 KB Jul
> > 09 05:38 hadoop-metrics2.properties
> > -rw-r--r-- 1 mapr root 9 KB Jul
> > 09 05:38 hadoop-policy.xml
> > -rw-r--r-- 1 mapr root 775 B Jul 09
> > 05:38 hdfs-site.xml
> > -rw-r--r-- 1 mapr root 1 KB Jul
> > 09 05:38 httpfs-env.sh
> > -rw-r--r-- 1 mapr root 2 KB Jul
> > 09 05:38 httpfs-log4j.properties
> > -rw-r--r-- 1 mapr root 21 B Jul
> > 09 05:38 httpfs-signature.secret
> > -rw-r--r-- 1 mapr root 620 B Jul 09
> > 05:38 httpfs-site.xml
> > -rw-r--r-- 1 mapr root 3 KB Jul
> > 09 05:38 kms-acls.xml
> > -rw-r--r-- 1 mapr root 1 KB Jul
> > 09 05:38 kms-env.sh
> > -rw-r--r-- 1 mapr root 2 KB Jul
> > 09 05:38 kms-log4j.properties
> > -rw-r--r-- 1 mapr root 5 KB Jul
> > 09 05:38 kms-site.xml
> > -rw-r--r-- 1 mapr root 11 KB Jul 09
> > 05:38 log4j.properties
> > -rw-r--r-- 1 mapr root 931 B Jul 09
> > 05:38 mapred-env.cmd
> > -rw-r--r-- 1 mapr root 1 KB Jul
> > 09 05:38 mapred-env.sh
> > -rw-r--r-- 1 mapr root 4 KB Jul
> > 09 05:38 mapred-queues.xml.template
> > -rw-r--r-- 1 mapr root 1 KB Nov
> > 02 18:22 mapred-site.xml
> > -rw-r--r-- 1 mapr root 1 KB Jul
> > 09 05:38 mapred-site.xml.template
> > -rw-r--r-- 1 mapr root 3 KB Nov
> > 18 15:57 myriad-config-default.yml
> > -rw-r--r-- 1 mapr root 10 B Jul
> > 09 05:38 slaves
> > -rw-r--r-- 1 mapr root 2 KB Nov
> > 02 18:22 ssl-client.xml
> > -rw-r--r-- 1 mapr root 2 KB Jul
> > 09 05:38 ssl-client.xml.example
> > -rw-r--r-- 1 mapr root 2 KB Nov
> > 02 18:22 ssl-server.xml
> > -rw-r--r-- 1 mapr root 2 KB Jul
> > 09 05:38 ssl-server.xml.example
> > -rw-r--r-- 1 mapr root 2 KB Jul
> > 09 05:38 yarn-env.cmd
> > -rw-r--r-- 1 mapr root 5 KB Jul
> > 09 05:38 yarn-env.sh
> > -rw-r--r-- 1 mapr root 2 KB Oct
> > 01 16:56 yarn-site-2015-10-01.20-56.xml
> > -rw-r--r-- 1 mapr root 2 KB Oct
> > 08 06:32 yarn-site-2015-10-08.10-32.xml
> > -rw-r--r-- 1 mapr root 2 KB Oct
> > 16 15:33 yarn-site-2015-10-16.19-33.xml
> > -rw-r--r-- 1 mapr root 2 KB Nov
> > 02 18:22 yarn-site-2015-11-02.23-22.xml
> > -rw-r--r-- 1 mapr mapr 76 KB Nov 18
> > 16:01 yarn-site.xml
> > -rw-r--r-- 1 mapr root 4 KB Jul
> > 09 05:38 yarn-site.xml.template
> >
> > On Wed, Nov 18, 2015 at 12:41 PM, John Omernik <j...@omernik.com> wrote:
> >
> >> So there are two issues currently I am looking into. The first is the
> >> permissions of directories. I'd still like to get the feelings from the
> >> group on that because I've not managed to get Myriad/Yarn working on one
> >> cluster (based on Ubuntu 14.04) but can't get it to work on another
> cluster
> >> based on Red Hat 7. It's strange from what I can tell everything is the
> >> same, but the Redhat cluster complains about the the permissions
> >> /etc/hadoop not being owned by root (it's owned by mapr:root but on the
> >> ubuntu cluster that works fine with the same ownership!) I do notice
> that
> >> my build times reported by mapr are different.. but that may just be the
> >> build for Redhat vs the build for Ubuntu? Still digging into that
> one...
> >>
> >> As to the hostname / mapr.host issue. I found a neat hack that may work
> >> for folks
> >>
> >> By setting this in my myriad config:
> >>
> >> yarnEnvironment:
> >>
> >> YARN_HOME: hadoop-2.7.0
> >>
> >> YARN_NODEMANAGER_OPTS: "-Dnodemanager.resource.io-spindles=4.0
> >> -Dmapr.host=$(hostname -f)"
> >>
> >>
> >> I am able to get the mapr.host set back to be the correct hostname where
> >> the nodemanager is running, this helps with a number of issues. I
> thought
> >> about this, and realized it would be better if I could get the hostname
> to
> >> the createTTVolume script but use a unique name for the mount point
> (What
> >> if I have multiple NMs on a single physical node?)
> >>
> >> So, I tried:
> >>
> >> YARN_NODEMANAGER_OPTS: "-Dnodemanager.resource.io-spindles=4.0
> >> -Dmapr.host=$(basename `pwd`)"
> >>
> >>
> >> Thinking that if I could get the directory name of the "run" in my
> >> sandbox, I should be reasonably assured of uniqueness. That seemed to
> work
> >> when the nodemanager kicked off the the command:
> >>
> >> /opt/mapr/server/createTTVolume.sh hadoopmapr2.brewingintel.com
> >> /var/mapr/local/48833481-0c7a-4728-8f93-bcf9b545ad81/mapred
> >> /var/mapr/local/48833481-0c7a-4728-8f93-bcf9b545ad81/mapred/nodeManager
> yarn
> >>
> >>
> >> However, the script never returned, and the task failed. So at this
> >> point, I think we can get good info passed to the MapR script, but I am
> not
> >> sure how the script or how mapr is creating the volume to work in
> >> conjunction with Yarn.
> >>
> >> To summarize: Right now I have Myriad working... but only on a Ubuntu
> >> Mesos cluster. I have NOT changed my /tmp location for the Mesos
> slaves, I
> >> can't get things working on my Redhat cluster, and I seemed to have
> found a
> >> hacky workaround for the mapr.host. Here is the script I am using to
> build
> >> my tgz file...
> >>
> >>
> >> #!/bin/bash
> >>
> >>
> >> BASEDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
> >>
> >>
> >> echo "#### Base Location: $BASEDIR"
> >>
> >>
> >> ################## BUILD VARIABLES
> >>
> >>
> >> HADOOP_VER="hadoop-2.7.0"
> >>
> >> HADOOP_BASE="/opt/mapr/hadoop"
> >>
> >> HADOOP_HOME="${HADOOP_BASE}/${HADOOP_VER}"
> >>
> >> MYRIAD_BUILD="/opt/mapr/myriad/incubator-myriad"
> >>
> >> CONF_LOC="/mapr/brewpot/mesos/myriad/conf"
> >>
> >> URI_LOC="/mapr/brewpot/mesos/myriad"
> >>
> >>
> >>
> >>
> >> #################################
> >>
> >> #Clean the working directory
> >>
> >> echo "#### Cleaning Build DIR and old tgz"
> >>
> >> sudo rm -rf ./${HADOOP_VER}
> >>
> >> sudo rm -rf ./${HADOOP_VER}.tgz
> >>
> >>
> >> #Copy a fresh copy of the hadoopz
> >>
> >> echo "#### Copying Clean Build"
> >>
> >>
> >> # I go here and tar with h and p. h is so all the symlinked items in the
> >> MapR get put into the tgz and p to preserver permissions
> >>
> >> cd $HADOOP_BASE
> >>
> >> sudo tar zcfhp ${BASEDIR}/${HADOOP_VER}.tgz $HADOOP_VER
> >>
> >>
> >> echo "#### Untaring New Build"
> >>
> >> # I untar things in a new location so I can play without affecting the
> >> "stock" install
> >>
> >> cd $BASEDIR
> >>
> >> sudo tar zxfp ${HADOOP_VER}.tgz
> >>
> >>
> >> echo "#### Now remove source tgz to get ready for build"
> >>
> >> sudo rm ${HADOOP_VER}.tgz
> >>
> >>
> >>
> >>
> >> # This permission combination seems to work. I go and grab the
> >> container-executor from the stock build so that I have the setuid
> version.
> >>
> >> echo "#### Cleaning Base Build Logs, yarn-site, and permissions"
> >>
> >> sudo rm $HADOOP_VER/etc/hadoop/yarn-site.xml
> >>
> >> sudo rm -rf $HADOOP_VER/logs/*
> >>
> >> sudo chown mapr:mapr ${HADOOP_VER}
> >>
> >> sudo chown -R mapr:root ${HADOOP_VER}/*
> >>
> >> sudo chown root:root ${HADOOP_VER}/etc/hadoop/container-executor.cfg
> >>
> >> sudo cp --preserve ${HADOOP_HOME}/bin/container-executor
> >> ${HADOOP_VER}/bin/
> >>
> >>
> >>
> >>
> >> #Copy the jars from Myriad into the Hadoop libs folders (You will need
> to
> >> have Myriad build first with the root of your build being $MYRIAD_BUILD
> >>
> >> echo "#### Copying Myriad Jars"
> >>
> >> sudo cp $MYRIAD_BUILD/myriad-scheduler/build/libs/*.jar
> >> $HADOOP_VER/share/hadoop/yarn/lib/
> >>
> >> sudo cp
> >> $MYRIAD_BUILD/myriad-executor/build/libs/myriad-executor-0.1.0.jar
> >> $HADOOP_VER/share/hadoop/yarn/lib/
> >>
> >>
> >> #Address Configs
> >>
> >>
> >> # First take the myriad-config-default.yml and put it into the
> >> $HADOOP/etc/hadoop so it's in the tarball
> >>
> >> echo "#### Updating myriad-config-default.yml"
> >>
> >> sudo cp ${CONF_LOC}/myriad-config-default.yml ${HADOOP_VER}/etc/hadoop/
> >>
> >>
> >> # Tar all the things with all the privs
> >>
> >> echo "#### Tarring all the things"
> >>
> >> sudo tar zcfhp ${HADOOP_VER}.tgz ${HADOOP_VER}/
> >>
> >>
> >> # Copy to the URI location... note I am using MapR so I cp it directly
> to
> >> the MapFS location via NFS share, it would probably be good to use a
> hadoop
> >> copy command for interoperability
> >>
> >> echo "#### Copying to HDFS Location"
> >>
> >> cp ${HADOOP_VER}.tgz ${URI_LOC}/
> >>
> >>
> >> # I do this because it worked... not sure if I remo
> >>
> >> #sudo chown mapr:mapr ${URI_LOC}/${HADOOP_VER}.tgz
> >>
> >>
> >> #echo "#### Cleaning unpacked location"
> >>
> >> sudo rm -rf ./${HADOOP_VER}
> >>
> >> sudo rm ./${HADOOP_VER}.tgz
> >>
> >>
> >>
> >>
> >> On Wed, Nov 18, 2015 at 9:40 AM, yuliya Feldman <
> >> yufeld...@yahoo.com.invalid> wrote:
> >>
> >>> I would love to have that piece of code "configurable", but it is not
> at
> >>> the moment.
> >>> Will send you patch offline.
> >>> Thanks,Yuliya
> >>> From: John Omernik <j...@omernik.com>
> >>> To: dev@myriad.incubator.apache.org; yuliya Feldman <
> >>> yufeld...@yahoo.com>
> >>> Sent: Wednesday, November 18, 2015 6:02 AM
> >>> Subject: Re: Struggling with Permissions
> >>>
> >>> Yuliya, I would be interested in the patch for MapR, is that a patch
> for
> >>> Myriad or a patch for Hadoop on MapR? I wonder if there is a hadoop
> env
> >>> file I could modified in my TGZ to help address the issue on my nodes
> as
> >>> well. Can you describe what "mapr.host" is and if I can force overwrite
> >>> that in my ENV file or will MapR clobber that at a later point in
> >>> execution? I am thinking that with some simple sed, I could "fix" the
> >>> conf
> >>> file.
> >>>
> >>> Wait, I suppose there is no way for me edit the command used to run the
> >>> node manager... there's a thought. Could Myriad provide an ENV value or
> >>> something that would allow us to edit the command or insert something
> >>> into
> >>> the command that is used to run the NM? (below is the the command on
> my
> >>> cluster) Basically, if there was a way to template that and alter it
> in
> >>> the Myriad config, I could add commands to update the variables in the
> >>> conf
> >>> file before it's copied to yarn-site on every node... just spitballing
> >>> ideas here...
> >>>
> >>>
> >>>
> >>> sudo tar -zxpf hadoop-2.7.0-NM.tar.gz && sudo chown mapr . && cp conf
> >>> hadoop-2.7.0/etc/hadoop/yarn-site.xml; export YARN_HOME=hadoop-2.7.0;
> >>> sudo
> >>> -E -u mapr -H env YARN_HOME=hadoop-2.7.0
> >>> YARN_NODEMANAGER_OPTS=-Dnodemanager.resource.io-spindles=4.0
> >>> -Dyarn.resourcemanager.hostname=myriad.marathon.mesos
> >>>
> >>>
> -Dyarn.nodemanager.container-executor.class=org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor
> >>> -Dnodemanager.resource.cpu-vcores=2
> -Dnodemanager.resource.memory-mb=8192
> >>> -Dmyriad.yarn.nodemanager.address=0.0.0.0:31984
> >>> -Dmyriad.yarn.nodemanager.localizer.address=0.0.0.0:31233
> >>> -Dmyriad.yarn.nodemanager.webapp.address=0.0.0.0:31716
> >>> -Dmyriad.mapreduce.shuffle.port=31786 /bin/yarn nodemanager
> >>>
> >>>
> >>>
> >>> On Tue, Nov 17, 2015 at 4:44 PM, yuliya Feldman
> >>> <yufeld...@yahoo.com.invalid
> >>> > wrote:
> >>>
> >>> > Hadoop (not Mapr) requires whole path starting from "/" be owned by
> >>> root
> >>> > and writable only by root
> >>> > The second problem is exactly what I was talking about configuration
> >>> being
> >>> > taken from RM that overwrites local one
> >>> > I can give you a patch to mitigate the issue for Mapr if you are
> >>> building
> >>> > from source.
> >>> > Thanks,Yuliya
> >>> > From: John Omernik <j...@omernik.com>
> >>> > To: dev@myriad.incubator.apache.org
> >>> > Sent: Tuesday, November 17, 2015 1:15 PM
> >>> > Subject: Re: Struggling with Permissions
> >>> >
> >>> > Well sure /tmp is world writeable but /tmp/mesos is not world
> writable
> >>> thus
> >>> > there is a sandbox to play in there... or am I missing something. Not
> >>> to
> >>> > mention my tmp is rwt which is world writable but only the creator or
> >>> root
> >>> > can modify (based on the googles).
> >>> > Yuliya:
> >>> >
> >>> > I am seeing a weird behavior with MapR as it relates to (I believe)
> the
> >>> > mapr_direct_shuffle.
> >>> >
> >>> > In the Node Manager logs, I see things starting and it saying
> >>> "Checking for
> >>> > local volume, if local volume is not present command will create and
> >>> mount
> >>> > it"
> >>> >
> >>> > Command invoked is : /opt/mapr/server/createTTVolume.sh
> >>> > hadoopmapr7.brewingintel.com /var/mapr/local/
> >>> > hadoopmapr2.brewingintel.com/mapred /var/mapr/local/
> >>> > hadoopmapr2.brewingintel.com/mapred/nodeManager yarn
> >>> >
> >>> >
> >>> > What is interesting here is hadoopmapr7 is the nodemanager it's
> trying
> >>> to
> >>> > start on, however the mount point it's trying to create is
> hadoopmapr2
> >>> > which is the node the resource manager happened to fall on... I was
> >>> very
> >>> > confused by that because in no place should hadoopmapr2 be "known" to
> >>> the
> >>> > nodemanager, because it thinks the resource manager hostname is
> >>> > myriad.marathon.mesos.
> >>> >
> >>> > So why was it hard coding to the node the resource manager is running
> >>> on?
> >>> >
> >>> > Well if I look at the conf file in the sandbox (the file that gets
> >>> copied
> >>> > to be yarn-site.xml for node managers. There ARE four references the
> >>> > hadoopmapr2. Three of the four say "source programatically" and one
> is
> >>> just
> >>> > set... that's mapr.host. Could there be some down stream hinkyness
> >>> going
> >>> > on with how MapR is setting hostnames? All of these variables seem
> >>> "wrong"
> >>> > in that mapr.host (on the node manager) should be hadoopmapr7 in this
> >>> case,
> >>> > and the resource managers should all be myriad.marathon.mesos. I'd
> be
> >>> > interested in your thoughts here, because I am stumped at how these
> are
> >>> > getting set.
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>>
> <property><name>yarn.resourcemanager.address</name><value>hadoopmapr2:8032</value><source>programatically</source></property>
> >>> > <property><name>mapr.host</name><value>hadoopmapr2.brewingintel.com
> >>> > </value></property>
> >>> >
> >>> >
> >>>
> <property><name>yarn.resourcemanager.resource-tracker.address</name><value>hadoopmapr2:8031</value><source>programatically</source></property>
> >>> >
> >>> >
> >>>
> <property><name>yarn.resourcemanager.admin.address</name><value>hadoopmapr2:8033</value><source>programatically</source></property>
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > On Tue, Nov 17, 2015 at 2:51 PM, Darin Johnson <
> >>> dbjohnson1...@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Yuliya: Are you referencing yarn.nodemanager.hostname or a mapr
> >>> specific
> >>> > > option?
> >>> > >
> >>> > > I'm working right now on passing a
> >>> > > -Dyarn.nodemanager.hostname=offer.getHostName(). Useful if you've
> >>> got
> >>> > > extra ip's for a san or management network.
> >>> > >
> >>> > > John: Yeah the permissions on the tarball are a pain to get right.
> >>> I'm
> >>> > > working on Docker Support and a build script for the tarball, which
> >>> > should
> >>> > > make things easier. Also, to the point of using world writable
> >>> > directories
> >>> > > it's a little scary from the security side of things to allow
> >>> executables
> >>> > > to run there, especially things running as privileged users. Many
> >>> > distro's
> >>> > > of linux will mount /tmp noexec.
> >>> > >
> >>> > > Darin
> >>> > >
> >>> > > On Tue, Nov 17, 2015 at 2:53 PM, yuliya Feldman
> >>> > > <yufeld...@yahoo.com.invalid
> >>> > > > wrote:
> >>> > >
> >>> > > > Please change workdir directory for mesos slave to one that is
> not
> >>> /tmp
> >>> > > > and make sure that dir is owned by root.
> >>> > > > There is one more caveat with binary distro and MapR - in Myriad
> >>> code
> >>> > for
> >>> > > > binary distro configuration is copied from RM to NMs - it doe
> snot
> >>> work
> >>> > > for
> >>> > > > MapR since we need hostname (yes for the sake of local volumes)
> to
> >>> be
> >>> > > > unique.
> >>> > > > MapR will have Myriad release to handle this situation.
> >>> > > > From: John Omernik <j...@omernik.com>
> >>> > > > To: dev@myriad.incubator.apache.org
> >>> > > > Sent: Tuesday, November 17, 2015 11:37 AM
> >>> > > > Subject: Re: Struggling with Permissions
> >>> > > >
> >>> > > > Oh hey, I found a post by me back on Sept 9. I looked at the
> >>> Jiras and
> >>> > > > followed the instructions with the same errors. At this point do
> I
> >>> > still
> >>> > > > need to have a place where the entire path is owned by root? That
> >>> seems
> >>> > > > like a an odd requirement (a changed of each node to facilitate a
> >>> > > > framework)
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > > On Tue, Nov 17, 2015 at 1:25 PM, John Omernik <j...@omernik.com>
> >>> > wrote:
> >>> > > >
> >>> > > > > Hey all, I am struggling with permissions on myriad, trying to
> >>> get
> >>> > the
> >>> > > > > right permissions in the tgz as well as who to run as. I am
> >>> running
> >>> > in
> >>> > > > > MapR, which means I need to run as mapr or root (otherwise my
> >>> volume
> >>> > > > > creation scripts will fail on MapR, MapR folks, we should talk
> >>> more
> >>> > > about
> >>> > > > > those scripts)
> >>> > > > >
> >>> > > > > But back to the code, I've had lots issues. When I run the
> >>> > > Frameworkuser
> >>> > > > > and Superuser as mapr, it unpacks everything as MapR and I get
> a
> >>> > > > > "/bin/container-executor" must be owned by root but is owned by
> >>> 700
> >>> > (my
> >>> > > > > mapr UID).
> >>> > > > >
> >>> > > > > So now I am running as root, and I am getting the error below
> as
> >>> it
> >>> > > > > relates to /tmp. I am not sure which /tmp this refers to. the
> >>> /tmp
> >>> > that
> >>> > > > my
> >>> > > > > slave is executing in? (i.e. my local mesos agent /tmp
> >>> directory) or
> >>> > my
> >>> > > > > MaprFS /tmp directory (both of which are world writable, as
> /tmp
> >>> > > > typically
> >>> > > > > is... or am I mistaken here?)
> >>> > > > >
> >>> > > > > Any thoughts on how to get this to resolve? This is when
> >>> nodemanager
> >>> > is
> >>> > > > > trying to start running as root and root for both of my Myriad
> >>> users.
> >>> > > > >
> >>> > > > > Thanks!
> >>> > > > >
> >>> > > > >
> >>> > > > > Caused by: ExitCodeException exitCode=24: File /tmp must not be
> >>> world
> >>> > > or
> >>> > > > group writable, but is 1777
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > >
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > > >
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >
>
>
>
