Hi Tim. I generally deploy bigtop using the vagrant recipes, which set this memory, users, init hdfs automatically . Meanwhile, when we test vendor distros, those settings are usually also set for you.
And you're right the pig tests aren't idemopotent. Would love to see a jira for test idemootency. I think some others are also not idemopotent. Glad the smoke-tests framework is working for you. I think your the 3rd person I know who has used it as a quick way to get up and running with bigtop smoke tests. Would you like to summarize your findings in a JIra outlining improvements that you'd like to see? If so I can work on them this wknd. > On Sep 24, 2014, at 5:35 PM, Tim Harsch <[email protected]> wrote: > > I'm glad to report that I got the pig gradle test to work. > > > I then run it as unix user 'tom' (the vmWare VM used in Hadoop for Dummies). > Problem no home directory. Fixed that. > > Next problem: > /tmp/hadoop-yarn/staging is perms 700 ( and not addressed by > /usr/lib/hadoop/libexec/init-hdfs.sh, should it be?) > Fix: > % hdfs dfs -chmod -R 1777 /tmp/hadoop-yarn/staging > > Next problem, pig jobs are in "ACCEPTED" state and never move to running, > with wonderfully no hints in logs. Solution: Up the memory settings in > mapred-site.xml > <property> > <name>mapreduce.map.memory.mb</name> > <value>2400</value> > </property> > <property> > <name>mapreduce.map.java.opts</name> > <value>-Xmx2048m</value> > </property> > <property> > <name>mapreduce.reduce.memory.mb</name> > <value>4400</value> > </property> > <property> > <name>mapreduce.reduce.java.opts</name> > <value>-Xmx4096m</value> > </property> > > I guess the defaults are just too low… > > Next problem, test aborts due to test resource clean up issue. Solved with > the following change to the test: > [tom@localhost pig]$ git diff -- TestPigSmoke.groovy > diff --git a/bigtop-tests/smoke-tests/pig/TestPigSmoke.groovy > b/bigtop-tests/smoke-tests/pig/TestPigSmoke.groovy > index 9902267..9511626 100644 > --- a/bigtop-tests/smoke-tests/pig/TestPigSmoke.groovy > +++ b/bigtop-tests/smoke-tests/pig/TestPigSmoke.groovy > @@ -41,6 +41,7 @@ class TestPigSmoke { > @AfterClass > public static void tearDown() { > sh.exec("hadoop fs -rmr -skipTrash pigsmoketest"); > + sh.exec("hadoop fs -rmr -skipTrash pig-output-wordcount"); > } > > @BeforeClass > > > So, not casting stones… but when are the smoke tests run? Seems like it > would be optimal if they would get run on a bigtop distro prior to a bigtop > release, on each of the supported OS's (sounds like a lot of work… is there > something like that?). For the case of users stepping in to bigtop for the > first time (raises hand…) it would be nice if the docs could say install > big top and then add a non-privileged user, up memory parameters and (??) > etc. Now run all the tests and ensure they pass. > > Tim > > > From: Jay Vyas <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Wednesday, September 24, 2014 10:36 AM > To: "[email protected]" <[email protected]> > Subject: Re: smoke tests in 0.7.0 > > Hi Tim. Great to hear your making progress.. > > Your on the right track but i forgot the details. But yes: you'll have to > run some simple commands as user hdfs to set up permissions for "root". > > You can try running your tests as user "hdfs". That is a good hammer to use > since hdfs is super user on Hadoop systems that use HDFS as the file system. > > In other systems like gluster, we usually have root as the super user. > > Directory perms are always a pain in Hadoop setup. Anything you suggest to > make it more user friendly maybe create a jira. On this route, we have done > bigtop-1200 which now encodes all info in a json file so that any FileSystem > Can use the bigtop for provisioner. I can discuss that with you also if you > want later on (send me a private message). > > I haven't merged that to replace init-hdfs , but it is functionally > equivalent , and can be found in the code base (see jiras bigtop-952 and > bigtop-1200 for details). > > > On Sep 24, 2014, at 12:50 PM, Tim Harsch <[email protected]> wrote: > >> Thanks that was helpful. So, I looked closely at the TestPigSmoke test and >> tried repeating it's steps manually, which really helped. I was able to >> track the issue down to a perms problem for running as user root. See this: >> >> [root@localhost pig]# hadoop fs -ls / >> Found 6 items >> drwxrwxrwx - hdfs supergroup 0 2014-09-24 00:32 /benchmarks >> drwxr-xr-x - hbase hbase 0 2014-09-24 00:32 /hbase >> drwxr-xr-x - solr solr 0 2014-09-24 00:32 /solr >> drwxrwxrwt - hdfs supergroup 0 2014-09-24 18:33 /tmp >> drwxr-xr-x - hdfs supergroup 0 2014-09-24 00:33 /user >> drwxr-xr-x - hdfs supergroup 0 2014-09-24 00:32 /var >> >> [root@localhost pig]# hadoop fs -ls /tmp >> Found 2 items >> drwxrwxrwx - mapred mapred 0 2014-09-24 00:37 /tmp/hadoop-yarn >> drwxr-xr-x - root supergroup 0 2014-09-24 01:29 >> /tmp/temp-1450563950 >> >> [root@localhost pig]# hadoop fs -ls /tmp/hadoop-yarn >> Found 1 items >> drwxrwx--- - mapred mapred 0 2014-09-24 00:37 >> /tmp/hadoop-yarn/staging >> >> [root@localhost pig]# hadoop fs -ls /tmp/hadoop-yarn/staging >> ls: Permission denied: user=root, access=READ_EXECUTE, >> inode="/tmp/hadoop-yarn/staging":mapred:mapred:drwxrwx--- >> >> OK, makes sense. But I'm a little confused.. I thought all the directories >> would be set up correctly by the script >> /usr/lib/hadoop/libexec/init-hdfs.sh, which as you can tell from the above >> output, I did run it. From the docs I've read the assumption is that after >> running /usr/lib/hadoop/libexec/init-hdfs.sh all tests should pass… but >> perhaps I missed some instruction somewhere. >> >> Tim >> >> >> >> From: jay vyas <[email protected]> >> Reply-To: "[email protected]" <[email protected]> >> Date: Wednesday, September 24, 2014 5:46 AM >> To: "[email protected]" <[email protected]> >> Subject: Re: smoke tests in 0.7.0 >> >> Thanks tim. It could be related to permissions on the DFS... depending on >> the user you are running the job as. >> >> Can you paste the error you got ? In general the errors should be eay to >> track down in smoke-tests (you can just hack some print statements into the >> groovy script under pig/). >> Also, the stack trace should give you some information ?
