Thank you sir, I appreciate your help! Do you think that it would make sense to also add try-catch block in the setUp method for NullPointerException and print out some warning message that hadoop was not correctly initialized and thus there will be failures in the tests? I believe that it might people that missed the umask requirement in docs to get it quickly resolved.
Jarcec On Mon, Oct 15, 2012 at 01:05:44AM -0700, Mohammad Islam wrote: > My previous reply was for another email. Sorry for the confusion. > > I found my umask also defaulted to 0022. That's why we never experienced the > issue. > I agree we should update that Oozie doc. > Created a JIRA: > https://issues.apache.org/jira/browse/OOZIE-1021 > > > Regards, > Mohammad > > > ________________________________ > From: Jarek Jarcec Cecho <[email protected]> > To: [email protected] > Sent: Sunday, October 14, 2012 3:16 PM > Subject: Re: Oozie tests seems to be requiring umask 022 > > We probably miss understand each other sir. I've experienced described issues > during building Oozie on my local box before any deployment for real life > tests. I've justed checkout repository and tried to build it (mvn package) - > and almost all tests failed as I described in my original email. Therefore I > believe that there is no other special log other than test related logs that > I've read and provided appropriate lines, or at least those lines that I > believe are related to the issue. I might send entire logs if you want me to. > > Jarcec > > On Sun, Oct 14, 2012 at 03:06:16PM -0700, Mohammad Islam wrote: > > You have to see the launcher log. > > 1. First click on the your ooze job. > > 2. Select the corresponding shell action. it will pop-up a new window. > > 3. Select the right of "Console URL" which will take you to the hadoop JT > > site. > > 4. Select hadoop map task and click through to find the log. > > > > Regards, > > Mohammad > > > > > > ________________________________ > > From: Jarek Jarcec Cecho <[email protected]> > > To: [email protected] > > Sent: Sunday, October 14, 2012 2:52 PM > > Subject: Re: Oozie tests seems to be requiring umask 022 > > > > Hi Mohamad, > > I've actually did not do anything special. I've just checked out svn > > repository and executed "mvn test". Based on the maven output I'm expecting > > that Oozie is compiling against 1.0.1 and 2.0.0-alpha, I'm not sure which > > version is used in tests thought. > > > > Appropriate maven output that used: > > [INFO] Apache Oozie Hadoop 1.0.1.oozie-3.4.0-SNAPSHOT .... SUCCESS [0.236s] > > [INFO] Apache Oozie Hadoop Distcp 1.0.1.oozie-3.4.0-SNAPSHOT SUCCESS > > [0.151s] > > [INFO] Apache Oozie Hadoop 1.0.1.oozie-3.4.0-SNAPSHOT Test SUCCESS [0.360s] > > [INFO] Apache Oozie Hadoop 2.0.0-alpha.oozie-3.4.0-SNAPSHOT SUCCESS > > [0.427s] > > [INFO] Apache Oozie Hadoop 2.0.0-alpha.oozie-3.4.0-SNAPSHOT Test SUCCESS > > [0.411s] > > [INFO] Apache Oozie Hadoop Distcp 2.0.0-alpha.oozie-3.4.0-SNAPSHOT SUCCESS > > [0.137s] > > > > Jarcec > > > > On Fri, Oct 12, 2012 at 09:55:09PM -0700, Mohammad Islam wrote: > > > Hi Jarek, > > > Your test case is running against which hadoop version. > > > Regards, > > > Mohammad > > > > > > > > > ________________________________ > > > From: Jarek Jarcec Cecho <[email protected]> > > > To: [email protected] > > > Sent: Friday, October 12, 2012 11:27 AM > > > Subject: Oozie tests seems to be requiring umask 022 > > > > > > Hi Oozie developers, > > > I've recently worked on OOZIE-1012 and I've noticed that on my box oozie > > > tests seems to be requiring umask 022 (linux, Ubuntu 12.04), which > > > actually seems to be implied by HDFS not by Oozie. > > > > > > If I run tests with umask 002 (my default), most of the tests will fail > > > with following message: > > > > > > Cannot lock storage build/test/data/dfs/name1. The directory is already > > > locked. > > > > > > I believe that this error message is just a consequence of following > > > NullPointerException that appears in setUp method of affected test cases: > > > > > > java.lang.NullPointerException > > > at > > > org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422) > > > at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) > > > at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124) > > > at > > > org.apache.oozie.test.XTestCase.setUpEmbeddedHadoop(XTestCase.java:708) > > > at org.apache.oozie.test.XTestCase.setUp(XTestCase.java:281) > > > at org.apache.oozie.test.XFsTestCase.setUp(XFsTestCase.java:58) > > > at > > > org.apache.oozie.action.hadoop.ActionExecutorTestCase.setUp(ActionExecutorTestCase.java:62) > > > > > > I believe that reason of this NullPointerException is absence of usable > > > data directories as *-output logs contains following fragments: > > > > > > 12/10/12 11:16:12 WARN datanode.DataNode: Invalid directory in > > > dfs.data.dir: Incorrect permission for build/test/data/dfs/data/data1, > > > expected: rwxr-xr-x, while actual: rwxrwxr-x > > > 12/10/12 11:16:12 WARN datanode.DataNode: Invalid directory in > > > dfs.data.dir: Incorrect permission for build/test/data/dfs/data/data2, > > > expected: rwxr-xr-x, while actual: rwxrwxr-x > > > 12/10/12 11:16:12 ERROR datanode.DataNode: All directories in > > > dfs.data.dir are invalid. > > > > > > > > > Please note that this output was generated with umask 002 and changing it > > > to 022 will fix the issue. > > > > > > Does anyone else noticed this behaviour (failures) as well? If so then it > > > might make sense to document this need on HowToContribute page [1] and > > > maybe improve the XTestCase.setUpEmbeddedHadoop by catching NPE and > > > informing developer that we're not able to bootstrap MiniDFSCluster? > > > > > > Jarcec > > > > > > Links: > > > 1: http://incubator.apache.org/oozie/HowToContribute.html
signature.asc
Description: Digital signature
