That actually make complete sense, thank you for pointing it out Alejandro. I'll try to extract the use case and report this bug in upstream Hadoop (HDFS) project.
I would still recommend documenting this requirement on Oozie web pages thought. Jarcec On Mon, Oct 15, 2012 at 10:47:08AM -0700, Alejandro Abdelnur wrote: > shouldnt the fix be done in hdfs? > > Alejandro > > On Oct 15, 2012, at 9:37 AM, Robert Kanter <[email protected]> wrote: > > > Catching the NPE exception and printing a more useful message is a good > > idea, but if there's other possible things that could cause an NPE, the > > "useful" message would be misleading. Perhaps a better solution might be > > to explicitly check the umask somehow and print/log a warning or something. > > Though from a quick Google search, it doesn't look like there's an easy > > way to check (or change) the umask from Java. > > > > - Robert > > > > > > On Mon, Oct 15, 2012 at 7:51 AM, Jarek Jarcec Cecho > > <[email protected]>wrote: > > > >> Thank you sir, > >> I appreciate your help! Do you think that it would make sense to also add > >> try-catch block in the setUp method for NullPointerException and print out > >> some warning message that hadoop was not correctly initialized and thus > >> there will be failures in the tests? I believe that it might people that > >> missed the umask requirement in docs to get it quickly resolved. > >> > >> Jarcec > >> > >> On Mon, Oct 15, 2012 at 01:05:44AM -0700, Mohammad Islam wrote: > >>> My previous reply was for another email. Sorry for the confusion. > >>> > >>> I found my umask also defaulted to 0022. That's why we never experienced > >> the issue. > >>> I agree we should update that Oozie doc. > >>> Created a JIRA: > >>> https://issues.apache.org/jira/browse/OOZIE-1021 > >>> > >>> > >>> Regards, > >>> Mohammad > >>> > >>> > >>> ________________________________ > >>> From: Jarek Jarcec Cecho <[email protected]> > >>> To: [email protected] > >>> Sent: Sunday, October 14, 2012 3:16 PM > >>> Subject: Re: Oozie tests seems to be requiring umask 022 > >>> > >>> We probably miss understand each other sir. I've experienced described > >> issues during building Oozie on my local box before any deployment for real > >> life tests. I've justed checkout repository and tried to build it (mvn > >> package) - and almost all tests failed as I described in my original email. > >> Therefore I believe that there is no other special log other than test > >> related logs that I've read and provided appropriate lines, or at least > >> those lines that I believe are related to the issue. I might send entire > >> logs if you want me to. > >>> > >>> Jarcec > >>> > >>> On Sun, Oct 14, 2012 at 03:06:16PM -0700, Mohammad Islam wrote: > >>>> You have to see the launcher log. > >>>> 1. First click on the your ooze job. > >>>> 2. Select the corresponding shell action. it will pop-up a new window. > >>>> 3. Select the right of "Console URL" which will take you to the hadoop > >> JT site. > >>>> 4. Select hadoop map task and click through to find the log. > >>>> > >>>> Regards, > >>>> Mohammad > >>>> > >>>> > >>>> ________________________________ > >>>> From: Jarek Jarcec Cecho <[email protected]> > >>>> To: [email protected] > >>>> Sent: Sunday, October 14, 2012 2:52 PM > >>>> Subject: Re: Oozie tests seems to be requiring umask 022 > >>>> > >>>> Hi Mohamad, > >>>> I've actually did not do anything special. I've just checked out svn > >> repository and executed "mvn test". Based on the maven output I'm expecting > >> that Oozie is compiling against 1.0.1 and 2.0.0-alpha, I'm not sure which > >> version is used in tests thought. > >>>> > >>>> Appropriate maven output that used: > >>>> [INFO] Apache Oozie Hadoop 1.0.1.oozie-3.4.0-SNAPSHOT .... SUCCESS > >> [0.236s] > >>>> [INFO] Apache Oozie Hadoop Distcp 1.0.1.oozie-3.4.0-SNAPSHOT SUCCESS > >> [0.151s] > >>>> [INFO] Apache Oozie Hadoop 1.0.1.oozie-3.4.0-SNAPSHOT Test SUCCESS > >> [0.360s] > >>>> [INFO] Apache Oozie Hadoop 2.0.0-alpha.oozie-3.4.0-SNAPSHOT SUCCESS > >> [0.427s] > >>>> [INFO] Apache Oozie Hadoop 2.0.0-alpha.oozie-3.4.0-SNAPSHOT Test > >> SUCCESS [0.411s] > >>>> [INFO] Apache Oozie Hadoop Distcp 2.0.0-alpha.oozie-3.4.0-SNAPSHOT > >> SUCCESS [0.137s] > >>>> > >>>> Jarcec > >>>> > >>>> On Fri, Oct 12, 2012 at 09:55:09PM -0700, Mohammad Islam wrote: > >>>>> Hi Jarek, > >>>>> Your test case is running against which hadoop version. > >>>>> Regards, > >>>>> Mohammad > >>>>> > >>>>> > >>>>> ________________________________ > >>>>> From: Jarek Jarcec Cecho <[email protected]> > >>>>> To: [email protected] > >>>>> Sent: Friday, October 12, 2012 11:27 AM > >>>>> Subject: Oozie tests seems to be requiring umask 022 > >>>>> > >>>>> Hi Oozie developers, > >>>>> I've recently worked on OOZIE-1012 and I've noticed that on my box > >> oozie tests seems to be requiring umask 022 (linux, Ubuntu 12.04), which > >> actually seems to be implied by HDFS not by Oozie. > >>>>> > >>>>> If I run tests with umask 002 (my default), most of the tests will > >> fail with following message: > >>>>> > >>>>> Cannot lock storage build/test/data/dfs/name1. The directory is > >> already locked. > >>>>> > >>>>> I believe that this error message is just a consequence of following > >> NullPointerException that appears in setUp method of affected test cases: > >>>>> > >>>>> java.lang.NullPointerException > >>>>> at > >> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422) > >>>>> at > >> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) > >>>>> at > >> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124) > >>>>> at > >> org.apache.oozie.test.XTestCase.setUpEmbeddedHadoop(XTestCase.java:708) > >>>>> at org.apache.oozie.test.XTestCase.setUp(XTestCase.java:281) > >>>>> at org.apache.oozie.test.XFsTestCase.setUp(XFsTestCase.java:58) > >>>>> at > >> org.apache.oozie.action.hadoop.ActionExecutorTestCase.setUp(ActionExecutorTestCase.java:62) > >>>>> > >>>>> I believe that reason of this NullPointerException is absence of > >> usable data directories as *-output logs contains following fragments: > >>>>> > >>>>> 12/10/12 11:16:12 WARN datanode.DataNode: Invalid directory in > >> dfs.data.dir: Incorrect permission for build/test/data/dfs/data/data1, > >> expected: rwxr-xr-x, while actual: rwxrwxr-x > >>>>> 12/10/12 11:16:12 WARN datanode.DataNode: Invalid directory in > >> dfs.data.dir: Incorrect permission for build/test/data/dfs/data/data2, > >> expected: rwxr-xr-x, while actual: rwxrwxr-x > >>>>> 12/10/12 11:16:12 ERROR datanode.DataNode: All directories in > >> dfs.data.dir are invalid. > >>>>> > >>>>> > >>>>> Please note that this output was generated with umask 002 and > >> changing it to 022 will fix the issue. > >>>>> > >>>>> Does anyone else noticed this behaviour (failures) as well? If so > >> then it might make sense to document this need on HowToContribute page [1] > >> and maybe improve the XTestCase.setUpEmbeddedHadoop by catching NPE and > >> informing developer that we're not able to bootstrap MiniDFSCluster? > >>>>> > >>>>> Jarcec > >>>>> > >>>>> Links: > >>>>> 1: http://incubator.apache.org/oozie/HowToContribute.html > >>
signature.asc
Description: Digital signature
