shouldnt the fix be done in hdfs? Alejandro
On Oct 15, 2012, at 9:37 AM, Robert Kanter <[email protected]> wrote: > Catching the NPE exception and printing a more useful message is a good > idea, but if there's other possible things that could cause an NPE, the > "useful" message would be misleading. Perhaps a better solution might be > to explicitly check the umask somehow and print/log a warning or something. > Though from a quick Google search, it doesn't look like there's an easy > way to check (or change) the umask from Java. > > - Robert > > > On Mon, Oct 15, 2012 at 7:51 AM, Jarek Jarcec Cecho <[email protected]>wrote: > >> Thank you sir, >> I appreciate your help! Do you think that it would make sense to also add >> try-catch block in the setUp method for NullPointerException and print out >> some warning message that hadoop was not correctly initialized and thus >> there will be failures in the tests? I believe that it might people that >> missed the umask requirement in docs to get it quickly resolved. >> >> Jarcec >> >> On Mon, Oct 15, 2012 at 01:05:44AM -0700, Mohammad Islam wrote: >>> My previous reply was for another email. Sorry for the confusion. >>> >>> I found my umask also defaulted to 0022. That's why we never experienced >> the issue. >>> I agree we should update that Oozie doc. >>> Created a JIRA: >>> https://issues.apache.org/jira/browse/OOZIE-1021 >>> >>> >>> Regards, >>> Mohammad >>> >>> >>> ________________________________ >>> From: Jarek Jarcec Cecho <[email protected]> >>> To: [email protected] >>> Sent: Sunday, October 14, 2012 3:16 PM >>> Subject: Re: Oozie tests seems to be requiring umask 022 >>> >>> We probably miss understand each other sir. I've experienced described >> issues during building Oozie on my local box before any deployment for real >> life tests. I've justed checkout repository and tried to build it (mvn >> package) - and almost all tests failed as I described in my original email. >> Therefore I believe that there is no other special log other than test >> related logs that I've read and provided appropriate lines, or at least >> those lines that I believe are related to the issue. I might send entire >> logs if you want me to. >>> >>> Jarcec >>> >>> On Sun, Oct 14, 2012 at 03:06:16PM -0700, Mohammad Islam wrote: >>>> You have to see the launcher log. >>>> 1. First click on the your ooze job. >>>> 2. Select the corresponding shell action. it will pop-up a new window. >>>> 3. Select the right of "Console URL" which will take you to the hadoop >> JT site. >>>> 4. Select hadoop map task and click through to find the log. >>>> >>>> Regards, >>>> Mohammad >>>> >>>> >>>> ________________________________ >>>> From: Jarek Jarcec Cecho <[email protected]> >>>> To: [email protected] >>>> Sent: Sunday, October 14, 2012 2:52 PM >>>> Subject: Re: Oozie tests seems to be requiring umask 022 >>>> >>>> Hi Mohamad, >>>> I've actually did not do anything special. I've just checked out svn >> repository and executed "mvn test". Based on the maven output I'm expecting >> that Oozie is compiling against 1.0.1 and 2.0.0-alpha, I'm not sure which >> version is used in tests thought. >>>> >>>> Appropriate maven output that used: >>>> [INFO] Apache Oozie Hadoop 1.0.1.oozie-3.4.0-SNAPSHOT .... SUCCESS >> [0.236s] >>>> [INFO] Apache Oozie Hadoop Distcp 1.0.1.oozie-3.4.0-SNAPSHOT SUCCESS >> [0.151s] >>>> [INFO] Apache Oozie Hadoop 1.0.1.oozie-3.4.0-SNAPSHOT Test SUCCESS >> [0.360s] >>>> [INFO] Apache Oozie Hadoop 2.0.0-alpha.oozie-3.4.0-SNAPSHOT SUCCESS >> [0.427s] >>>> [INFO] Apache Oozie Hadoop 2.0.0-alpha.oozie-3.4.0-SNAPSHOT Test >> SUCCESS [0.411s] >>>> [INFO] Apache Oozie Hadoop Distcp 2.0.0-alpha.oozie-3.4.0-SNAPSHOT >> SUCCESS [0.137s] >>>> >>>> Jarcec >>>> >>>> On Fri, Oct 12, 2012 at 09:55:09PM -0700, Mohammad Islam wrote: >>>>> Hi Jarek, >>>>> Your test case is running against which hadoop version. >>>>> Regards, >>>>> Mohammad >>>>> >>>>> >>>>> ________________________________ >>>>> From: Jarek Jarcec Cecho <[email protected]> >>>>> To: [email protected] >>>>> Sent: Friday, October 12, 2012 11:27 AM >>>>> Subject: Oozie tests seems to be requiring umask 022 >>>>> >>>>> Hi Oozie developers, >>>>> I've recently worked on OOZIE-1012 and I've noticed that on my box >> oozie tests seems to be requiring umask 022 (linux, Ubuntu 12.04), which >> actually seems to be implied by HDFS not by Oozie. >>>>> >>>>> If I run tests with umask 002 (my default), most of the tests will >> fail with following message: >>>>> >>>>> Cannot lock storage build/test/data/dfs/name1. The directory is >> already locked. >>>>> >>>>> I believe that this error message is just a consequence of following >> NullPointerException that appears in setUp method of affected test cases: >>>>> >>>>> java.lang.NullPointerException >>>>> at >> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:422) >>>>> at >> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:280) >>>>> at >> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124) >>>>> at >> org.apache.oozie.test.XTestCase.setUpEmbeddedHadoop(XTestCase.java:708) >>>>> at org.apache.oozie.test.XTestCase.setUp(XTestCase.java:281) >>>>> at org.apache.oozie.test.XFsTestCase.setUp(XFsTestCase.java:58) >>>>> at >> org.apache.oozie.action.hadoop.ActionExecutorTestCase.setUp(ActionExecutorTestCase.java:62) >>>>> >>>>> I believe that reason of this NullPointerException is absence of >> usable data directories as *-output logs contains following fragments: >>>>> >>>>> 12/10/12 11:16:12 WARN datanode.DataNode: Invalid directory in >> dfs.data.dir: Incorrect permission for build/test/data/dfs/data/data1, >> expected: rwxr-xr-x, while actual: rwxrwxr-x >>>>> 12/10/12 11:16:12 WARN datanode.DataNode: Invalid directory in >> dfs.data.dir: Incorrect permission for build/test/data/dfs/data/data2, >> expected: rwxr-xr-x, while actual: rwxrwxr-x >>>>> 12/10/12 11:16:12 ERROR datanode.DataNode: All directories in >> dfs.data.dir are invalid. >>>>> >>>>> >>>>> Please note that this output was generated with umask 002 and >> changing it to 022 will fix the issue. >>>>> >>>>> Does anyone else noticed this behaviour (failures) as well? If so >> then it might make sense to document this need on HowToContribute page [1] >> and maybe improve the XTestCase.setUpEmbeddedHadoop by catching NPE and >> informing developer that we're not able to bootstrap MiniDFSCluster? >>>>> >>>>> Jarcec >>>>> >>>>> Links: >>>>> 1: http://incubator.apache.org/oozie/HowToContribute.html >>
