I seem to find a way to get mapreduce job to run (with fix for YARN-429<https://issues.apache.org/jira/browse/YARN-429> in place).
See my attachment in YARN-449<https://issues.apache.org/jira/browse/YARN-449> FYI On Mon, Mar 4, 2013 at 3:56 PM, Ted Yu <[email protected]> wrote: > Here is another blocker for running mapreduce on top hadoop 2.0.3-alpha: > > https://issues.apache.org/jira/browse/YARN-449 > > > On Tue, Feb 26, 2013 at 12:03 PM, Ted Yu <[email protected]> wrote: > >> https://issues.apache.org/jira/browse/YARN-429 was logged for this issue. >> >> FYI >> >> >> On Fri, Feb 22, 2013 at 1:12 PM, Andrew Purtell <[email protected]>wrote: >> >>> I agree, thanks a lot for doing this. >>> >>> >>> On Fri, Feb 22, 2013 at 1:07 PM, Ted Yu <[email protected]> wrote: >>> >>> > bq. to nail down the actual issue will help making our case. >>> > >>> > That is what I am trying to do. Will ping Arun at the meetup this >>> afternoon >>> > about when the next release is coming. >>> > >>> > Cheers >>> > >>> > On Fri, Feb 22, 2013 at 1:02 PM, Stack <[email protected]> wrote: >>> > >>> > > Thanks for trying to fix it Ted. I think we need to carry our >>> difficulty >>> > > back up to our parent project brethren (Sounds like you have done >>> this >>> > > off-list; lets do it in jira/mail too). Your digging in to nail >>> down the >>> > > actual issue will help making our case. Maybe we can get/help the >>> > > upstream crew make another release (I don't see a note on a new 2.0 >>> > release >>> > > up on hadoop -- have I missed it)? >>> > > >>> > > Thanks, >>> > > St.Ack >>> > > >>> > > >>> > > On Fri, Feb 22, 2013 at 11:49 AM, Ted Yu <[email protected]> >>> wrote: >>> > > >>> > > > Trying out #2, I got: >>> > > > >>> > > > >>> > > > >>> > > >>> > >>> testExcludeAllFromMinorCompaction(org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat) >>> > > > Time elapsed: 35.746 sec <<< FAILURE! >>> > > > java.lang.AssertionError >>> > > > at org.junit.Assert.fail(Assert.java:86) >>> > > > at org.junit.Assert.assertTrue(Assert.java:41) >>> > > > at org.junit.Assert.assertTrue(Assert.java:52) >>> > > > at >>> > > > >>> > > > >>> > > >>> > >>> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.runIncrementalPELoad(TestHFileOutputFormat.java:468) >>> > > > at >>> > > > >>> > > > >>> > > >>> > >>> org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.testExcludeAllFromMinorCompaction(TestHFileOutputFormat.java:713) >>> > > > >>> > > > Arun Murthy got to know this defect. Another 2.0 release is coming >>> soon >>> > > > (next week, hopefully). >>> > > > >>> > > > I plan to wait for the new release. >>> > > > >>> > > > Thanks >>> > > > >>> > > > On Fri, Feb 22, 2013 at 11:39 AM, Ted Yu <[email protected]> >>> wrote: >>> > > > >>> > > > > The root cause was the removal of capacity-scheduler.xml< >>> > > > >>> > > >>> > >>> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf/capacity-scheduler.xml >>> > > > >>> > > > from >>> > > > > 2.0.3-alpha maven artifacts. >>> > > > > This config file is needed to start CapacityScheduler which is >>> the >>> > > > default. >>> > > > > >>> > > > > After talking to Sid, I got two suggestions: >>> > > > > 1. embed minimal capacity-scheduler.xml< >>> > > > >>> > > >>> > >>> http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf/capacity-scheduler.xml >>> > > > >>> > > > in >>> > > > > our src/test/resources >>> > > > > 2. use FiFo scheduler instead which doesn't require presence of >>> > config >>> > > > file >>> > > > > >>> > > > > I think #2, though temporary fix, is easy to change once a >>> future 2.0 >>> > > > > release addresses this issue. >>> > > > > >>> > > > > Once consensus is reached, I will attach another patch to >>> HBASE-7904 >>> > > > > >>> > > > > Cheers >>> > > > > >>> > > > > >>> > > > > On Fri, Feb 22, 2013 at 10:31 AM, Andrew Purtell < >>> > [email protected] >>> > > > >wrote: >>> > > > > >>> > > > >> For minicluster unit tests? In my opinion, no. >>> > > > >> >>> > > > >> >>> > > > >> On Fri, Feb 22, 2013 at 10:10 AM, Ted Yu <[email protected]> >>> > wrote: >>> > > > >> >>> > > > >> > I can try to find the proper usage. >>> > > > >> > >>> > > > >> > The question is: do we have to maintain compatibility with >>> > > > 2.0.2-alpha ? >>> > > > >> > >>> > > > >> > Thanks >>> > > > >> > >>> > > > >> > On Fri, Feb 22, 2013 at 10:07 AM, Andrew Purtell < >>> > > [email protected] >>> > > > >> > >wrote: >>> > > > >> > >>> > > > >> > > Sigh. Every release seems to break our uses of >>> MiniMRCluster all >>> > > > over >>> > > > >> > > again. >>> > > > >> > > >>> > > > >> > > >>> > > > >> > > On Fri, Feb 22, 2013 at 10:04 AM, Ted Yu < >>> [email protected]> >>> > > > wrote: >>> > > > >> > > >>> > > > >> > > > I temporarily reverted this change due to: >>> > > > >> > > > >>> > > > >> > > > org.apache.hadoop.yarn.YarnException: Failed to Start >>> > > > >> > > > org.apache.hadoop.mapred.MiniMRCluster >>> > > > >> > > > at >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > >>> > > > >> > >>> > > > >> >>> > > > >>> > > >>> > >>> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78) >>> > > > >> > > > at >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > >>> > > > >> > >>> > > > >> >>> > > > >>> > > >>> > >>> org.apache.hadoop.mapred.MiniMRClientClusterFactory.create(MiniMRClientClusterFactory.java:75) >>> > > > >> > > > at >>> > > > >> > > > >>> > > > >> >>> > org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:180) >>> > > > >> > > > at >>> > > > >> > > > >>> > > > >> >>> > org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:170) >>> > > > >> > > > at >>> > > > >> > > > >>> > > > >> >>> > org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:162) >>> > > > >> > > > at >>> > > > >> > > > >>> > > > >> >>> > org.apache.hadoop.mapred.MiniMRCluster.<init>(MiniMRCluster.java:123) >>> > > > >> > > > at >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > >>> > > > >> > >>> > > > >> >>> > > > >>> > > >>> > >>> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniMapReduceCluster(HBaseTestingUtility.java:1574) >>> > > > >> > > > ... >>> > > > >> > > > Caused by: org.apache.hadoop.yarn.YarnException: >>> > > > >> > > > java.lang.IllegalStateException: Queue configuration >>> missing >>> > > child >>> > > > >> > queue >>> > > > >> > > > names for root >>> > > > >> > > > at >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > >>> > > > >> > >>> > > > >> >>> > > > >>> > > >>> > >>> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:181) >>> > > > >> > > > at >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > >>> > > > >> > >>> > > > >> >>> > > > >>> > > >>> > >>> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) >>> > > > >> > > > ... 38 more >>> > > > >> > > > Caused by: java.lang.IllegalStateException: Queue >>> > configuration >>> > > > >> missing >>> > > > >> > > > child queue names for root >>> > > > >> > > > at >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > >>> > > > >> > >>> > > > >> >>> > > > >>> > > >>> > >>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:328) >>> > > > >> > > > at >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > >>> > > > >> > >>> > > > >> >>> > > > >>> > > >>> > >>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:255) >>> > > > >> > > > at >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > >>> > > > >> > >>> > > > >> >>> > > > >>> > > >>> > >>> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:220) >>> > > > >> > > > at >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > >>> > > > >> > >>> > > > >> >>> > > > >>> > > >>> > >>> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.init(ResourceManager.java:226) >>> > > > >> > > > at >>> > > > >> > > > >>> > > > >> > > > >>> > > > >> > > >>> > > > >> > >>> > > > >> >>> > > > >>> > > >>> > >>> org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.start(MiniYARNCluster.java:161) >>> > > > >> > > > >>> > > > >> > > > Cheers >>> > > > >> > > > >>> > > > >> > > > On Fri, Feb 22, 2013 at 9:55 AM, Andrew Purtell < >>> > > > >> [email protected]> >>> > > > >> > > > wrote: >>> > > > >> > > > >>> > > > >> > > > > Have you tried running the mapreduce unit tests with >>> this >>> > > > version >>> > > > >> > Ted? >>> > > > >> > > > Just >>> > > > >> > > > > curious. They have given us grief in the past. >>> > > > >> > > > > >>> > > > >> > > > > >>> > > > >> > > > > On Fri, Feb 22, 2013 at 9:35 AM, Ted Yu < >>> > [email protected]> >>> > > > >> wrote: >>> > > > >> > > > > >>> > > > >> > > > > > Hi, >>> > > > >> > > > > > I have integrated the following JIRA to trunk: >>> > > > >> > > > > > HBASE-7904 Upgrade hadoop 2.0 dependency to >>> 2.0.3-alpha >>> > > > >> > > > > > >>> > > > >> > > > > > Earlier this week, Ramkrishna encountered some problem >>> > where >>> > > > >> HBase >>> > > > >> > > was >>> > > > >> > > > > > built against 2.0.2-alpha but hadoop cluster >>> deployment >>> > was >>> > > > >> using >>> > > > >> > > > > > 2.0.3-alpha >>> > > > >> > > > > > >>> > > > >> > > > > > In case you use HBase against hadoop 2.0, please note >>> this >>> > > > >> change. >>> > > > >> > > > > > >>> > > > >> > > > > > Cheers >>> > > > >> > > > > > >>> > > > >> > > > > >>> > > > >> > > > > >>> > > > >> > > > > >>> > > > >> > > > > -- >>> > > > >> > > > > Best regards, >>> > > > >> > > > > >>> > > > >> > > > > - Andy >>> > > > >> > > > > >>> > > > >> > > > > Problems worthy of attack prove their worth by hitting >>> > back. - >>> > > > >> Piet >>> > > > >> > > Hein >>> > > > >> > > > > (via Tom White) >>> > > > >> > > > > >>> > > > >> > > > >>> > > > >> > > >>> > > > >> > > >>> > > > >> > > >>> > > > >> > > -- >>> > > > >> > > Best regards, >>> > > > >> > > >>> > > > >> > > - Andy >>> > > > >> > > >>> > > > >> > > Problems worthy of attack prove their worth by hitting >>> back. - >>> > > Piet >>> > > > >> Hein >>> > > > >> > > (via Tom White) >>> > > > >> > > >>> > > > >> > >>> > > > >> >>> > > > >> >>> > > > >> >>> > > > >> -- >>> > > > >> Best regards, >>> > > > >> >>> > > > >> - Andy >>> > > > >> >>> > > > >> Problems worthy of attack prove their worth by hitting back. - >>> Piet >>> > > Hein >>> > > > >> (via Tom White) >>> > > > >> >>> > > > > >>> > > > > >>> > > > >>> > > >>> > >>> >>> >>> >>> -- >>> Best regards, >>> >>> - Andy >>> >>> Problems worthy of attack prove their worth by hitting back. - Piet Hein >>> (via Tom White) >>> >> >> >
