[
https://issues.apache.org/jira/browse/HBASE-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-2063:
-------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Commmitted branch and trunk. Thanks for the nice patch Ruslan Salyakhov.
Thanks for verifying fix Ching-Shen Chen .
> For hfileoutputformat, on timeout/failure/kill clean up half-written hfile
> --------------------------------------------------------------------------
>
> Key: HBASE-2063
> URL: https://issues.apache.org/jira/browse/HBASE-2063
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.3
> Reporter: stack
> Fix For: 0.20.4, 0.21.0
>
> Attachments: HBASE-2063-v1.patch
>
>
> Below is from mailing list. Read from bottom to top:
> {code}
> I was going to write that perhaps you needed to turn
> mapred.reduce.tasks.speculative.execution off, but if enabling it and things
> work, that would seem to indicate that a our reducer first takes longer than
> the task timeout maximum and secondly, on failure, we should clean up the
> hfile.
> On the first issue, you are using KeyValueSortReducer? Are your values
> large? We set reducer status every 100 values. Maybe this is not enough?
> We should set status more frequently? If you call context setstatus more
> frequently, do things work w/o speculative execution?
>
> On the second, HFileOutputFormat close will set the metadata on the hfile and
> then close it. On kill, this code is not being called. Let me see if can
> do something about that (e.g. register a shutdown hook to clean away
> incomplete files -- ).
> Thanks,
> St.Ack
> On Sun, Dec 20, 2009 at 11:26 PM, ChingShen <[email protected]> wrote:
> I think I found a way.
> I set the "mapred.reduce.tasks.speculative.execution" to true and output
> hfiles again, then successfully load hfiles into hbase.
> Is it best solution? or HFileOutputFormat bug?
> Shen
> On Mon, Dec 21, 2009 at 8:25 AM, ChingShen <[email protected]> wrote:
> > Thanks, stack.
> >
> > I checked this file that isn't empty. But I found that as long as the
> > "Killed Task Attempts" > 0 in reduce phase, and run the loadtable.rb script
> > to load hfiles then failed.
> > How to avoid this problem?
> >
> > Thanks.
> >
> > Shen
> >
> >
> > On Sat, Dec 19, 2009 at 3:49 AM, stack <[email protected]> wrote:
> >
> >> Check the
> >> file
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/197894389945760574.
> >> Is it empty? Was there an error during running of your MR job? Perhaps
> >> a
> >> task failed?
> >>
> >> St.Ack
> >>
> >>
> >>
> >> On Thu, Dec 17, 2009 at 9:46 PM, ChingShen <[email protected]>
> >> wrote:
> >>
> >> > Hi,
> >> > I use the script loadtable.rb to load my hfiles into hbase, but I got
> >> an
> >> > exception as below.
> >> > Does anyone have any suggestions?
> >> >
> >> > 09/12/17 23:59:33 INFO loadtable: 18 read firstkey of -3.9290_52.5534
> >> from
> >> >
> >> >
> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/1978943899457605747
> >> > org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize':
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content? (NativeException)
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in `readTrailer'
> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in
> >> `loadFileInfo'
> >> > from sun.reflect.GeneratedMethodAccessor7:-1:in `invoke'
> >> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke'
> >> > from java/lang/reflect/Method.java:597:in `invoke'
> >> > from org/jruby/javasupport/JavaMethod.java:298:in
> >> > `invokeWithExceptionHandling'
> >> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke'
> >> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in `call'
> >> > ... 18 levels...
> >> > from org/jruby/Main.java:94:in `main'
> >> > from loadtable.rb:83:in `each'
> >> > from loadtable.rb:83
> >> > Complete Java stackTrace
> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size
> >> match
> >> > content?
> >> > at
> >> >
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1335)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813)
> >> > at
> >> >
> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:758)
> >> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> >> > at
> >> >
> >> >
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at
> >> >
> >> >
> >> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:298)
> >> > at org.jruby.javasupport.JavaMethod.invoke(JavaMethod.java:259)
> >> > at
> >> >
> >> >
> >> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36)
> >> > at
> >> > org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70)
> >> > at loadtable.ensure_1$RUBY$__ensure___2(loadtable.rb:86)
> >> > at loadtable.block_0$RUBY$__for__(loadtable.rb:85)
> >> > at loadtableBlockCallback$block_0$RUBY$__for__xx1.call(Unknown
> >> Source)
> >> > at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:102)
> >> > at org.jruby.runtime.Block.yield(Block.java:100)
> >> > at
> >> org.jruby.java.proxies.ArrayJavaProxy.each(ArrayJavaProxy.java:112)
> >> > at
> >> >
> >> >
> >> org.jruby.java.proxies.ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.call(org/jruby/java/proxies/ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.gen)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:263)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81)
> >> > at
> >> >
> >> >
> >> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:96)
> >> > at loadtable.__file__(loadtable.rb:83)
> >> > at loadtable.load(loadtable.rb)
> >> > at org.jruby.Ruby.runScript(Ruby.java:577)
> >> > at org.jruby.Ruby.runNormally(Ruby.java:480)
> >> > at org.jruby.Ruby.runFromMain(Ruby.java:354)
> >> > at org.jruby.Main.run(Main.java:229)
> >> > at org.jruby.Main.run(Main.java:110)
> >> > at org.jruby.Main.main(Main.java:94)
> >> >
> >>
> >
> >
> >
> >
> --
> *****************************************************
> Ching-Shen Chen
> Advanced Technology Center,
> Information & Communications Research Lab.
> E-mail: [email protected]
> Tel:+886-3-5915542
> *****************************************************
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
