[
https://issues.apache.org/jira/browse/HBASE-19371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16270428#comment-16270428
]
Duo Zhang commented on HBASE-19371:
-----------------------------------
OK, it is exactly what I concerned in HBASE-18601... We haven't call
TraceUtil.initTracer in WALPE so the sync method in AsyncFSWAL just bypass the
truck loading and cause an empty truck being published.
You can add this line in setConf to temporary fix the problem sir [~stack].
{code}
@Override
public void setConf(Configuration conf) {
super.setConf(conf);
TEST_UTIL = new HBaseTestingUtility(conf);
TraceUtil.initTracer(conf); // <---------- add this
}
{code}
I will file a new issue to address the problem. Seems the span in SyncFuture is
not used any more.
Thanks.
> Running WALPerformanceEvaluation against asyncfswal throws exceptions
> ---------------------------------------------------------------------
>
> Key: HBASE-19371
> URL: https://issues.apache.org/jira/browse/HBASE-19371
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 2.0.0-beta-1
>
>
> Was trying to do a perf eval on asyncfswal. I ran w/ these args:
> Performance counter stats for '/home/stack/hbase/bin/hbase --config
> /home/stack/conf_hbase org.apache.hadoop.hbase.wal.WALPerformanceEvaluation
> -path /user/stack/logs/ -verify -threads 25 -iterations 1000000 -keySize 50
> -valueSize 100 -syncInterval 10':
> The verify fails on all runs:
> Exception in thread "main" java.lang.IllegalStateException: Counted=12390228,
> expected=25000000
> at
> org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.run(WALPerformanceEvaluation.java:368)
>
> at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.innerMain(WALPerformanceEvaluation.java:597)
>
> at
> org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.main(WALPerformanceEvaluation.java:601)
> I need to fix test or figure what is wrong in asyncfswal.
> Also seeing these when I run w/ one thread only:
> 2017-11-28 21:25:49,952 WARN [AsyncFSWAL-1-1] wal.AsyncFSWAL:
> RingBufferTruck with unexpected type: EMPTY
> Log has a spew of them.
> Has stuff like this too:
> 2017-11-28 21:25:40,065 WARN [Close-WAL-Writer-3]
> wal.AsyncProtobufLogWriter: normal close failed, try recover
>
> java.io.IOException: stream already broken
>
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.endBlock(FanOutOneBlockAsyncDFSOutput.java:510)
>
> at
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutput.lambda$close$12(FanOutOneBlockAsyncDFSOutput.java:550)
>
> at
> org.apache.hadoop.hbase.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
> org.apache.hadoop.hbase.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
> at
> org.apache.hadoop.hbase.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
> at
> org.apache.hadoop.hbase.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
>
> at
> org.apache.hadoop.hbase.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
> at java.lang.Thread.run(Thread.java:748)
> Starts out spewing EMPTY here:
> 2017-11-28 21:16:52,051 INFO [main] regionserver.HRegion: Setting
> FlushNonSloppyStoresFirstPolicy for the
> region=WALPerformanceEvaluation:0,,1511932610787.deca03e0ca447fa25d02fe9cd6e31aa4.
> 2017-11-28 21:16:52,058 DEBUG [main] regionserver.HRegion: Found 0 recovered
> edits file(s) under
> hdfs://ve0524.halxg.cloudera.com:8020/user/stack/logs/data/WALPerformanceEvaluation/0/deca03e0ca447fa25d02fe9cd6e31aa4
> 2017-11-28 21:16:52,068 DEBUG [main] regionserver.FlushLargeStoresPolicy: No
> hbase.hregion.percolumnfamilyflush.size.lower.bound set in description of
> table WALPerformanceEvaluation:0, use config (134217728) instead
> 2017-11-28 21:16:52,084 DEBUG [main] wal.WALSplitter: Wrote
> file=hdfs://ve0524.halxg.cloudera.com:8020/user/stack/logs/data/WALPerformanceEvaluation/0/deca03e0ca447fa25d02fe9cd6e31aa4/recovered.edits/2.seqid,
> newSeqId=2, maxSeqId=0
> 2017-11-28 21:16:52,084 INFO [main] regionserver.HRegion: Onlined
> deca03e0ca447fa25d02fe9cd6e31aa4; next sequenceid=2
> 2017-11-28 21:16:52,185 WARN [AsyncFSWAL-1-1] wal.AsyncFSWAL:
> RingBufferTruck with unexpected type: EMPTY
> 2017-11-28 21:16:52,185 WARN [AsyncFSWAL-1-1] wal.AsyncFSWAL:
> RingBufferTruck with unexpected type: EMPTY
> 2017-11-28 21:16:52,185 WARN [AsyncFSWAL-1-1] wal.AsyncFSWAL:
> RingBufferTruck with unexpected type: EMPTY
> 2017-11-28 21:16:52,185 WARN [AsyncFSWAL-1-1] wal.AsyncFSWAL:
> RingBufferTruck with unexpected type: EMPTY
> 2017-11-28 21:16:52,185 WARN [AsyncFSWAL-1-1] wal.AsyncFSWAL:
> RingBufferTruck with unexpected type: EMPTY
> 2017-11-28 21:16:52,185 WARN [AsyncFSWAL-1-1] wal.AsyncFSWAL:
> RingBufferTruck with unexpected type: EMPTY
> 2017-11-28 21:16:52,185 WARN [AsyncFSWAL-1-1] wal.AsyncFSWAL:
> RingBufferTruck with unexpected type: EMPTY
> ...
> This is tip of branch-2 minus the commit of the change to the ringbuffer (I'd
> not pulled in that change).
> [~Apache9] in case you've seen it before else I'll dig in sir.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
