On my build machines,
TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS
fails intermittently due to time out fairly frequently on my 0.90, 0.92, and
trunk builds. (it is the second most frequent -- probably once of out every
5-10 runs)).
I haven't investigated beyond this (I just exclude this as a flaky) but
here's some output.
Error Message
test timed out after 180000 milliseconds
Stacktrace
java.lang.Exception: test timed out after 180000 milliseconds
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.MiniHBaseCluster.waitForActiveAndReadyMaster(MiniHBaseCluster.java:385)
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRITOnDeadRS(TestMasterFailover.java:919)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
Standard Error
Here's an exception that recurrs a lot:
2011-09-27 09:53:28,882 FATAL [Thread-562.logSyncer] wal.HLog(1086):
Could not sync. Requesting close of hlog
java.io.IOException: Reflection
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:188)
at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1073)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1174)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1041)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:186)
... 3 more
Caused by: java.io.IOException: DFSOutputStream is closed
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3239)
at
org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
at
org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
... 7 more
2011-09-27 09:53:28,882 ERROR [Thread-562.logSyncer]
wal.HLog$LogSyncer(1043): Error while syncing, requesting close of
hlog
java.io.IOException: Reflection
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:188)
at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1073)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1174)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1041)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:186)
... 3 more
Caused by: java.io.IOException: DFSOutputStream is closed
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3239)
at
org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
at
org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
... 7 more
On Tue, Sep 27, 2011 at 2:59 PM, Stack <[email protected]> wrote:
> TestMasterFailover? Because no output from it?
> St.Ack
>
> On Tue, Sep 27, 2011 at 2:36 PM, Ted Yu <[email protected]> wrote:
> > I see at least one hung test:
> >
> > Running org.apache.hadoop.hbase.master.TestMasterFailover
> > Running org.apache.hadoop.hbase.master.TestCatalogJanitor
> > Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.318 sec
> >
> >
> >
> > On Tue, Sep 27, 2011 at 2:18 PM, Apache Jenkins Server <
> > [email protected]> wrote:
> >
> >> See <https://builds.apache.org/job/HBase-0.92/23/changes>
> >>
> >> Changes:
> >>
> >> [stack] HBASE-4433: avoid extra next (potentially a seek) if done with
> >> column/row
> >>
> >> [stack] HBASE-4433: avoid extra next (potentially a seek) if done with
> >> column/row
> >>
> >> ------------------------------------------
> >>
> >
>
--
// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// [email protected]
