[
https://issues.apache.org/jira/browse/HBASE-10751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-10751:
--------------------------
Attachment: 10751.txt
So, the stack trace above is a bit of a red herring. It is happening because
we are interrupting the tests's background thread on our way out. It is
causing a DroppedSnapshotException to be thrown that we are ignoring (because
it is happening when we are 'done'). Because we are not 'exiting' on this DSE,
the memory accounting is all off so we are in strange state -- unable to
successfully flush yet memory accountings says there is stuff to flush (Because
we did not react to the original DSE).
Let me apply this small patch so we just ignore the second DSE that happens on
the way out (The reason this test failed).
> TestHRegion testWritesWhileScanning occasional fail since HBASE-10514 went in
> -----------------------------------------------------------------------------
>
> Key: HBASE-10751
> URL: https://issues.apache.org/jira/browse/HBASE-10751
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Attachments: 10751.txt
>
>
> I saw this here
> https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/213/testReport/junit/org.apache.hadoop.hbase.regionserver/TestHRegion/testWritesWhileScanning/
> This patch looks to have exposed a problem in our HStore commit logic. We
> are supposed to crash out if we fail to write but we keep going here. I am
> having trouble figuring why. Let me write a little test:
> {code}
> 2014-03-14 01:58:48,647 DEBUG [Thread-3] regionserver.HRegionFileSystem(339):
> Committing store file
> /home/jenkins/jenkins-slave/workspace/HBase-0.98-on-Hadoop-1.1/0.98-hadoop1.1/hbase-server/target/test-data/f7999012-e166-4619-ab3c-5014e0f65007/data/default/testWritesWhileScanning/306ea000673d780f06daf2469e7f9bab/.tmp/a0e6579af25f463ebb7eebe3c043b8a0
> as
> /home/jenkins/jenkins-slave/workspace/HBase-0.98-on-Hadoop-1.1/0.98-hadoop1.1/hbase-server/target/test-data/f7999012-e166-4619-ab3c-5014e0f65007/data/default/testWritesWhileScanning/306ea000673d780f06daf2469e7f9bab/family7/a0e6579af25f463ebb7eebe3c043b8a0
> 2014-03-14 01:58:48,647 INFO [Thread-2] regionserver.HRegion(5779): writing
> data to region
> testWritesWhileScanning,,1394762315120.306ea000673d780f06daf2469e7f9bab. with
> WAL disabled. Data may be lost in the event of a crash.
> 2014-03-14 01:58:48,648 ERROR [Thread-3]
> regionserver.HStore$StoreFlusherImpl(1964): Failed to commit store file
> /home/jenkins/jenkins-slave/workspace/HBase-0.98-on-Hadoop-1.1/0.98-hadoop1.1/hbase-server/target/test-data/f7999012-e166-4619-ab3c-5014e0f65007/data/default/testWritesWhileScanning/306ea000673d780f06daf2469e7f9bab/.tmp/a0e6579af25f463ebb7eebe3c043b8a0
> org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
> Trailer from file
> file:/home/jenkins/jenkins-slave/workspace/HBase-0.98-on-Hadoop-1.1/0.98-hadoop1.1/hbase-server/target/test-data/f7999012-e166-4619-ab3c-5014e0f65007/data/default/testWritesWhileScanning/306ea000673d780f06daf2469e7f9bab/family7/a0e6579af25f463ebb7eebe3c043b8a0
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:552)
> at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:580)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1019)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:211)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:350)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:445)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:551)
> at
> org.apache.hadoop.hbase.regionserver.HStore.commitFile(HStore.java:842)
> at
> org.apache.hadoop.hbase.regionserver.HStore.access$200(HStore.java:118)
> at
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:1961)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1706)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1583)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1498)
> at
> org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3034)
> Caused by: java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
> at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:282)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.seek(RawLocalFileSystem.java:111)
> at
> org.apache.hadoop.fs.BufferedFSInputStream.seek(BufferedFSInputStream.java:78)
> at
> org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:37)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:206)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237)
> at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:384)
> at org.apache.hadoop.fs.FSInputChecker.seek(FSInputChecker.java:365)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.seek(ChecksumFileSystem.java:271)
> at
> org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:37)
> at
> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:389)
> at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:537)
> ... 13 more
> 2014-03-14 01:58:48,657 DEBUG [pool-1-thread-1] regionserver.HRegion(1037):
> Closing
> testWritesWhileScanning,,1394762315120.306ea000673d780f06daf2469e7f9bab.:
> disabling compactions & flushes
> 2014-03-14 01:58:48,657 INFO [pool-1-thread-1] regionserver.HRegion(1045):
> Running close preflush of
> testWritesWhileScanning,,1394762315120.306ea000673d780f06daf2469e7f9bab.
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
