What happened in region server : 2012-04-19 19:32:31,752 INFO org.apache.hadoop.hbase.regionserver.SplitRequest: Successful rollback of failed split of TestUserBTProfile,141018963342131767,1334870127822.5fe2189f32a52f3f2bce55a92ae5df9d. 2012-04-19 19:50:10,392 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Flushing .META.,,1.1028785192 2012-04-19 19:50:10,529 INFO org.apache.hadoop.hbase.regionserver.Store: Added hdfs:// master-hadoop.example.com:8020/hbase/.META./1028785192/info/270b515bf7d34ab88552f0e244419a0d, entries=823, sequenceid=1825092, memsize=243.3k, filesize=102.2k 2012-04-19 19:50:10,535 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~483.5k/495080, currentsize=0.0/0 for region .META.,,1.1028785192 in 88ms, sequenceid=1825092, compaction requested=false 2012-04-19 19:50:10,614 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on info in region .META.,,1.1028785192 2012-04-19 19:50:10,615 INFO org.apache.hadoop.hbase.regionserver.Store: Starting compaction of 2 file(s) in info of .META.,,1.1028785192 into tmpdir=hdfs://master-hadoop.example.com:8020/hbase/.META./1028785192/.tmp, seqid=1825092, totalSize=541.0k 2012-04-19 19:50:10,813 INFO org.apache.hadoop.hbase.regionserver.Store: Renaming compacted file at hdfs:// master-hadoop.example.com:8020/hbase/.META./1028785192/.tmp/5f7bea0f8f8142109c0ef41ada5a509eto hdfs:// master-hadoop.example.com:8020/hbase/.META./1028785192/info/5f7bea0f8f8142109c0ef41ada5a509e2012-04-1919:50:10,927 INFO org.apache.hadoop.hbase.regionserver.Store: Completed major compaction of 2 file(s) in info of .META.,,1.1028785192 into 5f7bea0f8f8142109c0ef41ada5a509e, size=442.0k; total size for store is 442.0k2012-04-19 19:50:10,927 INFO org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest: completed compaction: regionName=.META.,,1.1028785192, storeName=info, fileCount=2, fileSize=541.0k, priority=1, time=6327679323413783; duration=0sec2012-04-19 20:04:14,620 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush of region userprofile,671599294440030433,1334171984301.075898f82c75fa049e7bfb88b7ca567d. due to global heap pressure
On Fri, Apr 20, 2012 at 11:23 AM, Ted Yu <[email protected]> wrote: > bq. Successful rollback of failed split of > Can you look for 'TestUserBTProfile,141018963342131767' in region / master > logs to see what happened next ? > > Thanks > > On Fri, Apr 20, 2012 at 10:33 AM, Shrijeet Paliwal > <[email protected]>wrote: > > > Here are the version details: > > > > Hadoop : CDH3u0 > > HBase: 0.92.1 (last commit sha : > 355e14491c78b56d9f2e270e9995d666d4ce6767) > > > > On Fri, Apr 20, 2012 at 10:26 AM, Shrijeet Paliwal > > <[email protected]>wrote: > > > > > Hi All, > > > > > > hbck is reporting this error: > > > > > > ERROR: Region UNKNOWN_REGION on ip-4.example.com,60020,1334705737039, > > > key=5fe2189f32a52f3f2bce55a92ae5df9d, not on HDFS or in META but > deployed > > > on ip-4.example.com,60020,1334705737039 > > > > > > Region server log : > > > http://pastie.org/3823850 > > > > > > Namenode audit log: > > > http://pastie.org/3823867 > > > > > > Master log: > > > http://pastie.org/3823871 > > > > > > From the region server log it seems there was a hiccup during a split > > > event as seen here: > > > > > > 2189f32a52f3f2bce55a92ae5df9d.; Took too long to split the files and > > > create the references, aborting split > > > java.io.IOException: Took too long to split the files and create the > > > references, aborting split > > > >-at > > > > > > org.apache.hadoop.hbase.regionserver.SplitTransaction.splitStoreFiles(SplitTransaction.java:613) > > > >-at > > > > > > org.apache.hadoop.hbase.regionserver.SplitTransaction.createDaughters(SplitTransaction.java:285) > > > >-at > > > > > > org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:450) > > > >-at > > > > > > org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:67) > > > >-at > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > >-at > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > >-at java.lang.Thread.run(Thread.java:619) > > > 2012-04-19 19:32:31,690 INFO > > > org.apache.hadoop.hbase.regionserver.StoreFile$Reader: Loaded ROWCOL > > > CompoundBloomFilter metadata for 877edbbabbf74e2e94888ddb29fb1f1b > > > 2012-04-19 19:32:31,730 INFO > > > org.apache.hadoop.hbase.regionserver.SplitTransaction: Cleaned up old > > > failed split transaction detritus: hdfs:// > > > > > > master-hadoop.example.com:8020/hbase/TestUserBTProfile/5fe2189f32a52f3f2bce55a92ae5df9d/splits > > > 2012-04-19 19:32:31,731 INFO > > org.apache.hadoop.hbase.regionserver.HRegion: > > > Onlined > > > > > > TestUserBTProfile,141018963342131767,1334870127822.5fe2189f32a52f3f2bce55a92ae5df9d.; > > > next sequenceid=1823595 > > > 2012-04-19 19:32:31,752 INFO > > > org.apache.hadoop.hbase.regionserver.SplitRequest: Successful rollback > of > > > failed split of > > > > > > TestUserBTProfile,141018963342131767,1334870127822.5fe2189f32a52f3f2bce55a92ae5df9d. > > > > > > -Shrijeet > > > > > > > > >
