[
https://issues.apache.org/jira/browse/HBASE-11776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wuchengzhi resolved HBASE-11776.
--------------------------------
Resolution: Duplicate
duplicate issue
> All RegionServers crash when compact when setting TTL
> -----------------------------------------------------
>
> Key: HBASE-11776
> URL: https://issues.apache.org/jira/browse/HBASE-11776
> Project: HBase
> Issue Type: Bug
> Components: Compaction
> Affects Versions: 0.96.1.1
> Environment: ubuntu 12.04
> jdk1.7.0_06
> Reporter: wuchengzhi
> Priority: Critical
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> As We create the table with TTL in columnFamily, When files was selected to
> compact and the files's KVs all expired, after this, it generate a file just
> contains some meta-info such as trail,but without kvs(size:564bytes). (and
> the storeFile.getReader().getMaxTimestamp() = -1)
> And then We put the data to this table so fast, so memStore will flush to
> storefile, and cause the compact task,unexpected thing happens: the
> storefiles's count keeps on increasing all the time.
> seeing the debug log :
> {code:title=hbase-regionServer.log|borderStyle=solid}
> 2014-08-17 15:41:02,689 DEBUG
> [regionserver60020-smallCompactions-1408258247722]
> regionserver.CompactSplitThread: CompactSplitThread Status:
> compaction_queue=(0:1), split_queue=0, merge_queue=0
> 2014-08-17 15:41:02,689 DEBUG
> [regionserver60020-smallCompactions-1408258247722]
> compactions.RatioBasedCompactionPolicy: Selecting compaction from 9 store
> files, 0 compacting, 9 eligible, 10 blocking
> 2014-08-17 15:41:02,689 INFO
> [regionserver60020-smallCompactions-1408258247722]
> compactions.RatioBasedCompactionPolicy: Deleting the expired store file by
> compaction:
> hdfs://hbase:9000/hbase/data/default/top_subchannel_2/0b47596c0bff1a60cf749cf1101eb642/s/c6392d54411a46cbb19350d706a298be
> whose maxTimeStamp is -1 while the max expired timestamp is 1408257662689
> 2014-08-17 15:41:02,689 DEBUG
> [regionserver60020-smallCompactions-1408258247722] regionserver.HStore:
> 0b47596c0bff1a60cf749cf1101eb642 - s: Initiating minor compaction
> 2014-08-17 15:41:02,689 INFO
> [regionserver60020-smallCompactions-1408258247722] regionserver.HRegion:
> Starting compaction on s in region
> top_subchannel_2,,1407982287422.0b47596c0bff1a60cf749cf1101eb642.
> 2014-08-17 15:41:02,689 INFO
> [regionserver60020-smallCompactions-1408258247722] regionserver.HStore:
> Starting compaction of 1 file(s) in s of
> top_subchannel_2,,1407982287422.0b47596c0bff1a60cf749cf1101eb642. into
> tmpdir=hdfs://hbase:9000/hbase/data/default/top_subchannel_2/0b47596c0bff1a60cf749cf1101eb642/.tmp,
> totalSize=564
> 2014-08-17 15:41:02,689 DEBUG
> [regionserver60020-smallCompactions-1408258247722] compactions.Compactor:
> Compacting
> hdfs://hbase:9000/hbase/data/default/top_subchannel_2/0b47596c0bff1a60cf749cf1101eb642/s/c6392d54411a46cbb19350d706a298be,
> keycount=0, bloomtype=NONE, size=564, encoding=FAST_DIFF, seqNum=45561
> 2014-08-17 15:41:02,711 INFO
> [regionserver60020-smallCompactions-1408258247722] regionserver.StoreFile:
> HFile Bloom filter type for f2e60ae4574a4d6eb89745d43582e9b4: NONE, but ROW
> specified in column family configuration
> 2014-08-17 15:41:02,713 DEBUG
> [regionserver60020-smallCompactions-1408258247722]
> regionserver.HRegionFileSystem: Committing store file
> hdfs://hbase:9000/hbase/data/default/top_subchannel_2/0b47596c0bff1a60cf749cf1101eb642/.tmp/f2e60ae4574a4d6eb89745d43582e9b4
> as
> hdfs://hbase:9000/hbase/data/default/top_subchannel_2/0b47596c0bff1a60cf749cf1101eb642/s/f2e60ae4574a4d6eb89745d43582e9b4
> 2014-08-17 15:41:02,726 INFO
> [regionserver60020-smallCompactions-1408258247722] regionserver.StoreFile:
> HFile Bloom filter type for f2e60ae4574a4d6eb89745d43582e9b4: NONE, but ROW
> specified in column family configuration
> 2014-08-17 15:41:02,727 DEBUG
> [regionserver60020-smallCompactions-1408258247722] regionserver.HStore:
> Removing store files after compaction...
> 2014-08-17 15:41:02,731 DEBUG
> [regionserver60020-smallCompactions-1408258247722] backup.HFileArchiver:
> Finished archiving from class
> org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile,
> file:hdfs://hbase:9000/hbase/data/default/top_subchannel_2/0b47596c0bff1a60cf749cf1101eb642/s/c6392d54411a46cbb19350d706a298be,
> to
> hdfs://hbase:9000/hbase/archive/data/default/top_subchannel_2/0b47596c0bff1a60cf749cf1101eb642/s/c6392d54411a46cbb19350d706a298be
> 2014-08-17 15:41:02,731 INFO
> [regionserver60020-smallCompactions-1408258247722] regionserver.HStore:
> Completed compaction of 1 file(s) in s of
> top_subchannel_2,,1407982287422.0b47596c0bff1a60cf749cf1101eb642. into
> f2e60ae4574a4d6eb89745d43582e9b4(size=564), total size for store is 25.8 M.
> This selection was in queue for 0sec, and took 0sec to execute.
> 2014-08-17 15:41:02,731 INFO
> [regionserver60020-smallCompactions-1408258247722]
> regionserver.CompactSplitThread: Completed compaction: Request =
> regionName=top_subchannel_2,,1407982287422.0b47596c0bff1a60cf749cf1101eb642.,
> storeName=s, fileCount=1, fileSize=564, priority=2, time=10316710063714;
> duration=0sec
> {code}
> Because of the "empty storefile" MaxTimestamp is -1,it have priority to
> compact first. it will be deleted,but generate a copy of it. though the
> compact task done,but the count didn't decrease.
> when the count of storefiles more than 10, so we got CompactPriority = -1,
> the CompactThread will require recursive enqueues again and again very soon.
> Terrible things happen: the debug log increasing so fast,the flusherThread
> blocking,and the "empty storefile" generate and remove again and again. and
> we are keeping on puting data to the table, and the client get a lot of
> RegionTooBusyException.After a while, the regionServer crash because the
> MemStoreFlusher is die. so the region was assigning to other regionServer. As
> doing the same things, also, the regionServer crashed,then all the
> regionserver crashed....
> I think this is a bug for ttl&compact,so we need to generate the store file
> without keyvalues after the compcat done?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
