1. Hadoop version is 1.1.2.
2. All servers are synched with NTP.
3. Table definition is: 'compound0', {
NAME => 'd',
DATA_BLOCK_ENCODING => 'NONE',
BLOOMFILTER => 'ROW',
REPLICATION_SCOPE => '0',
VERSIONS => '1',
COMPRESSION => 'SNAPPY',
MIN_VERSIONS => '0',
TTL => '8640000',
KEEP_DELETED_CELLS => 'false',
BLOCKSIZE => '65536',
IN_MEMORY => 'false',
ENCODE_ON_DISK => 'true',
BLOCKCACHE => 'true'
}The TTL is supposed to be 100 days. --Tom On Tue, Sep 24, 2013 at 10:53 AM, Jean-Marc Spaggiari < [email protected]> wrote: > Another important information why might be the root cause of this issue... > > Do you have any TTL defines for this table? > > JM > > > 2013/9/24 Jean-Marc Spaggiari <[email protected]> > > > Strange. > > > > Few questions then. > > 1) What is your hadoop version? > > 2) Is the clock on all your severs synched with NTP? > > 3) What is you table definition? Bloom filters, etc.? > > > > This is the reason why it keep compacting: > > > > 2013-09-24 10:04:00,548 INFO > org.apache.hadoop.hbase.regionserver.compactions.CompactSelection: Deleting > the expired store file by compaction: hdfs:// > hdpmgr001.pse.movenetworks.com:8020/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/7426f128469242ec8ee09f3965fd5a1awhose > maxTimeStamp is -1 while the max expired timestamp is 1371398640548 > > > > maxTimeStamp = -1 > > > > > > Each time there is a comparison between maxTimeStamp for this store file > > and the configured maxExpiredTimeStamp and since maxTimeStamp returns -1, > > it's always elected for a compaction. Now, we need to find why... > > > > JM > > > > > > 2013/9/24 Tom Brown <[email protected]> > > > >> My cluster is fully distributed (2 regionserver nodes). > >> > >> Here is a snippet of log entries that may explain why it started: > >> http://pastebin.com/wQECif8k. I had to go back 2 days to find when it > >> started for this region. > >> > >> This is not the only region experiencing this issue (but this is the > >> smallest one it's happened to). > >> > >> --Tom > >> > >> > >> On Tue, Sep 24, 2013 at 10:13 AM, Jean-Marc Spaggiari < > >> [email protected]> wrote: > >> > >> > Can you past logs a bit before that? To see if anything triggered the > >> > compaction? > >> > Before the 1M compactions entries. > >> > > >> > Also, what is your setup? Are you running in Standalone? Pseudo-Dist? > >> > Fully-Dist? > >> > > >> > Thanks, > >> > > >> > JM > >> > > >> > > >> > 2013/9/24 Tom Brown <[email protected]> > >> > > >> > > There is one column family, d. Each row has about 10 columns, and > each > >> > > row's total data size is less than 2K. > >> > > > >> > > Here is a small snippet of logs from the region server: > >> > > http://pastebin.com/S2jE4ZAx > >> > > > >> > > --Tom > >> > > > >> > > > >> > > On Tue, Sep 24, 2013 at 9:59 AM, Bharath Vissapragada < > >> > > [email protected] > >> > > > wrote: > >> > > > >> > > > It would help if you can show your RS log (via pastebin?) . Are > >> there > >> > > > frequent flushes for this region too? > >> > > > > >> > > > > >> > > > On Tue, Sep 24, 2013 at 9:20 PM, Tom Brown <[email protected]> > >> > wrote: > >> > > > > >> > > > > I have a region that is very small, only 5MB. Despite it's size, > >> it > >> > has > >> > > > 24 > >> > > > > store files. The logs show that it's compacting (over and over > >> > again). > >> > > > > > >> > > > > The odd thing is that even though there are 24 store files, it > >> only > >> > > does > >> > > > > one at a time. Even more strange is that my logs are filling up > >> with > >> > > > > compacting this one region. In the last 10 hours, there have > been > >> > > > 1,876,200 > >> > > > > log entries corresponding to compacting this region alone. > >> > > > > > >> > > > > My cluster is 0.94.10, and using almost all default settings. > >> Only a > >> > > few > >> > > > > are not default: > >> > > > > hbase.hregion.max.filesize = 4294967296 > >> > > > > hbase.hstore.compaction.min = 6 > >> > > > > > >> > > > > I am at a total loss as to why this behavior is occurring. Any > >> help > >> > is > >> > > > > appreciated. > >> > > > > > >> > > > > --Tom > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > -- > >> > > > Bharath Vissapragada > >> > > > <http://www.cloudera.com> > >> > > > > >> > > > >> > > >> > > > > >
