TTL seems to be fine. -1 is the default value for TimeRangeTracker.maximumTimestamp.
Can you run: hadoop fs -lsr hdfs:// hdpmgr001.pse.movenetworks.com:8020/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/ Thanks, JM 2013/9/24 Tom Brown <[email protected]> > 1. Hadoop version is 1.1.2. > 2. All servers are synched with NTP. > 3. Table definition is: 'compound0', { > NAME => 'd', > DATA_BLOCK_ENCODING => 'NONE', > BLOOMFILTER => 'ROW', > REPLICATION_SCOPE => '0', > VERSIONS => '1', > COMPRESSION => 'SNAPPY', > MIN_VERSIONS => '0', > TTL => '8640000', > KEEP_DELETED_CELLS => 'false', > BLOCKSIZE => '65536', > IN_MEMORY => 'false', > ENCODE_ON_DISK => 'true', > BLOCKCACHE => 'true' > } > > The TTL is supposed to be 100 days. > > --Tom > > > On Tue, Sep 24, 2013 at 10:53 AM, Jean-Marc Spaggiari < > [email protected]> wrote: > > > Another important information why might be the root cause of this > issue... > > > > Do you have any TTL defines for this table? > > > > JM > > > > > > 2013/9/24 Jean-Marc Spaggiari <[email protected]> > > > > > Strange. > > > > > > Few questions then. > > > 1) What is your hadoop version? > > > 2) Is the clock on all your severs synched with NTP? > > > 3) What is you table definition? Bloom filters, etc.? > > > > > > This is the reason why it keep compacting: > > > > > > 2013-09-24 10:04:00,548 INFO > > org.apache.hadoop.hbase.regionserver.compactions.CompactSelection: > Deleting > > the expired store file by compaction: hdfs:// > > > hdpmgr001.pse.movenetworks.com:8020/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/7426f128469242ec8ee09f3965fd5a1awhosemaxTimeStamp > is -1 while the max expired timestamp is 1371398640548 > > > > > > maxTimeStamp = -1 > > > > > > > > > Each time there is a comparison between maxTimeStamp for this store > file > > > and the configured maxExpiredTimeStamp and since maxTimeStamp returns > -1, > > > it's always elected for a compaction. Now, we need to find why... > > > > > > JM > > > > > > > > > 2013/9/24 Tom Brown <[email protected]> > > > > > >> My cluster is fully distributed (2 regionserver nodes). > > >> > > >> Here is a snippet of log entries that may explain why it started: > > >> http://pastebin.com/wQECif8k. I had to go back 2 days to find when it > > >> started for this region. > > >> > > >> This is not the only region experiencing this issue (but this is the > > >> smallest one it's happened to). > > >> > > >> --Tom > > >> > > >> > > >> On Tue, Sep 24, 2013 at 10:13 AM, Jean-Marc Spaggiari < > > >> [email protected]> wrote: > > >> > > >> > Can you past logs a bit before that? To see if anything triggered > the > > >> > compaction? > > >> > Before the 1M compactions entries. > > >> > > > >> > Also, what is your setup? Are you running in Standalone? > Pseudo-Dist? > > >> > Fully-Dist? > > >> > > > >> > Thanks, > > >> > > > >> > JM > > >> > > > >> > > > >> > 2013/9/24 Tom Brown <[email protected]> > > >> > > > >> > > There is one column family, d. Each row has about 10 columns, and > > each > > >> > > row's total data size is less than 2K. > > >> > > > > >> > > Here is a small snippet of logs from the region server: > > >> > > http://pastebin.com/S2jE4ZAx > > >> > > > > >> > > --Tom > > >> > > > > >> > > > > >> > > On Tue, Sep 24, 2013 at 9:59 AM, Bharath Vissapragada < > > >> > > [email protected] > > >> > > > wrote: > > >> > > > > >> > > > It would help if you can show your RS log (via pastebin?) . Are > > >> there > > >> > > > frequent flushes for this region too? > > >> > > > > > >> > > > > > >> > > > On Tue, Sep 24, 2013 at 9:20 PM, Tom Brown < > [email protected]> > > >> > wrote: > > >> > > > > > >> > > > > I have a region that is very small, only 5MB. Despite it's > size, > > >> it > > >> > has > > >> > > > 24 > > >> > > > > store files. The logs show that it's compacting (over and over > > >> > again). > > >> > > > > > > >> > > > > The odd thing is that even though there are 24 store files, it > > >> only > > >> > > does > > >> > > > > one at a time. Even more strange is that my logs are filling > up > > >> with > > >> > > > > compacting this one region. In the last 10 hours, there have > > been > > >> > > > 1,876,200 > > >> > > > > log entries corresponding to compacting this region alone. > > >> > > > > > > >> > > > > My cluster is 0.94.10, and using almost all default settings. > > >> Only a > > >> > > few > > >> > > > > are not default: > > >> > > > > hbase.hregion.max.filesize = 4294967296 > > >> > > > > hbase.hstore.compaction.min = 6 > > >> > > > > > > >> > > > > I am at a total loss as to why this behavior is occurring. Any > > >> help > > >> > is > > >> > > > > appreciated. > > >> > > > > > > >> > > > > --Tom > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > -- > > >> > > > Bharath Vissapragada > > >> > > > <http://www.cloudera.com> > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
