Another important information why might be the root cause of this issue... Do you have any TTL defines for this table?
JM 2013/9/24 Jean-Marc Spaggiari <[email protected]> > Strange. > > Few questions then. > 1) What is your hadoop version? > 2) Is the clock on all your severs synched with NTP? > 3) What is you table definition? Bloom filters, etc.? > > This is the reason why it keep compacting: > > 2013-09-24 10:04:00,548 INFO > org.apache.hadoop.hbase.regionserver.compactions.CompactSelection: Deleting > the expired store file by compaction: > hdfs://hdpmgr001.pse.movenetworks.com:8020/hbase/compound3/5ab5fdfcf2aff2633e1d6d5089c96aa2/d/7426f128469242ec8ee09f3965fd5a1a > whose maxTimeStamp is -1 while the max expired timestamp is 1371398640548 > > maxTimeStamp = -1 > > > Each time there is a comparison between maxTimeStamp for this store file > and the configured maxExpiredTimeStamp and since maxTimeStamp returns -1, > it's always elected for a compaction. Now, we need to find why... > > JM > > > 2013/9/24 Tom Brown <[email protected]> > >> My cluster is fully distributed (2 regionserver nodes). >> >> Here is a snippet of log entries that may explain why it started: >> http://pastebin.com/wQECif8k. I had to go back 2 days to find when it >> started for this region. >> >> This is not the only region experiencing this issue (but this is the >> smallest one it's happened to). >> >> --Tom >> >> >> On Tue, Sep 24, 2013 at 10:13 AM, Jean-Marc Spaggiari < >> [email protected]> wrote: >> >> > Can you past logs a bit before that? To see if anything triggered the >> > compaction? >> > Before the 1M compactions entries. >> > >> > Also, what is your setup? Are you running in Standalone? Pseudo-Dist? >> > Fully-Dist? >> > >> > Thanks, >> > >> > JM >> > >> > >> > 2013/9/24 Tom Brown <[email protected]> >> > >> > > There is one column family, d. Each row has about 10 columns, and each >> > > row's total data size is less than 2K. >> > > >> > > Here is a small snippet of logs from the region server: >> > > http://pastebin.com/S2jE4ZAx >> > > >> > > --Tom >> > > >> > > >> > > On Tue, Sep 24, 2013 at 9:59 AM, Bharath Vissapragada < >> > > [email protected] >> > > > wrote: >> > > >> > > > It would help if you can show your RS log (via pastebin?) . Are >> there >> > > > frequent flushes for this region too? >> > > > >> > > > >> > > > On Tue, Sep 24, 2013 at 9:20 PM, Tom Brown <[email protected]> >> > wrote: >> > > > >> > > > > I have a region that is very small, only 5MB. Despite it's size, >> it >> > has >> > > > 24 >> > > > > store files. The logs show that it's compacting (over and over >> > again). >> > > > > >> > > > > The odd thing is that even though there are 24 store files, it >> only >> > > does >> > > > > one at a time. Even more strange is that my logs are filling up >> with >> > > > > compacting this one region. In the last 10 hours, there have been >> > > > 1,876,200 >> > > > > log entries corresponding to compacting this region alone. >> > > > > >> > > > > My cluster is 0.94.10, and using almost all default settings. >> Only a >> > > few >> > > > > are not default: >> > > > > hbase.hregion.max.filesize = 4294967296 >> > > > > hbase.hstore.compaction.min = 6 >> > > > > >> > > > > I am at a total loss as to why this behavior is occurring. Any >> help >> > is >> > > > > appreciated. >> > > > > >> > > > > --Tom >> > > > > >> > > > >> > > > >> > > > >> > > > -- >> > > > Bharath Vissapragada >> > > > <http://www.cloudera.com> >> > > > >> > > >> > >> > >
