Hi Lars, Thank you for reply. > We also flush the memstore when the WAL is rolled, which also happens > once/hour.
You we’re absolutely right. I had a lot of log rolls in logs. > Note hbase.regionserver.optionalcacheflushinterval mostly exists to limit the > time edits written with SKIP_WAL should live in the memstore. The problem in my cluster is that I have a lot of very small flushes. By small I mean that most of the time they are in Kbytes. I could live with that, but this causes more serious problem with HBase snapshotting, which I do each hour. Each hour memstore flush happens, which could cause a compaction. After the compaction completes, old hfiles are not deleted, but moved to archive directory, since I have snapshots referencing to them. After running for 6 days with default configs, I have my archive directory bigger than data directory. So my plan was to flush only in case I have memory pressure(hbase.regionserver.global.memstore.lowerLimit), and trigger periodical flushes each 24 hours. Is this approach which will work, or I am doing things wrong. Is there any best practices for my case? Thank you. Kind regards, Akmal Abbasov > On 08 May 2015, at 17:16, lars hofhansl <[email protected]> wrote: > > Do you have some log messages from the flushes? > Maybe you're flushing due to hbase.regionserver.flush.per.changes (number of > edit before we flush). > > We also flush the memstore when the WAL is rolled, which also happens > once/hour. > > Note hbase.regionserver.optionalcacheflushinterval mostly exists to limit the > time edits written with SKIP_WAL should live in the memstore. > > > We need to make more sense of all these settings! > > > -- Lars > > > > ----- Original Message ----- > From: Akmal Abbasov <[email protected]> > To: [email protected] > Cc: > Sent: Thursday, May 7, 2015 2:40 PM > Subject: HBase memstore flush configurations > > Hi all, > I’ve added the following configurations to my hbase-site.xml file. > <property> > <name>hbase.regionserver.optionalcacheflushinterval</name> > <value>86400000</value> > </property> > > <property> > <name>hbase.regionserver.global.memstore.upperLimit</name> > <value>0.4</value> > </property> > > <property> > <name>hbase.regionserver.global.memstore.lowerLimit</name> > <value>0.3</value> > </property> > > But still I have hourly memstore flushes, which are as I understood the > default value for hbase.regionserver.optionalcacheflushinterval. > I restarted hbase daemons on all the nodes, including master and > regionservers. > What could be a reason for this? > Thank you. > > Regards, > Akmal Abbasov
