>From hbase-default.xml :
<name>hbase.regionserver.optionalcacheflushinterval</name>
<value>3600000</value>
<description>
Maximum amount of time an edit lives in memory before being
automatically flushed.
Default 1 hour. Set it to 0 to disable automatic flushing.
Can you adjust the above parameter to fit your workload ?
Cheers
On Tue, Dec 10, 2013 at 2:16 PM, Adrien Mogenet <[email protected]>wrote:
> Hi guys,
>
> I've upgraded to 0.94.11. Here is my "worst-case scenario" :
>
> - let say each regionserver has 3 GB memstore
> - let say compaction max filesize is ~200 GB, min. 2 files, max 10 files.
> - let say memstore is growing "slowly" (1 GB / hour per RS)
>
> Then, automatically flushing every hour will lead into 1 GB storefiles,
> being compacted into storefiles of 2 GB, 3 GB, 4.... up to 200 GB.
> Sometimes, my write-load becomes very low, and periodic flusher will flush
> perhaps 1 MB of data, it will trigger a minor compaction of hundreds
> gigabytes + 1 MB; it seems to be lots of IO just to merge 1 MB of data.
>
> Previously (ie. lack of periodic flusher) memstore was creating 3 GB
> storefiles, and thus creating (after minor compactions) 3 GB, 6 GB, 9 GB...
> up to 200 GB storefiles. And if memstore is growing slowly, it won't
> generate small storefiles on HDFS. If think it looks like a more reasonable
> IO-load, doesn't it?
>
> I deeply agree with Periodic Flusher relevance, but I don't think it's
> suitable for everyone. Do you share my opinion wrt. my workload?
>
>
> On Sun, Dec 8, 2013 at 10:36 PM, Ted Yu <[email protected]> wrote:
>
> > Adrien:
> > This config was introduced in 0.94.8
> >
> > Which release did you upgrade to ?
> >
> > As Jean-Marc said, telling us the issue (along with log snippet) would
> > help.
> >
> > Cheers
> >
> >
> > On Mon, Dec 9, 2013 at 1:26 AM, Jean-Marc Spaggiari <
> > [email protected]
> > > wrote:
> >
> > > Hi Adrien,
> > >
> > > What was the bad behavior you faced because of that? Maybe it's what
> need
> > > to be fixed more than the periodic flusher? Or put a bigger default
> > value?
> > >
> > > JM
> > >
> > >
> > > 2013/12/8 Adrien Mogenet <[email protected]>
> > >
> > > > Hi there,
> > > >
> > > > I'm wondering if the Periodic Flusher should be disabled by default?
> > > >
> > > > During a recent upgrade, I've noticed a strange behavior modification
> > on
> > > my
> > > > servers, and it was due to this new feature, supplied with a "1 hour"
> > > > default value. I think upgrades should keep default behavior as close
> > as
> > > in
> > > > previous versions. "By chance", logs were mentioning this "Periodic
> > > > Flusher" but this could have taken much more time to debug otherwise
> > :-)
> > > >
> > > > What are you thoughts guys?
> > > > (Perhaps should be cc'ed to dev list?)
> > > > --
> > > > Adrien Mogenet
> > > > http://www.borntosegfault.com
> > > >
> > >
> >
>
>
>
> --
> Adrien Mogenet
> http://www.borntosegfault.com
>
