Yep, I obviously turned it off for my use case since MTTR is not a big concern. I just wanted to talk about the default value, not about my personal case especially. If you consider MTTR is the main objective, then I agree 1 hour is a decent default value :)
Vladimir > you're right, I forgot to mention these points that made my scenario even worse. Thanks for sharing your opinions :) On Wed, Dec 11, 2013 at 4:56 PM, Jean-Marc Spaggiari < [email protected]> wrote: > There is no perfect setting. For Adrien's usecase, this should be turned > off. For some other usecases, this should be turned on. > > I'm -0 to turn it off. I will prefer people to usually have a good MTTR > rather than less compactions. I'm not sure if 200gb region size is a common > usecase. > > > 2013/12/10 lars hofhansl <[email protected]> > > > Should we default it to 0 (off) then? (As Ted pointed out, you can turn > > this off). > > Are you not worried about MTTR when a RegionServer dies with 3GB of logs > > to replay? > > > > > > -- Lars > > > > > > > > ________________________________ > > From: Adrien Mogenet <[email protected]> > > To: user <[email protected]> > > Sent: Tuesday, December 10, 2013 2:16 PM > > Subject: Re: Default value for Periodic Flusher > > > > > > Hi guys, > > > > I've upgraded to 0.94.11. Here is my "worst-case scenario" : > > > > - let say each regionserver has 3 GB memstore > > - let say compaction max filesize is ~200 GB, min. 2 files, max 10 files. > > - let say memstore is growing "slowly" (1 GB / hour per RS) > > > > Then, automatically flushing every hour will lead into 1 GB storefiles, > > being compacted into storefiles of 2 GB, 3 GB, 4.... up to 200 GB. > > Sometimes, my write-load becomes very low, and periodic flusher will > flush > > perhaps 1 MB of data, it will trigger a minor compaction of hundreds > > gigabytes + 1 MB; it seems to be lots of IO just to merge 1 MB of data. > > > > Previously (ie. lack of periodic flusher) memstore was creating 3 GB > > storefiles, and thus creating (after minor compactions) 3 GB, 6 GB, 9 > GB... > > up to 200 GB storefiles. And if memstore is growing slowly, it won't > > generate small storefiles on HDFS. If think it looks like a more > reasonable > > IO-load, doesn't it? > > > > I deeply agree with Periodic Flusher relevance, but I don't think it's > > suitable for everyone. Do you share my opinion wrt. my workload? > > > > > > On Sun, Dec 8, 2013 at 10:36 PM, Ted Yu <[email protected]> wrote: > > > > > Adrien: > > > This config was introduced in 0.94.8 > > > > > > Which release did you upgrade to ? > > > > > > As Jean-Marc said, telling us the issue (along with log snippet) would > > > help. > > > > > > Cheers > > > > > > > > > On Mon, Dec 9, 2013 at 1:26 AM, Jean-Marc Spaggiari < > > > [email protected] > > > > wrote: > > > > > > > Hi Adrien, > > > > > > > > What was the bad behavior you faced because of that? Maybe it's what > > need > > > > to be fixed more than the periodic flusher? Or put a bigger default > > > value? > > > > > > > > JM > > > > > > > > > > > > 2013/12/8 Adrien Mogenet <[email protected]> > > > > > > > > > Hi there, > > > > > > > > > > I'm wondering if the Periodic Flusher should be disabled by > default? > > > > > > > > > > During a recent upgrade, I've noticed a strange behavior > modification > > > on > > > > my > > > > > servers, and it was due to this new feature, supplied with a "1 > hour" > > > > > default value. I think upgrades should keep default behavior as > close > > > as > > > > in > > > > > previous versions. "By chance", logs were mentioning this "Periodic > > > > > Flusher" but this could have taken much more time to debug > otherwise > > > :-) > > > > > > > > > > What are you thoughts guys? > > > > > (Perhaps should be cc'ed to dev list?) > > > > > -- > > > > > Adrien Mogenet > > > > > http://www.borntosegfault.com > > > > > > > > > > > > > > > > > > > > > > -- > > Adrien Mogenet > > http://www.borntosegfault.com > > > -- Adrien Mogenet http://www.borntosegfault.com
