Another thing to add - Avoid bookie becoming RW and R in quick successions. We have observed that if we keep 95% threshold, when it goes to 95% it becomes Ronly, and as soon as it comes to 94% it becomes RW it kept toggling between RW and R .. which was not good.
On Tue, Aug 9, 2016 at 12:44 PM, Sijie Guo <[email protected]> wrote: > On Fri, Aug 5, 2016 at 9:05 AM, Venkateswara Rao Jujjuri < > [email protected]> > wrote: > > > Hi Enrico, > > All are valid points, and wondering if we need to have > > more offered from BookKeeper itself. May be we need to open a Jira for > > this, and also wondering how > > production deployments are dealing with this real issue. > > > > > > - Right now I believe our GC strategy is very basic: Major compaction, > > Minor Compaction with interval and %of reclamation. > > - User defined % of disk space max before going into readOnly; and no > real > > exit strategy once we are there. > > - How do we keep the ZK cookie valid even when we "add" ledger storage, > > which can be an exit strategy. > > > > - Compaction: Add Low Water Mark(LWM) and High Water Mark(HWM) to the > disk > > space. Have different compaction frequency and reclaim %s when the disk > > space is < low water mark , > LWM < HWM, > HWM. > > - Add more intelligence to the major and minor compaction when it can > run. > > Based on system load, and time of the day etc. > > - Have an option to keep 1 Ledger File size disk 'reserved' for exist > > strategy. > > - Make cookie computation so that cookie won't change when storage is > > added. > > > > Anything more? If we agree we can open one or more Jira tickets and work > on > > these. > > > > +1 Look great to me. > > > > > > Thanks, > > JV > > > > On Thu, Aug 4, 2016 at 11:40 PM, Enrico Olivelli - Diennea < > > [email protected]> wrote: > > > > > Hi Charan, > > > I think that the main problem is that GC need some disk space and > cannot > > > be performed if disks are full. > > > > > > Usually it is better to keep disk space under monitoring using external > > > tools (Zabbix...) and prevent the system to be in that situation. > > > > > > You can also keep on your disks some "parachute" to have the ability to > > > free up some space during emergency or at least remove some logs > > > > > > Maybe we can think to implement a way to configure a set of > > > emergency-directories to be used for GC operations as tmp space > > > > > > > > > Il giorno gio, 04/08/2016 alle 14.12 -0700, Charan Reddy G ha scritto: > > > > > > Hi, > > > > > > in LedgerDirsListener of InterleavedLedgerStorage, if the configured > > value > > > of isForceGCAllowWhenNoSpace is false, then when all disks are full, we > > are > > > suspending both Major and Minor GC. I'm wondering what is the exit > > strategy > > > from here? how will we come out of Readonly mode? Since we suspended > both > > > Major and Minor GC even deleting Ledgers wouldn't help > > > > > > @Override > > > > > > *public* *void* allDisksFull() { > > > > > > *if* (gcThread.isForceGCAllowWhenNoSpace) { > > > > > > gcThread.enableForceGC(); > > > > > > } *else* { > > > > > > gcThread.suspendMajorGC(); > > > > > > gcThread.suspendMinorGC(); > > > > > > } > > > > > > } > > > Thanks, > > > Charan > > > > > > > > > -- > > > Enrico Olivelli > > > Software Development Manager @Diennea > > > Tel.: (+39) 0546 066100 - Int. 925 > > > Viale G.Marconi 30/14 - 48018 Faenza (RA) > > > > > > MagNews - E-mail Marketing Solutions > > > http://www.magnews.it > > > Diennea - Digital Marketing Solutions > > > http://www.diennea.com > > > > > > > > > ________________________________ > > > > > > Iscriviti alla nostra newsletter per rimanere aggiornato su digital ed > > > email marketing! http://www.magnews.it/newsletter/ > > > > > > The information in this email is confidential and may be legally > > > privileged. If you are not the intended recipient please notify the > > sender > > > immediately and destroy this email. Any unauthorized, direct or > indirect, > > > disclosure, copying, storage, distribution or other use is strictly > > > forbidden. > > > > > > > > > > > -- > > Jvrao > > --- > > First they ignore you, then they laugh at you, then they fight you, then > > you win. - Mahatma Gandhi > > > -- Jvrao --- First they ignore you, then they laugh at you, then they fight you, then you win. - Mahatma Gandhi
