Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-18 Thread Anthony D'Atri
On older releases, at least, inflated DBs correlated with miserable recovery 
performance and lots of slow requests.  The  DB and OSDs were also on HDD FWIW. 
  A single drive failure would result in substantial RBD impact.  

> On Feb 18, 2019, at 3:28 AM, Dan van der Ster  wrote:
> 
> Not really.
> 
> You should just restart your mons though -- if done one at a time it
> has zero impact on your clients.
> 
> -- dan
> 
> 
> On Mon, Feb 18, 2019 at 12:11 PM M Ranga Swami Reddy
>  wrote:
>> 
>> Hi Sage - If the mon data increases, is this impacts the ceph cluster
>> performance (ie on ceph osd bench, etc)?
>> 
>> On Fri, Feb 15, 2019 at 3:13 PM M Ranga Swami Reddy
>>  wrote:
>>> 
>>> today I again hit the warn with 30G also...
>>> 
 On Thu, Feb 14, 2019 at 7:39 PM Sage Weil  wrote:
 
> On Thu, 7 Feb 2019, Dan van der Ster wrote:
> On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
>  wrote:
>> 
>> Hi Dan,
>>> During backfilling scenarios, the mons keep old maps and grow quite
>>> quickly. So if you have balancing, pg splitting, etc. ongoing for
>>> awhile, the mon stores will eventually trigger that 15GB alarm.
>>> But the intended behavior is that once the PGs are all active+clean,
>>> the old maps should be trimmed and the disk space freed.
>> 
>> old maps not trimmed after cluster reached to "all+clean" state for all 
>> PGs.
>> Is there (known) bug here?
>> As the size of dB showing > 15G, do I need to run the compact commands
>> to do the trimming?
> 
> Compaction isn't necessary -- you should only need to restart all
> peon's then the leader. A few minutes later the db's should start
> trimming.
 
 The next time someone sees this behavior, can you please
 
 - enable debug_mon = 20 on all mons (*before* restarting)
   ceph tell mon.* injectargs '--debug-mon 20'
 - wait for 10 minutes or so to generate some logs
 - add 'debug mon = 20' to ceph.conf (on mons only)
 - restart the monitors
 - wait for them to start trimming
 - remove 'debug mon = 20' from ceph.conf (on mons only)
 - tar up the log files, ceph-post-file them, and share them with ticket
 http://tracker.ceph.com/issues/38322
 
 Thanks!
 sage
 
 
 
 
> -- dan
> 
> 
>> 
>> Thanks
>> Swami
>> 
>>> On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster  
>>> wrote:
>>> 
>>> Hi,
>>> 
>>> With HEALTH_OK a mon data dir should be under 2GB for even such a large 
>>> cluster.
>>> 
>>> During backfilling scenarios, the mons keep old maps and grow quite
>>> quickly. So if you have balancing, pg splitting, etc. ongoing for
>>> awhile, the mon stores will eventually trigger that 15GB alarm.
>>> But the intended behavior is that once the PGs are all active+clean,
>>> the old maps should be trimmed and the disk space freed.
>>> 
>>> However, several people have noted that (at least in luminous
>>> releases) the old maps are not trimmed until after HEALTH_OK *and* all
>>> mons are restarted. This ticket seems related:
>>> http://tracker.ceph.com/issues/37875
>>> 
>>> (Over here we're restarting mons every ~2-3 weeks, resulting in the
>>> mon stores dropping from >15GB to ~700MB each time).
>>> 
>>> -- Dan
>>> 
>>> 
 On Wed, Feb 6, 2019 at 1:26 PM Sage Weil  wrote:
 
 Hi Swami
 
 The limit is somewhat arbitrary, based on cluster sizes we had seen 
 when
 we picked it.  In your case it should be perfectly safe to increase it.
 
 sage
 
 
> On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> 
> Hello -  Are the any limits for mon_data_size for cluster with 2PB
> (with 2000+ OSDs)?
> 
> Currently it set as 15G. What is logic behind this? Can we increase
> when we get the mon_data_size_warn messages?
> 
> I am getting the mon_data_size_warn message even though there a ample
> of free space on the disk (around 300G free disk)
> 
> Earlier thread on the same discusion:
> https://www.spinics.net/lists/ceph-users/msg42456.html
> 
> Thanks
> Swami
> 
> 
> 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-18 Thread M Ranga Swami Reddy
OK, sure will restart the ceph-mon (starting from non leader first,
and then last leader node).


On Mon, Feb 18, 2019 at 4:59 PM Dan van der Ster  wrote:
>
> Not really.
>
> You should just restart your mons though -- if done one at a time it
> has zero impact on your clients.
>
> -- dan
>
>
> On Mon, Feb 18, 2019 at 12:11 PM M Ranga Swami Reddy
>  wrote:
> >
> > Hi Sage - If the mon data increases, is this impacts the ceph cluster
> > performance (ie on ceph osd bench, etc)?
> >
> > On Fri, Feb 15, 2019 at 3:13 PM M Ranga Swami Reddy
> >  wrote:
> > >
> > > today I again hit the warn with 30G also...
> > >
> > > On Thu, Feb 14, 2019 at 7:39 PM Sage Weil  wrote:
> > > >
> > > > On Thu, 7 Feb 2019, Dan van der Ster wrote:
> > > > > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> > > > >  wrote:
> > > > > >
> > > > > > Hi Dan,
> > > > > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > > > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > > > >But the intended behavior is that once the PGs are all 
> > > > > > >active+clean,
> > > > > > >the old maps should be trimmed and the disk space freed.
> > > > > >
> > > > > > old maps not trimmed after cluster reached to "all+clean" state for 
> > > > > > all PGs.
> > > > > > Is there (known) bug here?
> > > > > > As the size of dB showing > 15G, do I need to run the compact 
> > > > > > commands
> > > > > > to do the trimming?
> > > > >
> > > > > Compaction isn't necessary -- you should only need to restart all
> > > > > peon's then the leader. A few minutes later the db's should start
> > > > > trimming.
> > > >
> > > > The next time someone sees this behavior, can you please
> > > >
> > > > - enable debug_mon = 20 on all mons (*before* restarting)
> > > >ceph tell mon.* injectargs '--debug-mon 20'
> > > > - wait for 10 minutes or so to generate some logs
> > > > - add 'debug mon = 20' to ceph.conf (on mons only)
> > > > - restart the monitors
> > > > - wait for them to start trimming
> > > > - remove 'debug mon = 20' from ceph.conf (on mons only)
> > > > - tar up the log files, ceph-post-file them, and share them with ticket
> > > > http://tracker.ceph.com/issues/38322
> > > >
> > > > Thanks!
> > > > sage
> > > >
> > > >
> > > >
> > > >
> > > > > -- dan
> > > > >
> > > > >
> > > > > >
> > > > > > Thanks
> > > > > > Swami
> > > > > >
> > > > > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster 
> > > > > >  wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > With HEALTH_OK a mon data dir should be under 2GB for even such a 
> > > > > > > large cluster.
> > > > > > >
> > > > > > > During backfilling scenarios, the mons keep old maps and grow 
> > > > > > > quite
> > > > > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > > > > But the intended behavior is that once the PGs are all 
> > > > > > > active+clean,
> > > > > > > the old maps should be trimmed and the disk space freed.
> > > > > > >
> > > > > > > However, several people have noted that (at least in luminous
> > > > > > > releases) the old maps are not trimmed until after HEALTH_OK 
> > > > > > > *and* all
> > > > > > > mons are restarted. This ticket seems related:
> > > > > > > http://tracker.ceph.com/issues/37875
> > > > > > >
> > > > > > > (Over here we're restarting mons every ~2-3 weeks, resulting in 
> > > > > > > the
> > > > > > > mon stores dropping from >15GB to ~700MB each time).
> > > > > > >
> > > > > > > -- Dan
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil  
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Swami
> > > > > > > >
> > > > > > > > The limit is somewhat arbitrary, based on cluster sizes we had 
> > > > > > > > seen when
> > > > > > > > we picked it.  In your case it should be perfectly safe to 
> > > > > > > > increase it.
> > > > > > > >
> > > > > > > > sage
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > > > > >
> > > > > > > > > Hello -  Are the any limits for mon_data_size for cluster 
> > > > > > > > > with 2PB
> > > > > > > > > (with 2000+ OSDs)?
> > > > > > > > >
> > > > > > > > > Currently it set as 15G. What is logic behind this? Can we 
> > > > > > > > > increase
> > > > > > > > > when we get the mon_data_size_warn messages?
> > > > > > > > >
> > > > > > > > > I am getting the mon_data_size_warn message even though there 
> > > > > > > > > a ample
> > > > > > > > > of free space on the disk (around 300G free disk)
> > > > > > > > >
> > > > > > > > > Earlier thread on the same discusion:
> > > > > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Swami
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > 

Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-18 Thread Dan van der Ster
Not really.

You should just restart your mons though -- if done one at a time it
has zero impact on your clients.

-- dan


On Mon, Feb 18, 2019 at 12:11 PM M Ranga Swami Reddy
 wrote:
>
> Hi Sage - If the mon data increases, is this impacts the ceph cluster
> performance (ie on ceph osd bench, etc)?
>
> On Fri, Feb 15, 2019 at 3:13 PM M Ranga Swami Reddy
>  wrote:
> >
> > today I again hit the warn with 30G also...
> >
> > On Thu, Feb 14, 2019 at 7:39 PM Sage Weil  wrote:
> > >
> > > On Thu, 7 Feb 2019, Dan van der Ster wrote:
> > > > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> > > >  wrote:
> > > > >
> > > > > Hi Dan,
> > > > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > > >But the intended behavior is that once the PGs are all active+clean,
> > > > > >the old maps should be trimmed and the disk space freed.
> > > > >
> > > > > old maps not trimmed after cluster reached to "all+clean" state for 
> > > > > all PGs.
> > > > > Is there (known) bug here?
> > > > > As the size of dB showing > 15G, do I need to run the compact commands
> > > > > to do the trimming?
> > > >
> > > > Compaction isn't necessary -- you should only need to restart all
> > > > peon's then the leader. A few minutes later the db's should start
> > > > trimming.
> > >
> > > The next time someone sees this behavior, can you please
> > >
> > > - enable debug_mon = 20 on all mons (*before* restarting)
> > >ceph tell mon.* injectargs '--debug-mon 20'
> > > - wait for 10 minutes or so to generate some logs
> > > - add 'debug mon = 20' to ceph.conf (on mons only)
> > > - restart the monitors
> > > - wait for them to start trimming
> > > - remove 'debug mon = 20' from ceph.conf (on mons only)
> > > - tar up the log files, ceph-post-file them, and share them with ticket
> > > http://tracker.ceph.com/issues/38322
> > >
> > > Thanks!
> > > sage
> > >
> > >
> > >
> > >
> > > > -- dan
> > > >
> > > >
> > > > >
> > > > > Thanks
> > > > > Swami
> > > > >
> > > > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster  
> > > > > wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > With HEALTH_OK a mon data dir should be under 2GB for even such a 
> > > > > > large cluster.
> > > > > >
> > > > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > > > But the intended behavior is that once the PGs are all active+clean,
> > > > > > the old maps should be trimmed and the disk space freed.
> > > > > >
> > > > > > However, several people have noted that (at least in luminous
> > > > > > releases) the old maps are not trimmed until after HEALTH_OK *and* 
> > > > > > all
> > > > > > mons are restarted. This ticket seems related:
> > > > > > http://tracker.ceph.com/issues/37875
> > > > > >
> > > > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > > > mon stores dropping from >15GB to ~700MB each time).
> > > > > >
> > > > > > -- Dan
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil  wrote:
> > > > > > >
> > > > > > > Hi Swami
> > > > > > >
> > > > > > > The limit is somewhat arbitrary, based on cluster sizes we had 
> > > > > > > seen when
> > > > > > > we picked it.  In your case it should be perfectly safe to 
> > > > > > > increase it.
> > > > > > >
> > > > > > > sage
> > > > > > >
> > > > > > >
> > > > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > > > >
> > > > > > > > Hello -  Are the any limits for mon_data_size for cluster with 
> > > > > > > > 2PB
> > > > > > > > (with 2000+ OSDs)?
> > > > > > > >
> > > > > > > > Currently it set as 15G. What is logic behind this? Can we 
> > > > > > > > increase
> > > > > > > > when we get the mon_data_size_warn messages?
> > > > > > > >
> > > > > > > > I am getting the mon_data_size_warn message even though there a 
> > > > > > > > ample
> > > > > > > > of free space on the disk (around 300G free disk)
> > > > > > > >
> > > > > > > > Earlier thread on the same discusion:
> > > > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Swami
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > ___
> > > > > > > ceph-users mailing list
> > > > > > > ceph-users@lists.ceph.com
> > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > >
> > > >
> > > >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-18 Thread Dan van der Ster
On Thu, Feb 14, 2019 at 2:31 PM Sage Weil  wrote:
>
> On Thu, 7 Feb 2019, Dan van der Ster wrote:
> > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> >  wrote:
> > >
> > > Hi Dan,
> > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > >But the intended behavior is that once the PGs are all active+clean,
> > > >the old maps should be trimmed and the disk space freed.
> > >
> > > old maps not trimmed after cluster reached to "all+clean" state for all 
> > > PGs.
> > > Is there (known) bug here?
> > > As the size of dB showing > 15G, do I need to run the compact commands
> > > to do the trimming?
> >
> > Compaction isn't necessary -- you should only need to restart all
> > peon's then the leader. A few minutes later the db's should start
> > trimming.
>
> The next time someone sees this behavior, can you please
>
> - enable debug_mon = 20 on all mons (*before* restarting)
>ceph tell mon.* injectargs '--debug-mon 20'
> - wait for 10 minutes or so to generate some logs
> - add 'debug mon = 20' to ceph.conf (on mons only)
> - restart the monitors
> - wait for them to start trimming
> - remove 'debug mon = 20' from ceph.conf (on mons only)
> - tar up the log files, ceph-post-file them, and share them with ticket
> http://tracker.ceph.com/issues/38322
>

Not sure if you noticed, but we sent some logs Friday.

-- dan

> Thanks!
> sage
>
>
>
>
> > -- dan
> >
> >
> > >
> > > Thanks
> > > Swami
> > >
> > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster  
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > With HEALTH_OK a mon data dir should be under 2GB for even such a large 
> > > > cluster.
> > > >
> > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > But the intended behavior is that once the PGs are all active+clean,
> > > > the old maps should be trimmed and the disk space freed.
> > > >
> > > > However, several people have noted that (at least in luminous
> > > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > > mons are restarted. This ticket seems related:
> > > > http://tracker.ceph.com/issues/37875
> > > >
> > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > mon stores dropping from >15GB to ~700MB each time).
> > > >
> > > > -- Dan
> > > >
> > > >
> > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil  wrote:
> > > > >
> > > > > Hi Swami
> > > > >
> > > > > The limit is somewhat arbitrary, based on cluster sizes we had seen 
> > > > > when
> > > > > we picked it.  In your case it should be perfectly safe to increase 
> > > > > it.
> > > > >
> > > > > sage
> > > > >
> > > > >
> > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > >
> > > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > > (with 2000+ OSDs)?
> > > > > >
> > > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > > when we get the mon_data_size_warn messages?
> > > > > >
> > > > > > I am getting the mon_data_size_warn message even though there a 
> > > > > > ample
> > > > > > of free space on the disk (around 300G free disk)
> > > > > >
> > > > > > Earlier thread on the same discusion:
> > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > >
> > > > > > Thanks
> > > > > > Swami
> > > > > >
> > > > > >
> > > > > >
> > > > > ___
> > > > > ceph-users mailing list
> > > > > ceph-users@lists.ceph.com
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-18 Thread M Ranga Swami Reddy
Hi Sage - If the mon data increases, is this impacts the ceph cluster
performance (ie on ceph osd bench, etc)?

On Fri, Feb 15, 2019 at 3:13 PM M Ranga Swami Reddy
 wrote:
>
> today I again hit the warn with 30G also...
>
> On Thu, Feb 14, 2019 at 7:39 PM Sage Weil  wrote:
> >
> > On Thu, 7 Feb 2019, Dan van der Ster wrote:
> > > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> > >  wrote:
> > > >
> > > > Hi Dan,
> > > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > >But the intended behavior is that once the PGs are all active+clean,
> > > > >the old maps should be trimmed and the disk space freed.
> > > >
> > > > old maps not trimmed after cluster reached to "all+clean" state for all 
> > > > PGs.
> > > > Is there (known) bug here?
> > > > As the size of dB showing > 15G, do I need to run the compact commands
> > > > to do the trimming?
> > >
> > > Compaction isn't necessary -- you should only need to restart all
> > > peon's then the leader. A few minutes later the db's should start
> > > trimming.
> >
> > The next time someone sees this behavior, can you please
> >
> > - enable debug_mon = 20 on all mons (*before* restarting)
> >ceph tell mon.* injectargs '--debug-mon 20'
> > - wait for 10 minutes or so to generate some logs
> > - add 'debug mon = 20' to ceph.conf (on mons only)
> > - restart the monitors
> > - wait for them to start trimming
> > - remove 'debug mon = 20' from ceph.conf (on mons only)
> > - tar up the log files, ceph-post-file them, and share them with ticket
> > http://tracker.ceph.com/issues/38322
> >
> > Thanks!
> > sage
> >
> >
> >
> >
> > > -- dan
> > >
> > >
> > > >
> > > > Thanks
> > > > Swami
> > > >
> > > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster  
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > With HEALTH_OK a mon data dir should be under 2GB for even such a 
> > > > > large cluster.
> > > > >
> > > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > > But the intended behavior is that once the PGs are all active+clean,
> > > > > the old maps should be trimmed and the disk space freed.
> > > > >
> > > > > However, several people have noted that (at least in luminous
> > > > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > > > mons are restarted. This ticket seems related:
> > > > > http://tracker.ceph.com/issues/37875
> > > > >
> > > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > > mon stores dropping from >15GB to ~700MB each time).
> > > > >
> > > > > -- Dan
> > > > >
> > > > >
> > > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil  wrote:
> > > > > >
> > > > > > Hi Swami
> > > > > >
> > > > > > The limit is somewhat arbitrary, based on cluster sizes we had seen 
> > > > > > when
> > > > > > we picked it.  In your case it should be perfectly safe to increase 
> > > > > > it.
> > > > > >
> > > > > > sage
> > > > > >
> > > > > >
> > > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > > >
> > > > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > > > (with 2000+ OSDs)?
> > > > > > >
> > > > > > > Currently it set as 15G. What is logic behind this? Can we 
> > > > > > > increase
> > > > > > > when we get the mon_data_size_warn messages?
> > > > > > >
> > > > > > > I am getting the mon_data_size_warn message even though there a 
> > > > > > > ample
> > > > > > > of free space on the disk (around 300G free disk)
> > > > > > >
> > > > > > > Earlier thread on the same discusion:
> > > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > > >
> > > > > > > Thanks
> > > > > > > Swami
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > ___
> > > > > > ceph-users mailing list
> > > > > > ceph-users@lists.ceph.com
> > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > >
> > >
> > >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-15 Thread M Ranga Swami Reddy
today I again hit the warn with 30G also...

On Thu, Feb 14, 2019 at 7:39 PM Sage Weil  wrote:
>
> On Thu, 7 Feb 2019, Dan van der Ster wrote:
> > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> >  wrote:
> > >
> > > Hi Dan,
> > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > >But the intended behavior is that once the PGs are all active+clean,
> > > >the old maps should be trimmed and the disk space freed.
> > >
> > > old maps not trimmed after cluster reached to "all+clean" state for all 
> > > PGs.
> > > Is there (known) bug here?
> > > As the size of dB showing > 15G, do I need to run the compact commands
> > > to do the trimming?
> >
> > Compaction isn't necessary -- you should only need to restart all
> > peon's then the leader. A few minutes later the db's should start
> > trimming.
>
> The next time someone sees this behavior, can you please
>
> - enable debug_mon = 20 on all mons (*before* restarting)
>ceph tell mon.* injectargs '--debug-mon 20'
> - wait for 10 minutes or so to generate some logs
> - add 'debug mon = 20' to ceph.conf (on mons only)
> - restart the monitors
> - wait for them to start trimming
> - remove 'debug mon = 20' from ceph.conf (on mons only)
> - tar up the log files, ceph-post-file them, and share them with ticket
> http://tracker.ceph.com/issues/38322
>
> Thanks!
> sage
>
>
>
>
> > -- dan
> >
> >
> > >
> > > Thanks
> > > Swami
> > >
> > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster  
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > With HEALTH_OK a mon data dir should be under 2GB for even such a large 
> > > > cluster.
> > > >
> > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > But the intended behavior is that once the PGs are all active+clean,
> > > > the old maps should be trimmed and the disk space freed.
> > > >
> > > > However, several people have noted that (at least in luminous
> > > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > > mons are restarted. This ticket seems related:
> > > > http://tracker.ceph.com/issues/37875
> > > >
> > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > mon stores dropping from >15GB to ~700MB each time).
> > > >
> > > > -- Dan
> > > >
> > > >
> > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil  wrote:
> > > > >
> > > > > Hi Swami
> > > > >
> > > > > The limit is somewhat arbitrary, based on cluster sizes we had seen 
> > > > > when
> > > > > we picked it.  In your case it should be perfectly safe to increase 
> > > > > it.
> > > > >
> > > > > sage
> > > > >
> > > > >
> > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > >
> > > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > > (with 2000+ OSDs)?
> > > > > >
> > > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > > when we get the mon_data_size_warn messages?
> > > > > >
> > > > > > I am getting the mon_data_size_warn message even though there a 
> > > > > > ample
> > > > > > of free space on the disk (around 300G free disk)
> > > > > >
> > > > > > Earlier thread on the same discusion:
> > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > >
> > > > > > Thanks
> > > > > > Swami
> > > > > >
> > > > > >
> > > > > >
> > > > > ___
> > > > > ceph-users mailing list
> > > > > ceph-users@lists.ceph.com
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-14 Thread M Ranga Swami Reddy
Sure, will this. For now I have creased the size to 30G (from 15G).

On Thu, Feb 14, 2019 at 7:39 PM Sage Weil  wrote:
>
> On Thu, 7 Feb 2019, Dan van der Ster wrote:
> > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> >  wrote:
> > >
> > > Hi Dan,
> > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > >But the intended behavior is that once the PGs are all active+clean,
> > > >the old maps should be trimmed and the disk space freed.
> > >
> > > old maps not trimmed after cluster reached to "all+clean" state for all 
> > > PGs.
> > > Is there (known) bug here?
> > > As the size of dB showing > 15G, do I need to run the compact commands
> > > to do the trimming?
> >
> > Compaction isn't necessary -- you should only need to restart all
> > peon's then the leader. A few minutes later the db's should start
> > trimming.
>
> The next time someone sees this behavior, can you please
>
> - enable debug_mon = 20 on all mons (*before* restarting)
>ceph tell mon.* injectargs '--debug-mon 20'
> - wait for 10 minutes or so to generate some logs
> - add 'debug mon = 20' to ceph.conf (on mons only)
> - restart the monitors
> - wait for them to start trimming
> - remove 'debug mon = 20' from ceph.conf (on mons only)
> - tar up the log files, ceph-post-file them, and share them with ticket
> http://tracker.ceph.com/issues/38322
>
> Thanks!
> sage
>
>
>
>
> > -- dan
> >
> >
> > >
> > > Thanks
> > > Swami
> > >
> > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster  
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > With HEALTH_OK a mon data dir should be under 2GB for even such a large 
> > > > cluster.
> > > >
> > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > But the intended behavior is that once the PGs are all active+clean,
> > > > the old maps should be trimmed and the disk space freed.
> > > >
> > > > However, several people have noted that (at least in luminous
> > > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > > mons are restarted. This ticket seems related:
> > > > http://tracker.ceph.com/issues/37875
> > > >
> > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > mon stores dropping from >15GB to ~700MB each time).
> > > >
> > > > -- Dan
> > > >
> > > >
> > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil  wrote:
> > > > >
> > > > > Hi Swami
> > > > >
> > > > > The limit is somewhat arbitrary, based on cluster sizes we had seen 
> > > > > when
> > > > > we picked it.  In your case it should be perfectly safe to increase 
> > > > > it.
> > > > >
> > > > > sage
> > > > >
> > > > >
> > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > >
> > > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > > (with 2000+ OSDs)?
> > > > > >
> > > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > > when we get the mon_data_size_warn messages?
> > > > > >
> > > > > > I am getting the mon_data_size_warn message even though there a 
> > > > > > ample
> > > > > > of free space on the disk (around 300G free disk)
> > > > > >
> > > > > > Earlier thread on the same discusion:
> > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > >
> > > > > > Thanks
> > > > > > Swami
> > > > > >
> > > > > >
> > > > > >
> > > > > ___
> > > > > ceph-users mailing list
> > > > > ceph-users@lists.ceph.com
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-14 Thread Sage Weil
On Thu, 7 Feb 2019, Dan van der Ster wrote:
> On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
>  wrote:
> >
> > Hi Dan,
> > >During backfilling scenarios, the mons keep old maps and grow quite
> > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > >But the intended behavior is that once the PGs are all active+clean,
> > >the old maps should be trimmed and the disk space freed.
> >
> > old maps not trimmed after cluster reached to "all+clean" state for all PGs.
> > Is there (known) bug here?
> > As the size of dB showing > 15G, do I need to run the compact commands
> > to do the trimming?
> 
> Compaction isn't necessary -- you should only need to restart all
> peon's then the leader. A few minutes later the db's should start
> trimming.

The next time someone sees this behavior, can you please

- enable debug_mon = 20 on all mons (*before* restarting)
   ceph tell mon.* injectargs '--debug-mon 20'
- wait for 10 minutes or so to generate some logs
- add 'debug mon = 20' to ceph.conf (on mons only)
- restart the monitors
- wait for them to start trimming
- remove 'debug mon = 20' from ceph.conf (on mons only)
- tar up the log files, ceph-post-file them, and share them with ticket 
http://tracker.ceph.com/issues/38322

Thanks!
sage



 
> -- dan
> 
> 
> >
> > Thanks
> > Swami
> >
> > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster  wrote:
> > >
> > > Hi,
> > >
> > > With HEALTH_OK a mon data dir should be under 2GB for even such a large 
> > > cluster.
> > >
> > > During backfilling scenarios, the mons keep old maps and grow quite
> > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > But the intended behavior is that once the PGs are all active+clean,
> > > the old maps should be trimmed and the disk space freed.
> > >
> > > However, several people have noted that (at least in luminous
> > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > mons are restarted. This ticket seems related:
> > > http://tracker.ceph.com/issues/37875
> > >
> > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > mon stores dropping from >15GB to ~700MB each time).
> > >
> > > -- Dan
> > >
> > >
> > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil  wrote:
> > > >
> > > > Hi Swami
> > > >
> > > > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > > > we picked it.  In your case it should be perfectly safe to increase it.
> > > >
> > > > sage
> > > >
> > > >
> > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > >
> > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > (with 2000+ OSDs)?
> > > > >
> > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > when we get the mon_data_size_warn messages?
> > > > >
> > > > > I am getting the mon_data_size_warn message even though there a ample
> > > > > of free space on the disk (around 300G free disk)
> > > > >
> > > > > Earlier thread on the same discusion:
> > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > >
> > > > > Thanks
> > > > > Swami
> > > > >
> > > > >
> > > > >
> > > > ___
> > > > ceph-users mailing list
> > > > ceph-users@lists.ceph.com
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-07 Thread M Ranga Swami Reddy
Alternatively, will increase the mon_data_size to 30G (from 15G)..

Thanks
Swami

On Thu, Feb 7, 2019 at 8:44 PM Dan van der Ster  wrote:
>
> On Thu, Feb 7, 2019 at 4:12 PM M Ranga Swami Reddy  
> wrote:
> >
> > >Compaction isn't necessary -- you should only need to restart all
> > >peon's then the leader. A few minutes later the db's should start
> > >trimming.
> >
> > As we on production cluster, which may not be safe to restart the
> > ceph-mon, instead prefer to do the compact on non-leader mons.
> > Is this ok?
> >
>
> Compaction doesn't solve this particular problem, because the maps
> have not yet been deleted by the ceph-mon process.
>
> -- dan
>
>
> > Thanks
> > Swami
> >
> > On Thu, Feb 7, 2019 at 6:30 PM Dan van der Ster  wrote:
> > >
> > > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> > >  wrote:
> > > >
> > > > Hi Dan,
> > > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > >But the intended behavior is that once the PGs are all active+clean,
> > > > >the old maps should be trimmed and the disk space freed.
> > > >
> > > > old maps not trimmed after cluster reached to "all+clean" state for all 
> > > > PGs.
> > > > Is there (known) bug here?
> > > > As the size of dB showing > 15G, do I need to run the compact commands
> > > > to do the trimming?
> > >
> > > Compaction isn't necessary -- you should only need to restart all
> > > peon's then the leader. A few minutes later the db's should start
> > > trimming.
> > >
> > > -- dan
> > >
> > >
> > > >
> > > > Thanks
> > > > Swami
> > > >
> > > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster  
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > With HEALTH_OK a mon data dir should be under 2GB for even such a 
> > > > > large cluster.
> > > > >
> > > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > > But the intended behavior is that once the PGs are all active+clean,
> > > > > the old maps should be trimmed and the disk space freed.
> > > > >
> > > > > However, several people have noted that (at least in luminous
> > > > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > > > mons are restarted. This ticket seems related:
> > > > > http://tracker.ceph.com/issues/37875
> > > > >
> > > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > > mon stores dropping from >15GB to ~700MB each time).
> > > > >
> > > > > -- Dan
> > > > >
> > > > >
> > > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil  wrote:
> > > > > >
> > > > > > Hi Swami
> > > > > >
> > > > > > The limit is somewhat arbitrary, based on cluster sizes we had seen 
> > > > > > when
> > > > > > we picked it.  In your case it should be perfectly safe to increase 
> > > > > > it.
> > > > > >
> > > > > > sage
> > > > > >
> > > > > >
> > > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > > >
> > > > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > > > (with 2000+ OSDs)?
> > > > > > >
> > > > > > > Currently it set as 15G. What is logic behind this? Can we 
> > > > > > > increase
> > > > > > > when we get the mon_data_size_warn messages?
> > > > > > >
> > > > > > > I am getting the mon_data_size_warn message even though there a 
> > > > > > > ample
> > > > > > > of free space on the disk (around 300G free disk)
> > > > > > >
> > > > > > > Earlier thread on the same discusion:
> > > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > > >
> > > > > > > Thanks
> > > > > > > Swami
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > ___
> > > > > > ceph-users mailing list
> > > > > > ceph-users@lists.ceph.com
> > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-07 Thread Dan van der Ster
On Thu, Feb 7, 2019 at 4:12 PM M Ranga Swami Reddy  wrote:
>
> >Compaction isn't necessary -- you should only need to restart all
> >peon's then the leader. A few minutes later the db's should start
> >trimming.
>
> As we on production cluster, which may not be safe to restart the
> ceph-mon, instead prefer to do the compact on non-leader mons.
> Is this ok?
>

Compaction doesn't solve this particular problem, because the maps
have not yet been deleted by the ceph-mon process.

-- dan


> Thanks
> Swami
>
> On Thu, Feb 7, 2019 at 6:30 PM Dan van der Ster  wrote:
> >
> > On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
> >  wrote:
> > >
> > > Hi Dan,
> > > >During backfilling scenarios, the mons keep old maps and grow quite
> > > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > > >But the intended behavior is that once the PGs are all active+clean,
> > > >the old maps should be trimmed and the disk space freed.
> > >
> > > old maps not trimmed after cluster reached to "all+clean" state for all 
> > > PGs.
> > > Is there (known) bug here?
> > > As the size of dB showing > 15G, do I need to run the compact commands
> > > to do the trimming?
> >
> > Compaction isn't necessary -- you should only need to restart all
> > peon's then the leader. A few minutes later the db's should start
> > trimming.
> >
> > -- dan
> >
> >
> > >
> > > Thanks
> > > Swami
> > >
> > > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster  
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > With HEALTH_OK a mon data dir should be under 2GB for even such a large 
> > > > cluster.
> > > >
> > > > During backfilling scenarios, the mons keep old maps and grow quite
> > > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > > But the intended behavior is that once the PGs are all active+clean,
> > > > the old maps should be trimmed and the disk space freed.
> > > >
> > > > However, several people have noted that (at least in luminous
> > > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > > mons are restarted. This ticket seems related:
> > > > http://tracker.ceph.com/issues/37875
> > > >
> > > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > > mon stores dropping from >15GB to ~700MB each time).
> > > >
> > > > -- Dan
> > > >
> > > >
> > > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil  wrote:
> > > > >
> > > > > Hi Swami
> > > > >
> > > > > The limit is somewhat arbitrary, based on cluster sizes we had seen 
> > > > > when
> > > > > we picked it.  In your case it should be perfectly safe to increase 
> > > > > it.
> > > > >
> > > > > sage
> > > > >
> > > > >
> > > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > > >
> > > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > > (with 2000+ OSDs)?
> > > > > >
> > > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > > when we get the mon_data_size_warn messages?
> > > > > >
> > > > > > I am getting the mon_data_size_warn message even though there a 
> > > > > > ample
> > > > > > of free space on the disk (around 300G free disk)
> > > > > >
> > > > > > Earlier thread on the same discusion:
> > > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > > >
> > > > > > Thanks
> > > > > > Swami
> > > > > >
> > > > > >
> > > > > >
> > > > > ___
> > > > > ceph-users mailing list
> > > > > ceph-users@lists.ceph.com
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-07 Thread M Ranga Swami Reddy
>Compaction isn't necessary -- you should only need to restart all
>peon's then the leader. A few minutes later the db's should start
>trimming.

As we on production cluster, which may not be safe to restart the
ceph-mon, instead prefer to do the compact on non-leader mons.
Is this ok?

Thanks
Swami

On Thu, Feb 7, 2019 at 6:30 PM Dan van der Ster  wrote:
>
> On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
>  wrote:
> >
> > Hi Dan,
> > >During backfilling scenarios, the mons keep old maps and grow quite
> > >quickly. So if you have balancing, pg splitting, etc. ongoing for
> > >awhile, the mon stores will eventually trigger that 15GB alarm.
> > >But the intended behavior is that once the PGs are all active+clean,
> > >the old maps should be trimmed and the disk space freed.
> >
> > old maps not trimmed after cluster reached to "all+clean" state for all PGs.
> > Is there (known) bug here?
> > As the size of dB showing > 15G, do I need to run the compact commands
> > to do the trimming?
>
> Compaction isn't necessary -- you should only need to restart all
> peon's then the leader. A few minutes later the db's should start
> trimming.
>
> -- dan
>
>
> >
> > Thanks
> > Swami
> >
> > On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster  wrote:
> > >
> > > Hi,
> > >
> > > With HEALTH_OK a mon data dir should be under 2GB for even such a large 
> > > cluster.
> > >
> > > During backfilling scenarios, the mons keep old maps and grow quite
> > > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > > awhile, the mon stores will eventually trigger that 15GB alarm.
> > > But the intended behavior is that once the PGs are all active+clean,
> > > the old maps should be trimmed and the disk space freed.
> > >
> > > However, several people have noted that (at least in luminous
> > > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > > mons are restarted. This ticket seems related:
> > > http://tracker.ceph.com/issues/37875
> > >
> > > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > > mon stores dropping from >15GB to ~700MB each time).
> > >
> > > -- Dan
> > >
> > >
> > > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil  wrote:
> > > >
> > > > Hi Swami
> > > >
> > > > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > > > we picked it.  In your case it should be perfectly safe to increase it.
> > > >
> > > > sage
> > > >
> > > >
> > > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > > >
> > > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > > (with 2000+ OSDs)?
> > > > >
> > > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > > when we get the mon_data_size_warn messages?
> > > > >
> > > > > I am getting the mon_data_size_warn message even though there a ample
> > > > > of free space on the disk (around 300G free disk)
> > > > >
> > > > > Earlier thread on the same discusion:
> > > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > > >
> > > > > Thanks
> > > > > Swami
> > > > >
> > > > >
> > > > >
> > > > ___
> > > > ceph-users mailing list
> > > > ceph-users@lists.ceph.com
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-07 Thread Dan van der Ster
On Thu, Feb 7, 2019 at 12:17 PM M Ranga Swami Reddy
 wrote:
>
> Hi Dan,
> >During backfilling scenarios, the mons keep old maps and grow quite
> >quickly. So if you have balancing, pg splitting, etc. ongoing for
> >awhile, the mon stores will eventually trigger that 15GB alarm.
> >But the intended behavior is that once the PGs are all active+clean,
> >the old maps should be trimmed and the disk space freed.
>
> old maps not trimmed after cluster reached to "all+clean" state for all PGs.
> Is there (known) bug here?
> As the size of dB showing > 15G, do I need to run the compact commands
> to do the trimming?

Compaction isn't necessary -- you should only need to restart all
peon's then the leader. A few minutes later the db's should start
trimming.

-- dan


>
> Thanks
> Swami
>
> On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster  wrote:
> >
> > Hi,
> >
> > With HEALTH_OK a mon data dir should be under 2GB for even such a large 
> > cluster.
> >
> > During backfilling scenarios, the mons keep old maps and grow quite
> > quickly. So if you have balancing, pg splitting, etc. ongoing for
> > awhile, the mon stores will eventually trigger that 15GB alarm.
> > But the intended behavior is that once the PGs are all active+clean,
> > the old maps should be trimmed and the disk space freed.
> >
> > However, several people have noted that (at least in luminous
> > releases) the old maps are not trimmed until after HEALTH_OK *and* all
> > mons are restarted. This ticket seems related:
> > http://tracker.ceph.com/issues/37875
> >
> > (Over here we're restarting mons every ~2-3 weeks, resulting in the
> > mon stores dropping from >15GB to ~700MB each time).
> >
> > -- Dan
> >
> >
> > On Wed, Feb 6, 2019 at 1:26 PM Sage Weil  wrote:
> > >
> > > Hi Swami
> > >
> > > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > > we picked it.  In your case it should be perfectly safe to increase it.
> > >
> > > sage
> > >
> > >
> > > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> > >
> > > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > > (with 2000+ OSDs)?
> > > >
> > > > Currently it set as 15G. What is logic behind this? Can we increase
> > > > when we get the mon_data_size_warn messages?
> > > >
> > > > I am getting the mon_data_size_warn message even though there a ample
> > > > of free space on the disk (around 300G free disk)
> > > >
> > > > Earlier thread on the same discusion:
> > > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > > >
> > > > Thanks
> > > > Swami
> > > >
> > > >
> > > >
> > > ___
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-07 Thread M Ranga Swami Reddy
Hi Sage

Sure, we will increase the mon_data_size to 30G to avoid this type of
warning. And currently we are using 500G disk here. I guss, which
should good enough here.

Thanks
Swami

On Wed, Feb 6, 2019 at 5:56 PM Sage Weil  wrote:
>
> Hi Swami
>
> The limit is somewhat arbitrary, based on cluster sizes we had seen when
> we picked it.  In your case it should be perfectly safe to increase it.
>
> sage
>
>
> On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
>
> > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > (with 2000+ OSDs)?
> >
> > Currently it set as 15G. What is logic behind this? Can we increase
> > when we get the mon_data_size_warn messages?
> >
> > I am getting the mon_data_size_warn message even though there a ample
> > of free space on the disk (around 300G free disk)
> >
> > Earlier thread on the same discusion:
> > https://www.spinics.net/lists/ceph-users/msg42456.html
> >
> > Thanks
> > Swami
> >
> >
> >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-07 Thread M Ranga Swami Reddy
Hi Dan,
>During backfilling scenarios, the mons keep old maps and grow quite
>quickly. So if you have balancing, pg splitting, etc. ongoing for
>awhile, the mon stores will eventually trigger that 15GB alarm.
>But the intended behavior is that once the PGs are all active+clean,
>the old maps should be trimmed and the disk space freed.

old maps not trimmed after cluster reached to "all+clean" state for all PGs.
Is there (known) bug here?
As the size of dB showing > 15G, do I need to run the compact commands
to do the trimming?

Thanks
Swami

On Wed, Feb 6, 2019 at 6:24 PM Dan van der Ster  wrote:
>
> Hi,
>
> With HEALTH_OK a mon data dir should be under 2GB for even such a large 
> cluster.
>
> During backfilling scenarios, the mons keep old maps and grow quite
> quickly. So if you have balancing, pg splitting, etc. ongoing for
> awhile, the mon stores will eventually trigger that 15GB alarm.
> But the intended behavior is that once the PGs are all active+clean,
> the old maps should be trimmed and the disk space freed.
>
> However, several people have noted that (at least in luminous
> releases) the old maps are not trimmed until after HEALTH_OK *and* all
> mons are restarted. This ticket seems related:
> http://tracker.ceph.com/issues/37875
>
> (Over here we're restarting mons every ~2-3 weeks, resulting in the
> mon stores dropping from >15GB to ~700MB each time).
>
> -- Dan
>
>
> On Wed, Feb 6, 2019 at 1:26 PM Sage Weil  wrote:
> >
> > Hi Swami
> >
> > The limit is somewhat arbitrary, based on cluster sizes we had seen when
> > we picked it.  In your case it should be perfectly safe to increase it.
> >
> > sage
> >
> >
> > On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
> >
> > > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > > (with 2000+ OSDs)?
> > >
> > > Currently it set as 15G. What is logic behind this? Can we increase
> > > when we get the mon_data_size_warn messages?
> > >
> > > I am getting the mon_data_size_warn message even though there a ample
> > > of free space on the disk (around 300G free disk)
> > >
> > > Earlier thread on the same discusion:
> > > https://www.spinics.net/lists/ceph-users/msg42456.html
> > >
> > > Thanks
> > > Swami
> > >
> > >
> > >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-06 Thread Dan van der Ster
Hi,

With HEALTH_OK a mon data dir should be under 2GB for even such a large cluster.

During backfilling scenarios, the mons keep old maps and grow quite
quickly. So if you have balancing, pg splitting, etc. ongoing for
awhile, the mon stores will eventually trigger that 15GB alarm.
But the intended behavior is that once the PGs are all active+clean,
the old maps should be trimmed and the disk space freed.

However, several people have noted that (at least in luminous
releases) the old maps are not trimmed until after HEALTH_OK *and* all
mons are restarted. This ticket seems related:
http://tracker.ceph.com/issues/37875

(Over here we're restarting mons every ~2-3 weeks, resulting in the
mon stores dropping from >15GB to ~700MB each time).

-- Dan


On Wed, Feb 6, 2019 at 1:26 PM Sage Weil  wrote:
>
> Hi Swami
>
> The limit is somewhat arbitrary, based on cluster sizes we had seen when
> we picked it.  In your case it should be perfectly safe to increase it.
>
> sage
>
>
> On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:
>
> > Hello -  Are the any limits for mon_data_size for cluster with 2PB
> > (with 2000+ OSDs)?
> >
> > Currently it set as 15G. What is logic behind this? Can we increase
> > when we get the mon_data_size_warn messages?
> >
> > I am getting the mon_data_size_warn message even though there a ample
> > of free space on the disk (around 300G free disk)
> >
> > Earlier thread on the same discusion:
> > https://www.spinics.net/lists/ceph-users/msg42456.html
> >
> > Thanks
> > Swami
> >
> >
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


Re: [ceph-users] ceph mon_data_size_warn limits for large cluster

2019-02-06 Thread Sage Weil
Hi Swami

The limit is somewhat arbitrary, based on cluster sizes we had seen when 
we picked it.  In your case it should be perfectly safe to increase it.

sage


On Wed, 6 Feb 2019, M Ranga Swami Reddy wrote:

> Hello -  Are the any limits for mon_data_size for cluster with 2PB
> (with 2000+ OSDs)?
> 
> Currently it set as 15G. What is logic behind this? Can we increase
> when we get the mon_data_size_warn messages?
> 
> I am getting the mon_data_size_warn message even though there a ample
> of free space on the disk (around 300G free disk)
> 
> Earlier thread on the same discusion:
> https://www.spinics.net/lists/ceph-users/msg42456.html
> 
> Thanks
> Swami
> 
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com