Re: [ceph-users] OSD log being spammed with BlueStore stupidallocator dump

2018-10-16 Thread Wido den Hollander
On 10/16/18 11:32 AM, Igor Fedotov wrote: > > > On 10/16/2018 6:57 AM, Wido den Hollander wrote: >> >> On 10/16/2018 12:04 AM, Igor Fedotov wrote: >>> On 10/15/2018 11:47 PM, Wido den Hollander wrote: Hi, On 10/15/2018 10:43 PM, Igor Fedotov wrote: > Hi Wido, > > once

Re: [ceph-users] OSD log being spammed with BlueStore stupidallocator dump

2018-10-16 Thread Igor Fedotov
On 10/16/2018 6:57 AM, Wido den Hollander wrote: On 10/16/2018 12:04 AM, Igor Fedotov wrote: On 10/15/2018 11:47 PM, Wido den Hollander wrote: Hi, On 10/15/2018 10:43 PM, Igor Fedotov wrote: Hi Wido, once you apply the PR you'll probably see the initial error in the log that triggers the

Re: [ceph-users] OSD log being spammed with BlueStore stupidallocator dump

2018-10-15 Thread Wido den Hollander
On 10/16/2018 12:04 AM, Igor Fedotov wrote: > > On 10/15/2018 11:47 PM, Wido den Hollander wrote: >> Hi, >> >> On 10/15/2018 10:43 PM, Igor Fedotov wrote: >>> Hi Wido, >>> >>> once you apply the PR you'll probably see the initial error in the log >>> that triggers the dump. Which is most probabl

Re: [ceph-users] OSD log being spammed with BlueStore stupidallocator dump

2018-10-15 Thread Igor Fedotov
On 10/15/2018 11:47 PM, Wido den Hollander wrote: Hi, On 10/15/2018 10:43 PM, Igor Fedotov wrote: Hi Wido, once you apply the PR you'll probably see the initial error in the log that triggers the dump. Which is most probably the lack of space reported by _balance_bluefs_freespace() function.

Re: [ceph-users] OSD log being spammed with BlueStore stupidallocator dump

2018-10-15 Thread Wido den Hollander
Hi, On 10/15/2018 10:43 PM, Igor Fedotov wrote: > Hi Wido, > > once you apply the PR you'll probably see the initial error in the log > that triggers the dump. Which is most probably the lack of space > reported by _balance_bluefs_freespace() function. If so this means that > BlueFS rebalance is

Re: [ceph-users] OSD log being spammed with BlueStore stupidallocator dump

2018-10-15 Thread Igor Fedotov
Hi Wido, once you apply the PR you'll probably see the initial error in the log that triggers the dump. Which is most probably the lack of space reported by _balance_bluefs_freespace() function. If so this means that BlueFS rebalance is unable to allocate contiguous 1M chunk at main device to

Re: [ceph-users] OSD log being spammed with BlueStore stupidallocator dump

2018-10-15 Thread Wido den Hollander
On 10/15/2018 08:23 PM, Gregory Farnum wrote: > I don't know anything about the BlueStore code, but given the snippets > you've posted this appears to be a debug thing that doesn't expect to be > invoked (or perhaps only in an unexpected case that it's trying hard to > recover from). Have you che

Re: [ceph-users] OSD log being spammed with BlueStore stupidallocator dump

2018-10-15 Thread Gregory Farnum
I don't know anything about the BlueStore code, but given the snippets you've posted this appears to be a debug thing that doesn't expect to be invoked (or perhaps only in an unexpected case that it's trying hard to recover from). Have you checked where the dump() function is invoked from? I'd imag

Re: [ceph-users] OSD log being spammed with BlueStore stupidallocator dump

2018-10-15 Thread Wido den Hollander
On 10/11/2018 12:08 AM, Wido den Hollander wrote: > Hi, > > On a Luminous cluster running a mix of 12.2.4, 12.2.5 and 12.2.8 I'm > seeing OSDs writing heavily to their logfiles spitting out these lines: > > > 2018-10-10 21:52:04.019037 7f90c2f0f700 0 stupidalloc 0x0x55828ae047d0 > dump 0x15

Re: [ceph-users] OSD log being spammed with BlueStore stupidallocator dump

2018-10-11 Thread Wido den Hollander
On 10/11/2018 03:12 AM, David Turner wrote: > Not a resolution, but an idea that you've probably thought of.  > Disabling logging on any affected OSDs (possibly just all of them) seems > like a needed step to be able to keep working with this cluster to > finish the upgrade and get it healthier.

Re: [ceph-users] OSD log being spammed with BlueStore stupidallocator dump

2018-10-10 Thread David Turner
Not a resolution, but an idea that you've probably thought of. Disabling logging on any affected OSDs (possibly just all of them) seems like a needed step to be able to keep working with this cluster to finish the upgrade and get it healthier. On Wed, Oct 10, 2018 at 6:37 PM Wido den Hollander w

Re: [ceph-users] OSD log being spammed with BlueStore stupidallocator dump

2018-10-10 Thread Wido den Hollander
On 10/11/2018 12:08 AM, Wido den Hollander wrote: > Hi, > > On a Luminous cluster running a mix of 12.2.4, 12.2.5 and 12.2.8 I'm > seeing OSDs writing heavily to their logfiles spitting out these lines: > > > 2018-10-10 21:52:04.019037 7f90c2f0f700 0 stupidalloc 0x0x55828ae047d0 > dump 0x15

[ceph-users] OSD log being spammed with BlueStore stupidallocator dump

2018-10-10 Thread Wido den Hollander
Hi, On a Luminous cluster running a mix of 12.2.4, 12.2.5 and 12.2.8 I'm seeing OSDs writing heavily to their logfiles spitting out these lines: 2018-10-10 21:52:04.019037 7f90c2f0f700 0 stupidalloc 0x0x55828ae047d0 dump 0x15cd2078000~34000 2018-10-10 21:52:04.019038 7f90c2f0f700 0 stupidallo