Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic
> Can't say anything about latency. >>Anybody else? Wido? I'm running it on mimic since 1 month, no problem until now, and it's definility fixing the latency increasing over time. (aka need restart osd each week) Memory usage is almost the same than before. - Mail original - De: "Konstantin Shalygin" À: "Marc Roos" , "Wido den Hollander" Cc: "ceph-users" Envoyé: Mercredi 10 Juillet 2019 05:56:35 Objet: Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic On 5/28/19 5:16 PM, Marc Roos wrote: > I switched first of may, and did not notice to much difference in memory > usage. After the restart of the osd's on the node I see the memory > consumption gradually getting back to as before. > Can't say anything about latency. Anybody else? Wido? I see many patches from Igor comes to Luminous. And also bitmap allocator (default in Nautilus) is tries to kill Brett Chancellor cluster for a week [1] [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-July/035726.html k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic
On 7/10/19 5:56 AM, Konstantin Shalygin wrote: > On 5/28/19 5:16 PM, Marc Roos wrote: >> I switched first of may, and did not notice to much difference in memory >> usage. After the restart of the osd's on the node I see the memory >> consumption gradually getting back to as before. >> Can't say anything about latency. > > > Anybody else? Wido? > I haven't tried it yet as I don't have a cluster running with Luminous I can test this on. So right now I've been sticking with Stupidallocator under Luminous. > I see many patches from Igor comes to Luminous. And also bitmap > allocator (default in Nautilus) is tries to kill Brett Chancellor > cluster for a week [1] > > > > [1] > http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-July/035726.html > > k > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic
On 5/28/19 5:16 PM, Marc Roos wrote: I switched first of may, and did not notice to much difference in memory usage. After the restart of the osd's on the node I see the memory consumption gradually getting back to as before. Can't say anything about latency. Anybody else? Wido? I see many patches from Igor comes to Luminous. And also bitmap allocator (default in Nautilus) is tries to kill Brett Chancellor cluster for a week [1] [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-July/035726.html k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic
I switched first of may, and did not notice to much difference in memory usage. After the restart of the osd's on the node I see the memory consumption gradually getting back to as before. Can't say anything about latency. -Original Message- From: Konstantin Shalygin Sent: dinsdag 28 mei 2019 11:52 To: Wido den Hollander Cc: ceph-users@lists.ceph.com Subject: *SPAM* Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic Hi, With the release of 12.2.12 the bitmap allocator for BlueStore is now available under Mimic and Luminous. [osd] bluestore_allocator = bitmap bluefs_allocator = bitmap Before setting this in production: What might the implications be and what should be thought of? >From what I've read the bitmap allocator seems to be better in read performance and uses less memory. In Nautilus bitmap is the default, but L and M still default to stupid. Since the bitmap allocator was backported there must be a use-case to use the bitmap allocator instead of stupid. Thanks! Wido Wido, do you setted allocator to bitmap on L installations past this months? Any improvements? k test-memory.png Description: Binary data ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic
Hi, With the release of 12.2.12 the bitmap allocator for BlueStore is now available under Mimic and Luminous. [osd] bluestore_allocator = bitmap bluefs_allocator = bitmap Before setting this in production: What might the implications be and what should be thought of? >/From what I've read the bitmap allocator seems to be better in read /performance and uses less memory. In Nautilus bitmap is the default, but L and M still default to stupid. Since the bitmap allocator was backported there must be a use-case to use the bitmap allocator instead of stupid. Thanks! Wido Wido, do you setted allocator to bitmap on L installations past this months? Any improvements? k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic
On Mon, 15 Apr 2019 at 19:40, Wido den Hollander wrote: > > Hi, > > With the release of 12.2.12 the bitmap allocator for BlueStore is now > available under Mimic and Luminous. > > [osd] > bluestore_allocator = bitmap > bluefs_allocator = bitmap Hi! Have you tried this? :) -- End of message. Next message? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic
On 4/15/2019 4:17 PM, Wido den Hollander wrote: On 4/15/19 2:55 PM, Igor Fedotov wrote: Hi Wido, the main driver for this backport were multiple complains on write ops latency increasing over time. E.g. see thread named: "ceph osd commit latency increase over time, until restart" here. Or http://tracker.ceph.com/issues/38738 Most symptoms showed Stupid Allocator as a root cause for that. Hence we've got a decision to backport bitmap allocator which should work a fix/workaround. I see, that makes things clear. Anything users should take into account when setting: [osd] bluestore_allocator = bitmap bluefs_allocator = bitmap Writing this here for archival purposes so that users who have the same question can find it easily. Nothing specific but a bit different memory usage pattern: stupid allocator has more dynamic memory usage approach while bitmap allocator is absolutely static in this respect. So depending on the use case OSD might require more or less RAM. E.g. on fresh deployment stupid allocator memory requirements are most probably less that bitmap allocator ones. But RAM usage for bitmap allocator doesn't change with OSD evolution while ones for stupid allocator might grow unexpectedly high. FWIW resulting disk fragmentation might be different too. The same apply to their performance but I'm not sure if the latter is visible with the full Ceph stack. Wido Thanks, Igor On 4/15/2019 3:39 PM, Wido den Hollander wrote: Hi, With the release of 12.2.12 the bitmap allocator for BlueStore is now available under Mimic and Luminous. [osd] bluestore_allocator = bitmap bluefs_allocator = bitmap Before setting this in production: What might the implications be and what should be thought of? From what I've read the bitmap allocator seems to be better in read performance and uses less memory. In Nautilus bitmap is the default, but L and M still default to stupid. Since the bitmap allocator was backported there must be a use-case to use the bitmap allocator instead of stupid. Thanks! Wido ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic
On 4/15/19 2:55 PM, Igor Fedotov wrote: > Hi Wido, > > the main driver for this backport were multiple complains on write ops > latency increasing over time. > > E.g. see thread named: "ceph osd commit latency increase over time, > until restart" here. > > Or http://tracker.ceph.com/issues/38738 > > Most symptoms showed Stupid Allocator as a root cause for that. > > Hence we've got a decision to backport bitmap allocator which should > work a fix/workaround. > I see, that makes things clear. Anything users should take into account when setting: [osd] bluestore_allocator = bitmap bluefs_allocator = bitmap Writing this here for archival purposes so that users who have the same question can find it easily. Wido > > Thanks, > > Igor > > > On 4/15/2019 3:39 PM, Wido den Hollander wrote: >> Hi, >> >> With the release of 12.2.12 the bitmap allocator for BlueStore is now >> available under Mimic and Luminous. >> >> [osd] >> bluestore_allocator = bitmap >> bluefs_allocator = bitmap >> >> Before setting this in production: What might the implications be and >> what should be thought of? >> >> From what I've read the bitmap allocator seems to be better in read >> performance and uses less memory. >> >> In Nautilus bitmap is the default, but L and M still default to stupid. >> >> Since the bitmap allocator was backported there must be a use-case to >> use the bitmap allocator instead of stupid. >> >> Thanks! >> >> Wido >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic
Hi Wido, the main driver for this backport were multiple complains on write ops latency increasing over time. E.g. see thread named: "ceph osd commit latency increase over time, until restart" here. Or http://tracker.ceph.com/issues/38738 Most symptoms showed Stupid Allocator as a root cause for that. Hence we've got a decision to backport bitmap allocator which should work a fix/workaround. Thanks, Igor On 4/15/2019 3:39 PM, Wido den Hollander wrote: Hi, With the release of 12.2.12 the bitmap allocator for BlueStore is now available under Mimic and Luminous. [osd] bluestore_allocator = bitmap bluefs_allocator = bitmap Before setting this in production: What might the implications be and what should be thought of? From what I've read the bitmap allocator seems to be better in read performance and uses less memory. In Nautilus bitmap is the default, but L and M still default to stupid. Since the bitmap allocator was backported there must be a use-case to use the bitmap allocator instead of stupid. Thanks! Wido ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
