I tried with disk based swap on a SATA SSD.
I think that might be the last option. I have exported already all the down
PG's from the OSD that they are waiting for.
Kind Regards
Lee
On Thu, 6 Jan 2022 at 20:00, Alexander E. Patrakov
wrote:
> пт, 7 янв. 2022 г. в 00:50, Alexander E. Patrakov
пт, 7 янв. 2022 г. в 00:50, Alexander E. Patrakov :
> чт, 6 янв. 2022 г. в 12:21, Lee :
>
>> I've tried add a swap and that fails also.
>>
>
> How exactly did it fail? Did you put it on some disk, or in zram?
>
> In the past I had to help a customer who hit memory over-use when
> upgrading Ceph
чт, 6 янв. 2022 г. в 12:21, Lee :
> I've tried add a swap and that fails also.
>
How exactly did it fail? Did you put it on some disk, or in zram?
In the past I had to help a customer who hit memory over-use when upgrading
Ceph (due to shallow_fsck), and we were able to fix it by adding 64 GB
> I assume the huge memory consumption is temporary. Once the OSD is up and
> stable, it would release the memory.
>
> So how about allocate a large swap temporarily just to let the OSD up. I
> remember that someone else on the list have resolved a similar issue with
> swap.
But is this
Running your osd's with resource limitations is not so straightforward. I can
guess that if you are running close to full resource utilization on your nodes,
it makes more sense to make sure everything stays as much within their
specified limits. (Aside from the question if you would even want
Hi Lee,
could you please raise debug-bluestore and debug-osd to 20 (via ceph
tell osd.N injectargs command) when OSD starts to eat up the RAM. Then
drop it back to defaults after a few seconds (10s is enough) to avoid
huge log size and share the resulting OSD log.
Also I'm curious if you
For Example
top - 22:53:47 up 1:29, 2 users, load average: 2.23, 2.08, 1.92
Tasks: 255 total, 2 running, 253 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.2 us, 4.5 sy, 0.0 ni, 91.1 id, 0.1 wa, 0.0 hi, 0.1 si,
0.0 st
MiB Mem : 161169.7 total, 23993.9 free, 132036.5 used, 5139.3
The first OSD took 156Gb of Ram to boot.. :(
Is there a easy way to stop Mempool pulling so much memory.
On Wed, 5 Jan 2022 at 22:12, Mazzystr wrote:
> and that is exactly why I run osds containerized with limited cpu and
> memory as well as "bluestore cache size", "osd memory target", and
and that is exactly why I run osds containerized with limited cpu and
memory as well as "bluestore cache size", "osd memory target", and "mds
cache memory limit". Osd processes have become noisy neighbors in the last
few versions.
On Wed, Jan 5, 2022 at 1:47 PM Lee wrote:
> I'm not rushing,
It's nice to hear that. You can also decrease the osd ram usage from
4gb to 2gb. If you have enough spare ram go for it.
Good luck.
Lee , 6 Oca 2022 Per, 00:46 tarihinde şunu yazdı:
>
> I'm not rushing,
>
> I have found the issue, Im am getting OOM errors as the OSD boots, basically
> is starts
I'm not rushing,
I have found the issue, Im am getting OOM errors as the OSD boots,
basically is starts to process the PG's and then the node runs out of
memory and the daemon kill's
2022-01-05 20:09:08 bb-ceph-enc-rm63-osd03-31 osd.51
2022-01-05T20:09:01.024+ 7fce3c6bc700 10 osd.51 24448261
First of all, do not rush into bad decisions.
Production is down and you wanna make it online but you should fix the
problem and be sure first. If a second crash occurs in a healing state
you will lose metadata.
You don't need to debug first!
You didn't mention your cluster status and we don't
12 matches
Mail list logo
