In the end I solved it by restarting cluster target with systemd, I guess
something was stuck.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
>> With today’s networking, _maybe_ a super-dense NVMe box needs 100Gb/s where
>> a less-dense probably is fine with 25Gb/s. And of course PCI lanes.
>>
>>
On 23/09/2020 17:58, vita...@yourcmc.ru wrote:
I have no idea how you get 66k write iops with one OSD )
I've just repeated a test by creating a test pool on one NVMe OSD with 8 PGs
(all pinned to the same OSD with pg-upmap). Then I ran 4x fio randwrite q128
over 4 RBD images. I got 17k
On 9/23/20 2:21 PM, Alexander E. Patrakov wrote:
On Wed, Sep 23, 2020 at 8:12 PM Anthony D'Atri wrote:
With today’s networking, _maybe_ a super-dense NVMe box needs 100Gb/s where a
less-dense probably is fine with 25Gb/s. And of course PCI lanes.
On Wed, Sep 23, 2020 at 8:12 PM Anthony D'Atri wrote:
> With today’s networking, _maybe_ a super-dense NVMe box needs 100Gb/s where a
> less-dense probably is fine with 25Gb/s. And of course PCI lanes.
>
>
Thanks for the feedback everyone! It seems we have more to look into regarding
NVMe enterprise storage solutions. The workload doesn’t demand NVMe
performance, so SSD seems to be the most cost effective way to handle this.
The performance discussion is very interesting!
Regards,
Brent
Hi
Thanks for the reply.
cephadm runs ceph containers automatically. How to set privileged mode
in ceph container?
--
El 23/9/20 a las 13:24, Daniel Gryniewicz escribió:
NFSv3 needs privileges to connect to the portmapper. Try running
your docker container in privileged mode, and see if
On 9/23/20 12:18 PM, Mark Nelson wrote:
On 9/23/20 10:58 AM, vita...@yourcmc.ru wrote:
I have no idea how you get 66k write iops with one OSD )
I've just repeated a test by creating a test pool on one NVMe OSD
with 8 PGs (all pinned to the same OSD with pg-upmap). Then I ran 4x
fio randwrite
On 9/23/20 10:58 AM, vita...@yourcmc.ru wrote:
I have no idea how you get 66k write iops with one OSD )
I've just repeated a test by creating a test pool on one NVMe OSD with 8 PGs
(all pinned to the same OSD with pg-upmap). Then I ran 4x fio randwrite q128
over 4 RBD images. I got 17k iops.
Hi Michael,
yes, you can use ceph-bluestore-tool to do that. E.g.
bin/ceph-bluestore-tool --path dev/osd0 --devs-source dev/osd0/block.wal
--dev-target dev/osd0/block.db --command bluefs-bdev-migrate
inferring bluefs devices from bluestore path
device removed:0 dev/osd0/block.wal
Hi Eugen,
Am 23.09.2020 um 14:51 schrieb Eugen Block:
I don't think there's a way to remove WAL/DB without rebuilding the OSD.
ceph-bluestore-tool bluefs-bdev-migrate expects a target device to
migrate the data since it's a migration. I can't read the full thread (I
get a server error), what
I have no idea how you get 66k write iops with one OSD )
I've just repeated a test by creating a test pool on one NVMe OSD with 8 PGs
(all pinned to the same OSD with pg-upmap). Then I ran 4x fio randwrite q128
over 4 RBD images. I got 17k iops.
OK, in fact that's not the worst result for
I don't think you need a bucket under host for the two LVs. It's unnecessary.
September 23, 2020 6:45 AM, "George Shuklin" wrote:
> On 23/09/2020 10:54, Marc Roos wrote:
>
>>> Depends on your expected load not? I already read here numerous of times
>> that osd's can not keep up with nvme's,
Apologies for not consolidating these replys. My UMA is not my friend today.
> With 10 NVMe drives per node, I'm guessing that a single EPYC 7451 is
> going to be CPU bound for small IO workloads (2.4c/4.8t per OSD), but
> will be network bound for large IO workloads unless you are sticking
>
Hi,
I have a CEPH 15.2.5 running in a docker , i configure nfs ganesha with
nfs version 3 but i can not mount it.
If configure ganesha with nfs version 4 i can mounted without problems
but i need the version 3 .
The error is mount.nfs: Protocol not supported
Can help me?
Thanks.
--
> How they did it?
You can create partitions / LVs by hand and build OSDs on them, or you can use
ceph-volume lvm batch –osds-per-device
> I have an idea to create a new bucket type under host, and put two LV from
> each ceph osd VG into that new bucket. Rules are the same (different host),
Hi Stefan,
thanks for your answer. I think the deprecated option is still supported and I
found something else - I will update to the new option though. On the ceph
side, I see in the log now:
client session with non-allowable root '/' denied (client.31382084
192.168.48.135:0/2576875769)
Hello,
it seems that ceph-volume-systemd makes this confusion by missing
python3.6 stuff here:
[root@ceph1n011 system]# /usr/sbin/ceph-volume-systemd
Traceback (most recent call last):
File "/usr/sbin/ceph-volume-systemd", line 6, in
from pkg_resources import load_entry_point
File
On 9/23/20 8:23 AM, George Shuklin wrote:
I've just finishing doing our own benchmarking, and I can say, you
want to do something very unbalanced and CPU bounded.
1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per
ceph-osd at top-performance (see the recent thread on 'ceph
On 2020-09-23 11:00, Frank Schilder wrote:
> Dear all,
>
> maybe someone has experienced this before. We are setting up a SAMBA gateway
> and would like to use the vfs_ceph module. In case of several file systems
> one needs to choose an mds namespace. There is an option in ceph.conf:
>
>
I would put that data on the ceph.com website. Eg. A performance/test
page with every release and compared to the previous release. Some
default fio tests like you now have in the spreadsheet. And maybe some
io patterns that relate to real world use cases like databases. Like eg
how these
Hello,
does anyone tried to update 15.2.4 on Centos7 to 15.2.5?
I did a full: yum -y update on my first OSD node and after this no OSD
on this want to start anymore. No log will be written, so osd process
stops immediately i think.
Starting osd daemon in forground shows that no tmpfs will
Hi Lenz,
thanks for that, this should do. Please retain the copy until all is migrated :)
Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
From: Lenz Grimmer
Sent: 23 September 2020 10:55:13
To: ceph-users@ceph.io
I've just finishing doing our own benchmarking, and I can say, you
want to do something very unbalanced and CPU bounded.
1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per
ceph-osd at top-performance (see the recent thread on 'ceph on brd')
with more realistic numbers
Update: setting "ceph fs set-default CEPH-FS-NAME" allows to do a kernel fs
mount without providing the mds_namespace mount option, but the vfs_ceph module
still fails with either
cephwrap_connect: [CEPH] Error return: Operation not permitted
or
cephwrap_connect: [CEPH] Error return:
> https://docs.google.com/spreadsheets/d/1e5eTeHdZnSizoY6AUjH0knb4jTCW7KMU4RoryLX9EHQ/edit?usp=sharing
I see that in your tests Octopus delivers more than twice iops with 1 OSD.
Can I ask you what's my problem then? :-)
I have a 4-node Ceph cluster with 14 NVMe drives and fast CPUs
> I'm curious if you've tried octopus+ yet?
Why don't you publish results of your test cluster? You cannot expect
all new users to buy 4 servers with 40 disks, and try if the performance
is ok.
Get a basic cluster and start publishing results, and document changes
to the test cluster.
Dear all,
maybe someone has experienced this before. We are setting up a SAMBA gateway
and would like to use the vfs_ceph module. In case of several file systems one
needs to choose an mds namespace. There is an option in ceph.conf:
client mds namespace = CEPH-FS-NAME
Unfortunately, it
Hello, I got this error after I tried upgrade from ceph:v15.2.4 to ceph:v15.2.5
with dead osd.6
cluster:
id: 4e01640b-951b-4f75-8dca-0bad4faf1b11
health: HEALTH_ERR
Module 'cephadm' has failed: auth get failed: failed to find osd.6
in keyring retval: -2
In short I
Hi,
I don't think there's a way to remove WAL/DB without rebuilding the
OSD. ceph-bluestore-tool bluefs-bdev-migrate expects a target device
to migrate the data since it's a migration. I can't read the full
thread (I get a server error), what is the goal here?
Regards,
Eugen
Zitat von
On 9/23/20 5:41 AM, George Shuklin wrote:
I've just finishing doing our own benchmarking, and I can say, you
want to do something very unbalanced and CPU bounded.
1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per
ceph-osd at top-performance (see the recent thread on 'ceph
Hi Andreas,
Am 22.09.2020 um 22:35 schrieb Andreas John:
and then removing the journal
enough?
any hints on how to remove the journal?
Regards,
Michael
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to
Sounds like you just want to create 2 OSDs per drive? It's OK, everyone does
that :) I tested Ceph with 2 OSDs per SATA SSD when comparing it to my
Vitastor, Micron also tested Ceph with 2 OSDs per SSD in their PDF and so on.
> On 23/09/2020 10:54, Marc Roos wrote:
>
>>> Depends on your
On 2020-09-23 07:39, Brent Kennedy wrote:
> We currently run a SSD cluster and HDD clusters and are looking at possibly
> creating a cluster for NVMe storage. For spinners and SSDs, it seemed the
> max recommended per osd host server was 16 OSDs ( I know it depends on the
> CPUs and RAM, like 1
Working with estate agents might not be the first choice for many of us. But if
you are trying to find someone who can help you with your house for sale in
Chichester or finding a renter, the estate agent can help you to meet potential
investors and buyers. Estate agents in Chichester always
On 23/09/2020 04:09, Alexander E. Patrakov wrote:
Sometimes this doesn't help. For data recovery purposes, the most
helpful step if you get the "bluefs enospc" error is to add a separate
db device, like this:
systemctl disable --now ceph-osd@${OSDID}
truncate -s 32G
On 23/09/2020 10:54, Marc Roos wrote:
Depends on your expected load not? I already read here numerous of times
that osd's can not keep up with nvme's, that is why people put 2 osd's
on a single nvme. So on a busy node, you probably run out of cores? (But
better verify this with someone that
I've just finishing doing our own benchmarking, and I can say, you want
to do something very unbalanced and CPU bounded.
1. Ceph consume a LOT of CPU. My peak value was around 500% CPU per
ceph-osd at top-performance (see the recent thread on 'ceph on brd')
with more realistic numbers
Good morning,
you might have seen my previous mails and I wanted to discuss some
findings over the last day+night over what happened and why it happened
here.
As the system behaved inexplicitly for us, we are now looking for
someone to analyse the root cause on consultancy basis - if you are
Slow. https://yourcmc.ru/wiki/Ceph_performance :-)
> Hi,
>
> we're considering running KVM virtual machine images on Ceph RBD block
> devices. How does Ceph RBD perform with the synchronous writes of
> databases (MariaDB)?
>
> Best regards,
>
> Renne
Thanks Marc :)
It's easier to write code than to cooperate :) I can do whatever I want in my
own project.
Ceph is rather complex. For example, I failed to find bottlenecks in OSD when I
tried to profile it - I'm not an expert of course, but still... The only
bottleneck I found was
Hi
> We currently run a SSD cluster and HDD clusters and are looking at possibly
> creating a cluster for NVMe storage. For spinners and SSDs, it seemed the
> max recommended per osd host server was 16 OSDs ( I know it depends on the
> CPUs and RAM, like 1 cpu core and 2GB memory ).
What do you
I love how it’s not possible to delete inodes yet. Data loss would be a thing
of the past!
Jokes aside, interesting project.
Sent from mobile
> Op 23 sep. 2020 om 00:45 heeft vita...@yourcmc.ru het volgende geschreven:
>
> Hi!
>
> After almost a year of development in my spare time I
Hi,
this has been discussed a couple of times [1]. Changing an ec profile
won't affect existing pools, only new pools created with this updated
profile will apply the device-class.
Make sure to provide all parameters for the profile update, not just
the device-class.
Regards,
Eugen
[1]
I was wondering if switching to ceph-volume requires me to change the
default centos lvm.conf? Eg. The default has issue_discards = 0
Also I wonder if trimming is the default on lvm's on ssds? I read
somewhere that the dmcrypt passthrough of trimming was still secure in
combination with a
Hi Brent,
> 1. If we do a jbod setup, the servers can hold 48 NVMes, if the servers
> were bought with 48 cores and 100+ GB of RAM, would this make sense?
Do you seriously mean 48 NVMes per server? How would you even come remotely
close to supporting them with connection (to board) and network
Hi,
we're considering running KVM virtual machine images on Ceph RBD block
devices. How does Ceph RBD perform with the synchronous writes of
databases (MariaDB)?
Best regards,
Renne
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe
It succeeds to load with LD_PRELOAD as, as I understand, block_register() gets
called. QAPI and new QAPI-based block device syntax don't work though because
they're based on IDLs built into QEMU... QAPI will require patching, yeah. It
would be nicer if QAPI supported plugins too... :-)
--
Depends on your expected load not? I already read here numerous of times
that osd's can not keep up with nvme's, that is why people put 2 osd's
on a single nvme. So on a busy node, you probably run out of cores? (But
better verify this with someone that has an nvme cluster ;))
Vitaliy you are crazy ;) But really cool work. Why not combine efforts
with ceph? Especially with something as important as SDS and PB's of
clients data stored on it, everyone with a little bit of brain chooses a
solution from a 'reliable' source. For me it was decisive to learn that
CERN
Hi,
We are running a Nautilus cluster and have some old and new erasure
code profiles. For example:
# ceph osd erasure-code-profile get m_erasure
crush-device-class=hdd
crush-failure-domain=host
crush-root=default
jerasure-per-chunk-alignment=false
k=6
m=2
plugin=jerasure
To troubleshoot with the hp printer issue, you are required to download the HP
Print and Scan Doctor from its official website. After that, you have to run
HPPdr.exe from the download location on your system. And then, hit on the Start
link and select your printer accordingly. Now, click fix
52 matches
Mail list logo
