Re: [ceph-users] All flash ceph witch NVMe and SPDK
> Op 3 augustus 2017 om 15:28 schreef Mike A : > > > Hello > > Our goal it is make fast storage as possible. > By now our configuration of 6 servers look like that: > * 2 x CPU Intel Gold 6150 20 core 2.4Ghz > * 2 x 16 Gb NVDIMM DDR4 DIMM > * 6 x 16 Gb RAM DDR4 > * 6 x Intel DC P4500 4Tb NVMe 2.5" > * 2 x Mellanox ConnectX-4 EN Lx 25Gb dualport > To get the maximum out of your NVMe you will need higher clocked CPUs. 3.5Ghz or something. However, I'm still not convinced you will get the maximum out of your NVMe with Ceph. Although you are looking into 'partitioning' your NVMe with SPDK I would look at less core which are clocked higher. Wido > What a status in ceph of RDMA, NVDIMM access using libpmem and SPDK software? > How mature this technologes in Ceph? Ready for prodaction use? > > Mike > > — > Mike, runs. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] jewel: bug? forgotten rbd files?
Hello Greg,
even after 48h the files are still there and the PG is still in:
active+clean+inconsistent+snaptrim state.
Greets,
Stefan
Am 05.08.2017 um 21:43 schrieb Gregory Farnum:
> is OSD 20 actually a member of the pg right now? It could be stray data
> that is slowly getting cleaned up.
>
> Also, you've got "snapdir" listings there. Those indicate the object is
> snapshotted but the "head" got deleted. So it may just be delayed
> cleanup of snapshots.
>
> On Sat, Aug 5, 2017 at 12:34 PM Stefan Priebe - Profihost AG
> mailto:s.pri...@profihost.ag>> wrote:
>
> Hello,
>
> today i deleted an rbd image which had the following
> prefix:
>
> block_name_prefix: rbd_data.106dd406b8b4567
>
> the rm command went fine.
>
> also the rados list command does not show any objects with that string:
> # rados -p rbd ls | grep 106dd406b8b4567
>
> But find on an osd still has them?
>
> osd.20]# find . -name "*106dd406b8b4567*" -exec ls -la "{}" \;
> -rw-r--r-- 1 ceph ceph 4194304 Aug 5 09:32
>
> ./current/3.61a_head/DIR_A/DIR_1/DIR_6/DIR_8/rbd\udata.106dd406b8b4567.2315__9d5e4_9E65861A__3
> -rw-r--r-- 1 ceph ceph 4194304 Aug 5 09:36
>
> ./current/3.61a_head/DIR_A/DIR_1/DIR_6/DIR_8/rbd\udata.106dd406b8b4567.2315__9d84a_9E65861A__3
> -rw-r--r-- 1 ceph ceph 0 Aug 5 11:47
>
> ./current/3.61a_head/DIR_A/DIR_1/DIR_6/DIR_8/rbd\udata.106dd406b8b4567.2315__snapdir_9E65861A__3
> -rw-r--r-- 1 ceph ceph 4194304 Aug 5 09:49
>
> ./current/3.61a_head/DIR_A/DIR_1/DIR_6/DIR_A/rbd\udata.106dd406b8b4567.018c__9d455_BCB2A61A__3
> -rw-r--r-- 1 ceph ceph 1400832 Aug 5 09:32
>
> ./current/3.61a_head/DIR_A/DIR_1/DIR_6/DIR_A/rbd\udata.106dd406b8b4567.018c__9d5e4_BCB2A61A__3
> -rw-r--r-- 1 ceph ceph 1400832 Aug 5 09:32
>
> ./current/3.61a_head/DIR_A/DIR_1/DIR_6/DIR_A/rbd\udata.106dd406b8b4567.018c__9d84a_BCB2A61A__3
> -rw-r--r-- 1 ceph ceph 0 Aug 5 11:47
>
> ./current/3.61a_head/DIR_A/DIR_1/DIR_6/DIR_A/rbd\udata.106dd406b8b4567.018c__snapdir_BCB2A61A__3
>
> Greets,
> Stefan
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] download.ceph.com rsync errors
Hi, The rsync target (rsync://download.ceph.com/ceph/) has been throwing the following errors for a while: rsync: send_files failed to open "/debian-hammer/db/lockfile" (in ceph): Permission denied (13) rsync: send_files failed to open "/debian-jewel/db/lockfile" (in ceph): Permission denied (13) rsync: send_files failed to open "/debian-jewel/pool/main/c/ceph/.ceph-fuse-dbg_10.1.0-1~bpo80+1_amd64.deb.h0JvHM" (in ceph): Permission denied (13) rsync: send_files failed to open "/debian-luminous/db/lockfile" (in ceph): Permission denied (13) rsync: send_files failed to open "/debian-testing/db/lockfile" (in ceph): Permission denied (13) rsync: send_files failed to open "/rpm-jewel/el7/x86_64/.ceph-10.1.0-0.el7.x86_64.rpm.2FtlL3" (in ceph): Permission denied (13) rsync: send_files failed to open "/rpm-luminous/el7/aarch64/.ceph-debuginfo-12.0.3-0.el7.aarch64.rpm.yQ0WpX" (in ceph): Permission denied (13) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1518) [generator=3.0.9] I posted on the Ceph mirror admin list, although I never received a response. Is anyone able to sort this out? Thanks, Matthew. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Pg inconsistent / export_files error -5
On Sat, Aug 5, 2017 at 1:21 AM, Marc Roos wrote: > > I have got a placement group inconsistency, and saw some manual where > you can export and import this on another osd. But I am getting an > export error on every osd. > > What does this export_files error -5 actually mean? I thought 3 copies #define EIO 5 /* I/O error */ > should be enough to secure your data. > > >> PG_DAMAGED Possible data damage: 1 pg inconsistent >>pg 17.36 is active+clean+inconsistent, acting [9,0,12] > > >> 2017-08-04 05:39:51.534489 7f2f623d6700 -1 log_channel(cluster) log > [ERR] : 17.36 soid > 17:6ca1f70a:::rbd_data.1f114174b0dc51.0974:4: failed to pick > suitable object info >> 2017-08-04 05:41:12.715393 7f2f623d6700 -1 log_channel(cluster) log > [ERR] : 17.36 deep-scrub 3 errors >> 2017-08-04 15:21:12.445799 7f2f623d6700 -1 log_channel(cluster) log > [ERR] : 17.36 soid > 17:6ca1f70a:::rbd_data.1f114174b0dc51.0974:4: failed to pick > suitable object info >> 2017-08-04 15:22:35.646635 7f2f623d6700 -1 log_channel(cluster) log > [ERR] : 17.36 repair 3 errors, 0 fixed > > ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12 --pgid 17.36 > --op export --file /tmp/recover.17.36 Can you run this command under strace like so? # strace -fvttyyTo /tmp/strace.out -s 1024 ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12 --pgid 17.36 --op export --file /tmp/recover.17.36 Then see if you can find which syscall is returning EIO. # grep "= \-5" /tmp/strace.out > > ... > Read #17:6c9f811c:::rbd_data.1b42f52ae8944a.1a32:head# > Read #17:6ca035fc:::rbd_data.1fff61238e1f29.b31a:head# > Read #17:6ca0b4f8:::rbd_data.1fff61238e1f29.6fcc:head# > Read #17:6ca0ffbc:::rbd_data.1fff61238e1f29.a214:head# > Read #17:6ca10b29:::rbd_data.1fff61238e1f29.9923:head# > Read #17:6ca11ab9:::rbd_data.1fa8ef2ae8944a.11b4:head# > Read #17:6ca13bed:::rbd_data.1f114174b0dc51.02c6:head# > Read #17:6ca1a791:::rbd_data.1fff61238e1f29.f101:head# > Read #17:6ca1f70a:::rbd_data.1f114174b0dc51.0974:4# > export_files error -5 Running the command with "--debug" appended will give more output which may shed more light as well. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Cheers, Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
