[ceph-users] Rados bench behaves oddly

We have a CEPH storage cluster which is having problems. When I run a rados bench I get the behaviour below. Has anyone seen this sort of thing before? # rados bench -p scbench 10 seq hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0

Re: [ceph-users] Failed to encode map errors

rtin Verges > Managing director > > Mobile: +49 174 9335695 > E-Mail: martin.ver...@croit.io > Chat: https://t.me/MartinVerges > > croit GmbH, Freseniusstr. 31h, 81247 Munich > CEO: Martin Verges - VAT-ID: DE310638492 > Com. register: Amtsgericht Munich HRB 231263

[ceph-users] Failed to encode map errors

And me again for the second time in one day. ceph -w is now showing messages like this: 2019-12-03 15:17:22.426988 osd.6 [WRN] failed to encode map e28961 with expected crc Any advice please? -- *Kheiron Medical Technologies* kheironmed.com | supporting

Re: [ceph-users] Osd auth del

Thankyou. ceph auth add did work I did try ceph auth get-or-create this does not read from an input file - it will generate a new key. On Tue, 3 Dec 2019 at 13:50, Willem Jan Withagen wrote: > On 3-12-2019 11:43, Wido den Hollander wrote: > > > > > > On 12/3/19

[ceph-users] Osd auth del

I had a fat fingered moment yesterday I typed ceph auth del osd.3 Where osd.3 is an otherwise healthy little osd I have not set noout or down on osd.3 yet This is a Nautilus cluster. ceph health reports everything is OK However ceph tell osd.* version hangs when it

[ceph-users] Erasure coded pools on Ambedded - advice please

I am setting up a storage cluster on Ambedded ARM hardware, which is nice! I find that I can set up an erasure coded pool with the default k=2,m=1 The cluster has 9x OSD with HDD and 12xOSD with SSD If I configure another erasure profile such as k=7 m=2 then the pool creates, but the pgs stick

[ceph-users] Cloudstack and CEPH Day London

I will be attending the Cloudstack and CEPH Day in London today. Please say hello - rotund Scottish guy, not much hair. Glaswegian accent! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iostat and dashboard freezing

Try running gstack on the ceph mgr process when it is frozen? This could be a name resolution problem, as you suspect. Maybe gstack will show where the process is 'stuck'and this might be a call to your name resolution service. On Tue, 27 Aug 2019 at 14:25, Jake Grimmett wrote: > Whoops, I'm

Re: [ceph-users] Ubuntu 19.04

You can compile from source :-) I Can't comment on the compatibility of the packages between 18.04 and 19.04, sorry. On Sat, 6 Jul 2019 at 15:44, Ashley Merrick wrote: > Hello, > > Looking at the possibility of upgrading my personal storage cluster from > Ubuntu 18.04 -> 19.04 to benefit from a

Re: [ceph-users] Massive TCP connection on radosgw

I found similar behaviour on a Nautilus cluster on Friday. Around 300 000 open connections which I think were the result of a benchmarking run which was terminated. I restarted the radosgw service to get rid of them. On Mon, 20 May 2019 at 06:56, Li Wang wrote: > Dear ceph community members, >

Re: [ceph-users] Nautilus upgrade but older releases reported by features

Sure # ceph versions { "mon": { "ceph version 14.2.0 (3a54b2b6d167d4a2a19e003a705696d4fe619afc) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.0 (3a54b2b6d167d4a2a19e003a705696d4fe619afc) nautilus (stable)": 2 }, "osd": { "ceph version 14.2.0

[ceph-users] Nautilus upgrade but older releases reported by features

We recently updated a cluster to the Nautlius release by updating Debian packages from the Ceph site. Then rebooted all servers. ceph features still reports older releases, for example the osd "osd": [ { "features": "0x3ffddff8ffac", "release": "luminous",

Re: [ceph-users] v14.2.0 Nautilus released

Martin, my thanks to Croit for making this repository available. I have been building Ceph from source on Ubuntu Cosmic for the last few days. It is much more convenient to use a repo. On Thu, 21 Mar 2019 at 09:32, Martin Verges wrote: > Hello, > > we strongly believe it would be good for Ceph

Re: [ceph-users] Ceph Nautilus for Ubuntu Cosmic?

't have them ;) > > > > > -Original Message- > From: John Hearns > Sent: 18 March 2019 17:00 > To: ceph-users > Subject: [ceph-users] Ceph Nautilus for Ubuntu Cosmic? > > May I ask if there is a repository for the latest Ceph Nautilus for > Ubuntu? > Speci

[ceph-users] Ceph Nautilus for Ubuntu Cosmic?

May I ask if there is a repository for the latest Ceph Nautilus for Ubuntu? Specifically Ubuntu 18.10 Cosmic Cuttlefish. Perhaps I am payig a penalty for living on the bleeding edge. But one does have to have some excitement in life. Thanks ___

Re: [ceph-users] ceph migration

We did a similar upgrade on a test system yesterday, from mimic to nautilus. All of the PGSstayed offlien till we issued this command: ceph osd require-osd-release nautlius --yes-i-really-mean-it} On Wed, 27 Feb 2019 at 12:19, Zhenshi Zhou wrote: > Hi, > > The servers have moved to the new

Re: [ceph-users] Diskprediction - smart returns

To answer my own question version 7.0 of the smartmontools package is needed. This has the --json flag See: http://debian.2.n7.nabble.com/Bug-918535-smartmontools-New-upstream-release-7-0-td4447595.html On Wed, 27 Feb 2019 at 11:09, John Hearns wrote: > I am looking at the diskprediction hea

[ceph-users] Diskprediction - smart returns

ot;/dev/sdb", "error": "smartctl returned invalid JSON" } I am guessing a more up to date smartmontools is needed? John Hearns ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

This is a general question for the ceph list. Should Jesper be looking at these vm tunables? vm.dirty_ratio vm.dirty_centisecs What effect do they have when using Cephfs? On Sun, 14 Oct 2018 at 14:24, John Hearns wrote: > Hej Jesper. > Sorry I do not have a direct answer to your qu

Re: [ceph-users] cephfs kernel client - page cache being invaildated.

Hej Jesper. Sorry I do not have a direct answer to your question. When looking at memory usage, I often use this command: watch cat /rpoc/meminfo On Sun, 14 Oct 2018 at 13:22, wrote: > Hi > > We have a dataset of ~300 GB on CephFS which as being used for computations > over and over agian

Re: [ceph-users] SAN or DAS for Production ceph

James, you also use the words enterprise and production ready. Is Redhat support important to you? On Tue, 28 Aug 2018 at 23:56, John Hearns wrote: > James, well for a start don't use a SAN. I speak as someone who managed a > SAN with Brocade switches and multipathing for an F1 team

Re: [ceph-users] SAN or DAS for Production ceph

James, well for a start don't use a SAN. I speak as someone who managed a SAN with Brocade switches and multipathing for an F1 team. CEPH is Software Defined Storage. You want discreet storage servers with a high bandwidth Ethernet (or maybe Infiniband) fabric. Fibrechannel still has it place

Re: [ceph-users] Design a PetaByte scale CEPH object storage

James, I would recommend that you do the following a) write out a clear set of requirements and use cases for this system. Do not mention any specific technology b) plan to install and test a small ProofOfConcept system. You can then assess if it meets the requirement in (a) On Mon, 27 Aug 2018

Re: [ceph-users] Design a PetaByte scale CEPH object storage

there should be a way to do this using a high bandwidth connection - 40 or 100Gbps Ethernet. I would look at basing your solution around a modern high bandwidth network anyway. Are you based in the UK? If so, we should talk off list. John Hearns On Mon, 27 Aug 2018 at 03:39, Christian Balzer wrote

Re: [ceph-users] active directory integration with cephfs

NFS Ganesha certainly works with Cephfs. I would investigate that also. http://docs.ceph.com/docs/master/cephfs/nfs/ Regarding Active Directory, I have done a lot of work recently with sssd. Not entirely relevant to this list, please send me a mail offline. Not sure if this is any direct use

Re: [ceph-users] iSCSI to a Ceph node with 2 network adapters - how to ?

3) Check multipath configuration > > > On 18-06-01 05:08 +0200, Marc Roos wrote: > >> >> >> Indeed, you have to add routes and rules to routing table. Just bond >> them. >> >> >> -Original Message- >> From: John Hearns [mailto

Re: [ceph-users] iSCSI to a Ceph node with 2 network adapters - how to ?

Errr is this very wise ? I have both its Ethernets connected to the same LAN, with different IPs in the same subnet (like, 192.168.200.230/24 and 192.168.200.231/24) In my experience setting up to interfaces on the same subnet means that your ssystem doesnt know which one

Re: [ceph-users] multi site with cephfs

tency between them. > > Leon > > On Wed, May 16, 2018, 16:13 John Hearns <hear...@googlemail.com> wrote: > >> Leon, >> I was at a Lenovo/SuSE seminar yesterday and asked a similar question >> regarding separated sites. >> How far apart are these two geographical

Re: [ceph-users] multi site with cephfs

Leon, I was at a Lenovo/SuSE seminar yesterday and asked a similar question regarding separated sites. How far apart are these two geographical locations? It does matter. On 16 May 2018 at 15:07, Up Safe wrote: > Hi, > > I'm trying to build a multi site setup. > But the

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

Blair, methinks someone is doing bitcoin mining on your systems when they are idle :-) I WAS going to say that maybe the cpupower utility needs an update to cope with that generation of CPUs. But 7proc/cpuinfo never lies (does it ?) On 16 May 2018 at 13:22, Blair Bethwaite

Re: [ceph-users] ceph's UID/GID 65045 in conflict with user's UID/GID in a ldap

Hello Yoann. I am working with similar issues at the moment in a biotech company in Denmark. First of all what authentication setup are you using? If you are using sssd there is a very simple and useful utility called sss_override You can óverride' the uid which you get from LDAP with the genuine

Re: [ceph-users] Intel Xeon Scalable and CPU frequency scaling on NVMe/SSD Ceph OSDs

Wido, I am going to put my rather large foot in it here. I am sure it is understood that the Turbo mode will not keep all cores at the maximum frequency at any given time. There is a thermal envelope for the chip, and the chip works to keep the power dissipation within that envelope. >From what I

Re: [ceph-users] CentOS release 7.4.1708 and selinux-policy-base >= 3.13.1-166.el7_4.9

Anton if you still cannot install the ceph RPMs, becuse of that dependency, do as Ruben suggests - install selinux-policy-targeted Then you use the RPM option --nodeps which will ignore the dependency requirements. Do not be afraid to use this option - and do not use it blindly either.

Re: [ceph-users] Please help me get rid of Slow / blocked requests

>Sounds like one of the following could be happening: > 1) RBD write caching doing the 37K IOPS, which will need to flush at some point which causes the drop. I am not sure this will help Shantur. But you could try running 'watch cat /proc/meminfo' during a benchmark run. You might be able to

Re: [ceph-users] Cluster degraded after Ceph Upgrade 12.2.1 => 12.2.2

Ronny, talking about reboots, has anyone had experience of live kernel patching with CEPH? I am asking out of simple curiosity. On 25 April 2018 at 19:40, Ronny Aasen wrote: > the difference in cost between 2 and 3 servers are not HUGE. but the > reliability

Re: [ceph-users] Bluestore caching, flawed by design?

Christian, you mention single socket systems for storage servers. I often thought that the Xeon-D would be ideal as a building block for storage servers https://www.intel.com/content/www/us/en/products/processors/xeon/d-processors.html Low power, and a complete System-On-Chip with 10gig Ethernet.

Re: [ceph-users] Bluestore caching, flawed by design?

> A long time ago I was responsible for validating the performance of CXFS on an SGI Altix UV distributed shared-memory supercomputer. As it turns out, we could achieve about 22GB/s writes with XFS (a huge >number at the time), but CXFS was 5-10x slower. A big part of that turned out to be the