Re: [ceph-users] Cephfs MDS slow requests

On Wed, Mar 14, 2018 at 3:17 AM, David C wrote: > Hi All > > I have a Samba server that is exporting directories from a Cephfs Kernel > mount. Performance has been pretty good for the last year but users have > recently been complaining of short "freezes", these seem to

Re: [ceph-users] Fwd: Slow requests troubleshooting in Luminous - details missing

On Mon, Mar 12, 2018 at 12:21 PM, Alex Gorbachev wrote: > On Mon, Mar 12, 2018 at 7:53 AM, Дробышевский, Владимир > wrote: >> >> I was following this conversation on tracker and got the same question. I've >> got a situation with slow requests and had

Re: [ceph-users] Cephfs MDS slow requests

On Friday, March 16, 2018 at 00:07, Deepak Naidu wrote: > cephFS is not great for small files(in KB’s) but works great with large file > sizes(MB or GB’s). So using like filer(NFS/SMB) use-case needs administration > attention. > Got to disagree with you there. CephFS (Luminous) performs

Re: [ceph-users] Disk write cache - safe?

After reading Reeds comments about losing power to his data center, I think he brings up a lot of good points. So take Dells advice I linked into consideration with your own environment We also have 8TB disks with Intel P3700 for journal Our large ups and new generators which are tested

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On 03/15/2018 02:32 PM, Maxim Patlasov wrote: > On Thu, Mar 15, 2018 at 12:48 AM, Mike Christie > wrote: > > ... > > It looks like there is a bug. > > 1. A regression was added when I stopped killing the iscsi connection > when

Re: [ceph-users] Backfilling on Luminous

Did you use perf top or iotop to try to identify where the osd is stuck? Did you try increasing the op thread suicide timeout from 180s? Splitting should log at the beginning and end of an op, so it should be clear if it's taking longer than the timeout. .. Dan On Mar 15, 2018 9:23 PM, "David

Re: [ceph-users] Disk write cache - safe?

Hi We're using SUSE Ent Storage - Ceph And have Dell 730xd and expansion trays with 8 tB disks We initially had the controller cache turned off as per ceph documentation (so configured as jboss in Dell Bios) We reconfigured as raid0 and use the cache now for both internal and expansion

Re: [ceph-users] Cephfs MDS slow requests

David, few inputs based on my working experience on cephFS. Might or might not be relevant to the current issue seen in your cluster. 1. Create Metadata pool on NVMe. Folks can claim not needed, but I have seen worst perf when on HDD though the Metadata size is very small. 2. In cephFS,

Re: [ceph-users] Backfilling on Luminous

I am aware of the filestore splitting happening. I manually split all of the subfolders a couple weeks ago on this cluster, but every time we have backfilling the newly moved PGs have a chance to split before the backfilling is done. When that has happened in the past it causes some blocked

Re: [ceph-users] Backfilling on Luminous

Hi, Do you see any split or merge messages in the osd logs? I recall some surprise filestore splitting on a few osds after the luminous upgrade. .. Dan On Mar 15, 2018 6:04 PM, "David Turner" wrote: I upgraded a [1] cluster from Jewel 10.2.7 to Luminous 12.2.2 and

Re: [ceph-users] Backfilling on Luminous

We haven't used jemalloc for anything. The only thing in our /etc/sysconfig/ceph configuration is increasing TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES. I didn't see anything in dmesg on one of the recent hosts that had an osd segfault. I looked at your ticket and that looks like something with PGs

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On Thu, Mar 15, 2018 at 12:48 AM, Mike Christie wrote: > ... > > It looks like there is a bug. > > 1. A regression was added when I stopped killing the iscsi connection > when the lock is taken away from us to handle a failback bug where it > was causing ping ponging. That

Re: [ceph-users] Disk write cache - safe?

I had a recent battle with performance on two of our nodes and it turned out to be a result of using non-raid mode. We ended up rebuilding them one by one in raid-0 with controller cache enabled on the OSD disks. I discussed it on the mailing list:

Re: [ceph-users] Disk write cache - safe?

Thank you Christian, David, and Reed for your responses. My servers have the Dell H730 RAID controller in them, but I have the OSD disks in Non-RAID mode. When initially testing I compared single RAID-0 containers with Non-RAID and the Non-RAID performance was acceptable, so I opted for the

Re: [ceph-users] Backfilling on Luminous

Hi David, Am 15.03.18 um 18:03 schrieb David Turner: > I upgraded a [1] cluster from Jewel 10.2.7 to Luminous 12.2.2 and last > week I added 2 nodes to the cluster.  The backfilling has been > ATROCIOUS.  I have OSDs consistently [2] segfaulting during recovery.  > There's no pattern of which

[ceph-users] seeking maintainer for ceph-deploy (was Re: ceph-deploy's currentstatus)

Adding ceph-users to get a bit broader distribution. tl;dr: is anyone interested in stepping up to help maintain ceph-deploy? Thanks! sage On Fri, 9 Mar 2018, Alfredo Deza wrote: > Since about October of 2015, ceph-deploy has gone without a dedicated lead > doing full time development on it.

Re: [ceph-users] Backfilling on Luminous

Hi David, for me something similar happened when I've upgraded from jewel to luminous and I have discovered that the problem is the memory allocator. I've tried to change to JEMAlloc in jewel to improve performance, and when upgraded to bluestore in luminous my osds started to crash. I;ve

Re: [ceph-users] Luminous | PG split causing slow requests

The settings don't completely disable it, they push it back so that it won't happen for a very long time. I based doing my offline splitting every month because after 5-6 weeks I found that our PGs started splitting on their own with the increased settings... so I schedule a task to split them

[ceph-users] Backfilling on Luminous

I upgraded a [1] cluster from Jewel 10.2.7 to Luminous 12.2.2 and last week I added 2 nodes to the cluster. The backfilling has been ATROCIOUS. I have OSDs consistently [2] segfaulting during recovery. There's no pattern of which OSDs are segfaulting, which hosts have segfaulting OSDs, etc...

[ceph-users] Crush Bucket move crashes mons

Hi All, Having some interesting challenges. I am trying to move 2 new nodes + 2 new racks into my default root, I have added them to the cluster outside of the Root=default. They are all in and up - happy it seems. The new nodes have all 12 OSDs in them and they are all 'UP' So when going to

Re: [ceph-users] rctime not tracking inode ctime

On Wed, Mar 14, 2018 at 11:43 PM, Patrick Donnelly wrote: > On Wed, Mar 14, 2018 at 9:22 AM, Dan van der Ster wrote: >> Hi all, >> >> On our luminous v12.2.4 ceph-fuse clients / mds the rctime is not >> tracking the latest inode ctime, but only the

Re: [ceph-users] Issue with fstrim and Nova hw_disk_discard=unmap

OK, last suggestion just to narrow the issue down: ensure you have a functional admin socket and librbd log file as documented here [1]. With the VM running, before you execute "fstrim", run "ceph --admin-daemon /path/to/the/asok/file conf set debug_rbd 20" on the hypervisor host, execute "fstrim"

Re: [ceph-users] Issue with fstrim and Nova hw_disk_discard=unmap

Hallo Jason, I am really thankful for your time! Changed the volume features: rbd image 'volume-80838a69-e544-47eb-b981-a4786be89736': . features: layering, exclusive-lock, deep-flatten I had to create several dummy files before seeing and increase with "rbd du": to me, this is

Re: [ceph-users] Object Gateway - Server Side Encryption

On 14/03/18 12:31, Amardeep Singh wrote: > Though I have now another issue because I am using Multisite setup > with one zone for data and second zone for metadata with elastic > search tier. > > http://docs.ceph.com/docs/master/radosgw/elastic-sync-module/ > > When document is encrypted the

Re: [ceph-users] Instrument librbd+qemu IO from hypervisor

Self-follow-up: The ceph version is 0.80.11 in the cluster I'm working. So quite old. Adding: admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok log file = /var/log/ceph/ to /etc/ceph.conf, and then in my case tweaking apparmor (for test disabling it): service apparmor

[ceph-users] Instrument librbd+qemu IO from hypervisor

Dear fellow cephalopods, does anyone have any pointers on how to instrument librbd as-driven-by qemu IO performance from a hypervisor? Are there less intrusive ways than perf or equivalent? Can librbd be told to dump statistics somewhere (per volume) - clientside? This would come in real handy

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On 03/14/2018 04:28 PM, Maxim Patlasov wrote: > On Wed, Mar 14, 2018 at 12:05 PM, Michael Christie > wrote: > > On 03/14/2018 01:27 PM, Michael Christie wrote: > > On 03/14/2018 01:24 PM, Maxim Patlasov wrote: > >> On Wed,

Re: [ceph-users] Problem with UID starting with underscores

Hi, I have the same issue. Try to use two underscores: radosgw-admin user info --uid=“__pro_" I have user with two underscores on hammer and i can work with him with one underscore:) I recommend you remove this user and not use underscore in user names and access_keys because after upgrade

Re: [ceph-users] Disk write cache - safe?

Hello, what has been said by others before is essentially true, as in if you want: - as much data conservation as possible and have - RAID controllers with decent amounts of cache and a BBU then disabling the on disk cache is the way to go. But as you found out, w/o those caches and a