[ceph-users] IRQ balancing, distribution

2014-09-22 Thread Christian Balzer
Hello, not really specific to Ceph, but since one of the default questions by the Ceph team when people are facing performance problems seems to be Have you tried turning it off and on again? ^o^ err, Are all your interrupts on one CPU? I'm going to wax on about this for a bit and hope for

Re: [ceph-users] IRQ balancing, distribution

2014-09-22 Thread Stijn De Weirdt
hi christian, we once were debugging some performance isssues, and IRQ balancing was one of the issues we looked in, but no real benefit there for us. all interrupts on one cpu is only an issue if the hardware itself is not the bottleneck. we were running some default SAS HBA (Dell H200), and

Re: [ceph-users] IRQ balancing, distribution

2014-09-22 Thread Christian Balzer
Hello, On Mon, 22 Sep 2014 09:35:10 +0200 Stijn De Weirdt wrote: hi christian, we once were debugging some performance isssues, and IRQ balancing was one of the issues we looked in, but no real benefit there for us. all interrupts on one cpu is only an issue if the hardware itself is not

Re: [ceph-users] IRQ balancing, distribution

2014-09-22 Thread Florian Haas
On Mon, Sep 22, 2014 at 10:21 AM, Christian Balzer ch...@gol.com wrote: The linux scheduler usually is quite decent in keeping processes where the action is, thus you see for example a clear preference of DRBD or KVM vnet processes to be near or on the CPU(s) where the IRQs are. Since you're

Re: [ceph-users] IRQ balancing, distribution

2014-09-22 Thread Stijn De Weirdt
but another issue is the OSD processes: do you pin those as well? and how much data do they actually handle. to checksum, the OSD process needs all data, so that can also cause a lot of NUMA traffic, esp if they are not pinned. That's why all my (production) storage nodes have only a single 6

Re: [ceph-users] IRQ balancing, distribution

2014-09-22 Thread Anand Bhat
as issued, lots of cycles are lost for cache coherency. Regards, Anand -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Stijn De Weirdt Sent: Monday, September 22, 2014 2:36 PM To: ceph-users@lists.ceph.com Subject: Re: [ceph-users] IRQ

Re: [ceph-users] IRQ balancing, distribution

2014-09-22 Thread Stijn De Weirdt
] IRQ balancing, distribution but another issue is the OSD processes: do you pin those as well? and how much data do they actually handle. to checksum, the OSD process needs all data, so that can also cause a lot of NUMA traffic, esp if they are not pinned. That's why all my (production) storage

Re: [ceph-users] IRQ balancing, distribution

2014-09-22 Thread Mark Nelson
On 09/22/2014 01:55 AM, Christian Balzer wrote: Hello, not really specific to Ceph, but since one of the default questions by the Ceph team when people are facing performance problems seems to be Have you tried turning it off and on again? ^o^ err, Are all your interrupts on one CPU? I'm going

Re: [ceph-users] IRQ balancing, distribution

2014-09-22 Thread Christian Balzer
Hello, On Mon, 22 Sep 2014 08:55:48 -0500 Mark Nelson wrote: On 09/22/2014 01:55 AM, Christian Balzer wrote: Hello, not really specific to Ceph, but since one of the default questions by the Ceph team when people are facing performance problems seems to be Have you tried turning it