Re: [ceph-users] Kernel mounted RBD's hanging

2017-07-31 Thread Nick Fisk
> -Original Message- > From: Ilya Dryomov [mailto:idryo...@gmail.com] > Sent: 31 July 2017 11:36 > To: Nick Fisk > Cc: Ceph Users > Subject: Re: [ceph-users] Kernel mounted RBD's hanging > > On Thu, Jul 13, 2017 at 12:54 PM, Ilya Dryomov wrote: > &

Re: [ceph-users] Kernel mounted RBD's hanging

2017-07-31 Thread Ilya Dryomov
On Thu, Jul 13, 2017 at 12:54 PM, Ilya Dryomov wrote: > On Wed, Jul 12, 2017 at 7:15 PM, Nick Fisk wrote: >>> Hi Ilya, >>> >>> I have managed today to capture the kernel logs with debugging turned on >>> and the ms+osd debug logs from the mentioned OSD. >>> However, this is from a few minutes af

Re: [ceph-users] Kernel mounted RBD's hanging

2017-07-13 Thread Ilya Dryomov
On Wed, Jul 12, 2017 at 7:15 PM, Nick Fisk wrote: >> Hi Ilya, >> >> I have managed today to capture the kernel logs with debugging turned on and >> the ms+osd debug logs from the mentioned OSD. >> However, this is from a few minutes after the stall starts, not before. The >> very random nature o

Re: [ceph-users] Kernel mounted RBD's hanging

2017-07-12 Thread Nick Fisk
> -Original Message- > From: Nick Fisk [mailto:n...@fisk.me.uk] > Sent: 12 July 2017 13:47 > To: 'Ilya Dryomov' > Cc: 'Ceph Users' > Subject: RE: [ceph-users] Kernel mounted RBD's hanging > > > -Original Message- > >

Re: [ceph-users] Kernel mounted RBD's hanging

2017-07-08 Thread Nick Fisk
> -Original Message- > From: Ilya Dryomov [mailto:idryo...@gmail.com] > Sent: 07 July 2017 11:32 > To: Nick Fisk > Cc: Ceph Users > Subject: Re: [ceph-users] Kernel mounted RBD's hanging > > On Fri, Jul 7, 2017 at 12:10 PM, Nick Fisk wrote: > > M

Re: [ceph-users] Kernel mounted RBD's hanging

2017-07-07 Thread Ilya Dryomov
On Fri, Jul 7, 2017 at 12:10 PM, Nick Fisk wrote: > Managed to catch another one, osd.75 again, not sure if that is an indication > of anything or just a co-incidence. osd.75 is one of 8 OSD's in a cache tier, > so all IO will be funnelled through them. > > > cat > /sys/kernel/debug/ceph/d027d5

Re: [ceph-users] Kernel mounted RBD's hanging

2017-07-07 Thread Nick Fisk
> -Original Message- > From: Ilya Dryomov [mailto:idryo...@gmail.com] > Sent: 01 July 2017 13:19 > To: Nick Fisk > Cc: Ceph Users > Subject: Re: [ceph-users] Kernel mounted RBD's hanging > > On Sat, Jul 1, 2017 at 9:29 AM, Nick Fisk wrote: > >>

Re: [ceph-users] Kernel mounted RBD's hanging

2017-07-01 Thread Ilya Dryomov
On Sat, Jul 1, 2017 at 9:29 AM, Nick Fisk wrote: >> -Original Message- >> From: Ilya Dryomov [mailto:idryo...@gmail.com] >> Sent: 30 June 2017 14:06 >> To: Nick Fisk >> Cc: Ceph Users >> Subject: Re: [ceph-users] Kernel mounted RBD's hanging &g

Re: [ceph-users] Kernel mounted RBD's hanging

2017-07-01 Thread Nick Fisk
> -Original Message- > From: Ilya Dryomov [mailto:idryo...@gmail.com] > Sent: 30 June 2017 14:06 > To: Nick Fisk > Cc: Ceph Users > Subject: Re: [ceph-users] Kernel mounted RBD's hanging > > On Fri, Jun 30, 2017 at 2:14 PM, Nick Fisk wrote: >

Re: [ceph-users] Kernel mounted RBD's hanging

2017-06-30 Thread Alex Gorbachev
On Fri, Jun 30, 2017 at 8:12 AM Nick Fisk wrote: > *From:* Alex Gorbachev [mailto:a...@iss-integration.com] > *Sent:* 30 June 2017 03:54 > *To:* Ceph Users ; n...@fisk.me.uk > > > *Subject:* Re: [ceph-users] Kernel mounted RBD's hanging > > > > > > O

Re: [ceph-users] Kernel mounted RBD's hanging

2017-06-30 Thread Ilya Dryomov
On Fri, Jun 30, 2017 at 2:14 PM, Nick Fisk wrote: > > >> -Original Message- >> From: Ilya Dryomov [mailto:idryo...@gmail.com] >> Sent: 29 June 2017 18:54 >> To: Nick Fisk >> Cc: Ceph Users >> Subject: Re: [ceph-users] Kernel mounted RBD'

Re: [ceph-users] Kernel mounted RBD's hanging

2017-06-30 Thread Maged Mokhtar
On 2017-06-29 16:30, Nick Fisk wrote: > Hi All, > > Putting out a call for help to see if anyone can shed some light on this. > > Configuration: > Ceph cluster presenting RBD's->XFS->NFS->ESXi > Running 10.2.7 on the OSD's and 4.11 kernel on the NFS gateways in a > pacemaker cluster > Both OSD's

Re: [ceph-users] Kernel mounted RBD's hanging

2017-06-30 Thread Nick Fisk
> -Original Message- > From: Ilya Dryomov [mailto:idryo...@gmail.com] > Sent: 29 June 2017 18:54 > To: Nick Fisk > Cc: Ceph Users > Subject: Re: [ceph-users] Kernel mounted RBD's hanging > > On Thu, Jun 29, 2017 at 6:22 PM, Nick Fisk wrote: > >>

Re: [ceph-users] Kernel mounted RBD's hanging

2017-06-30 Thread Nick Fisk
From: Alex Gorbachev [mailto:a...@iss-integration.com] Sent: 30 June 2017 03:54 To: Ceph Users ; n...@fisk.me.uk Subject: Re: [ceph-users] Kernel mounted RBD's hanging On Thu, Jun 29, 2017 at 10:30 AM Nick Fisk mailto:n...@fisk.me.uk> > wrote: Hi All, Putting out a call for

Re: [ceph-users] Kernel mounted RBD's hanging

2017-06-29 Thread Alex Gorbachev
On Thu, Jun 29, 2017 at 10:30 AM Nick Fisk wrote: > Hi All, > > Putting out a call for help to see if anyone can shed some light on this. > > Configuration: > Ceph cluster presenting RBD's->XFS->NFS->ESXi > Running 10.2.7 on the OSD's and 4.11 kernel on the NFS gateways in a > pacemaker cluster >

Re: [ceph-users] Kernel mounted RBD's hanging

2017-06-29 Thread Ilya Dryomov
On Thu, Jun 29, 2017 at 6:22 PM, Nick Fisk wrote: >> -Original Message- >> From: Ilya Dryomov [mailto:idryo...@gmail.com] >> Sent: 29 June 2017 16:58 >> To: Nick Fisk >> Cc: Ceph Users >> Subject: Re: [ceph-users] Kernel mounted RBD's hanging &g

Re: [ceph-users] Kernel mounted RBD's hanging

2017-06-29 Thread Nick Fisk
> -Original Message- > From: Ilya Dryomov [mailto:idryo...@gmail.com] > Sent: 29 June 2017 16:58 > To: Nick Fisk > Cc: Ceph Users > Subject: Re: [ceph-users] Kernel mounted RBD's hanging > > On Thu, Jun 29, 2017 at 4:30 PM, Nick Fisk wrote: > > Hi

Re: [ceph-users] Kernel mounted RBD's hanging

2017-06-29 Thread Ilya Dryomov
On Thu, Jun 29, 2017 at 4:30 PM, Nick Fisk wrote: > Hi All, > > Putting out a call for help to see if anyone can shed some light on this. > > Configuration: > Ceph cluster presenting RBD's->XFS->NFS->ESXi > Running 10.2.7 on the OSD's and 4.11 kernel on the NFS gateways in a > pacemaker cluster >

[ceph-users] Kernel mounted RBD's hanging

2017-06-29 Thread Nick Fisk
Hi All, Putting out a call for help to see if anyone can shed some light on this. Configuration: Ceph cluster presenting RBD's->XFS->NFS->ESXi Running 10.2.7 on the OSD's and 4.11 kernel on the NFS gateways in a pacemaker cluster Both OSD's and clients are go into a pair of switches, single L2 do