Re: [ceph-users] Testing CephFS

2015-09-01 Thread Yan, Zheng 

> On Sep 1, 2015, at 16:13, Simon Hallam <s...@pml.ac.uk> wrote:
> 
> Hi Greg, Zheng,
> 
> Is this fixed in a later version of the kernel client? Or would it be wise 
> for us to start using the fuse client?
> 
> Cheers,

I just wrote a fix 
https://github.com/ceph/ceph-client/commit/33b68dde7f27927a7cb1a7691e3c5b6f847ffd14
 
<https://github.com/ceph/ceph-client/commit/33b68dde7f27927a7cb1a7691e3c5b6f847ffd14>.
  Yes, you should try ceps-fuse if this bug causes problems for you.

Regards
Yan, Zheng

> 
> Simon
> 
>> -Original Message-
>> From: Gregory Farnum [mailto:gfar...@redhat.com]
>> Sent: 31 August 2015 13:02
>> To: Yan, Zheng
>> Cc: Simon Hallam; Zheng Yan; ceph-users@lists.ceph.com
>> Subject: Re: [ceph-users] Testing CephFS
>> 
>> On Mon, Aug 31, 2015 at 12:16 PM, Yan, Zheng <uker...@gmail.com> wrote:
>>> On Mon, Aug 24, 2015 at 6:38 PM, Gregory Farnum
>> <gfar...@redhat.com> wrote:
>>>> On Mon, Aug 24, 2015 at 11:35 AM, Simon  Hallam <s...@pml.ac.uk>
>> wrote:
>>>>> Hi Greg,
>>>>> 
>>>>> The MDS' detect that the other one went down and started the replay.
>>>>> 
>>>>> I did some further testing with 20 client machines. Of the 20 client
>> machines, 5 hung with the following error:
>>>>> 
>>>>> [Aug24 10:53] ceph: mds0 caps stale
>>>>> [Aug24 10:54] ceph: mds0 caps stale
>>>>> [Aug24 10:58] ceph: mds0 hung
>>>>> [Aug24 11:03] ceph: mds0 came back
>>>>> [  +8.803334] libceph: mon2 10.15.0.3:6789 socket closed (con state
>> OPEN)
>>>>> [  +0.18] libceph: mon2 10.15.0.3:6789 session lost, hunting for new
>> mon
>>>>> [Aug24 11:04] ceph: mds0 reconnect start
>>>>> [  +0.084938] libceph: mon2 10.15.0.3:6789 session established
>>>>> [  +0.008475] ceph: mds0 reconnect denied
>>>> 
>>>> Oh, this might be a kernel bug, failing to ask for mdsmap updates when
>>>> the connection goes away. Zheng, does that sound familiar?
>>>> -Greg
>>>> 
>>> 
>>> I reproduced this locally (use SIGSTOP to stop the monitor) . I think
>>> the root cause is that kernel client does not implement
>>> CEPH_FEATURE_MSGR_KEEPALIVE2. So the kernel client couldn't reliably
>>> detect the event that network cable got unplugged. It kept waiting for
>>> new events from the disconnected connection.
>> 
>> Yeah, the userspace client maintains an ongoing MDSMap subscription
>> from the monitors in order to hear about this. It puts more load on
>> the monitors but right now that's the solution we're going with: the
>> monitor times out the MDS, publishes a series of new maps (pushed to
>> the clients) in order to activate a standby, and the clients see that
>> they need to connect to the new MDS instance.
>> -Greg
> 
> 
> Please visit our new website at www.pml.ac.uk and follow us on Twitter  
> @PlymouthMarine
> 
> Winner of the Environment & Conservation category, the Charity Awards 2014.
> 
> Plymouth Marine Laboratory (PML) is a company limited by guarantee registered 
> in England & Wales, company number 4178503. Registered Charity No. 1091222. 
> Registered Office: Prospect Place, The Hoe, Plymouth  PL1 3DH, UK. 
> 
> This message is private and confidential. If you have received this message 
> in error, please notify the sender and remove it from your system. You are 
> reminded that e-mail communications are not secure and may contain viruses; 
> PML accepts no liability for any loss or damage which may be caused by 
> viruses.
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Testing CephFS

2015-09-01 Thread Simon Hallam 
Hi Greg, Zheng,

Is this fixed in a later version of the kernel client? Or would it be wise for 
us to start using the fuse client?

Cheers,

Simon

> -Original Message-
> From: Gregory Farnum [mailto:gfar...@redhat.com]
> Sent: 31 August 2015 13:02
> To: Yan, Zheng
> Cc: Simon Hallam; Zheng Yan; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Testing CephFS
> 
> On Mon, Aug 31, 2015 at 12:16 PM, Yan, Zheng <uker...@gmail.com> wrote:
> > On Mon, Aug 24, 2015 at 6:38 PM, Gregory Farnum
> <gfar...@redhat.com> wrote:
> >> On Mon, Aug 24, 2015 at 11:35 AM, Simon  Hallam <s...@pml.ac.uk>
> wrote:
> >>> Hi Greg,
> >>>
> >>> The MDS' detect that the other one went down and started the replay.
> >>>
> >>> I did some further testing with 20 client machines. Of the 20 client
> machines, 5 hung with the following error:
> >>>
> >>> [Aug24 10:53] ceph: mds0 caps stale
> >>> [Aug24 10:54] ceph: mds0 caps stale
> >>> [Aug24 10:58] ceph: mds0 hung
> >>> [Aug24 11:03] ceph: mds0 came back
> >>> [  +8.803334] libceph: mon2 10.15.0.3:6789 socket closed (con state
> OPEN)
> >>> [  +0.18] libceph: mon2 10.15.0.3:6789 session lost, hunting for new
> mon
> >>> [Aug24 11:04] ceph: mds0 reconnect start
> >>> [  +0.084938] libceph: mon2 10.15.0.3:6789 session established
> >>> [  +0.008475] ceph: mds0 reconnect denied
> >>
> >> Oh, this might be a kernel bug, failing to ask for mdsmap updates when
> >> the connection goes away. Zheng, does that sound familiar?
> >> -Greg
> >>
> >
> > I reproduced this locally (use SIGSTOP to stop the monitor) . I think
> > the root cause is that kernel client does not implement
> > CEPH_FEATURE_MSGR_KEEPALIVE2. So the kernel client couldn't reliably
> > detect the event that network cable got unplugged. It kept waiting for
> > new events from the disconnected connection.
> 
> Yeah, the userspace client maintains an ongoing MDSMap subscription
> from the monitors in order to hear about this. It puts more load on
> the monitors but right now that's the solution we're going with: the
> monitor times out the MDS, publishes a series of new maps (pushed to
> the clients) in order to activate a standby, and the clients see that
> they need to connect to the new MDS instance.
> -Greg


Please visit our new website at www.pml.ac.uk and follow us on Twitter  
@PlymouthMarine

Winner of the Environment & Conservation category, the Charity Awards 2014.

Plymouth Marine Laboratory (PML) is a company limited by guarantee registered 
in England & Wales, company number 4178503. Registered Charity No. 1091222. 
Registered Office: Prospect Place, The Hoe, Plymouth  PL1 3DH, UK. 

This message is private and confidential. If you have received this message in 
error, please notify the sender and remove it from your system. You are 
reminded that e-mail communications are not secure and may contain viruses; PML 
accepts no liability for any loss or damage which may be caused by viruses.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Testing CephFS

2015-08-31 Thread Gregory Farnum 
On Mon, Aug 31, 2015 at 12:16 PM, Yan, Zheng  wrote:
> On Mon, Aug 24, 2015 at 6:38 PM, Gregory Farnum  wrote:
>> On Mon, Aug 24, 2015 at 11:35 AM, Simon  Hallam  wrote:
>>> Hi Greg,
>>>
>>> The MDS' detect that the other one went down and started the replay.
>>>
>>> I did some further testing with 20 client machines. Of the 20 client 
>>> machines, 5 hung with the following error:
>>>
>>> [Aug24 10:53] ceph: mds0 caps stale
>>> [Aug24 10:54] ceph: mds0 caps stale
>>> [Aug24 10:58] ceph: mds0 hung
>>> [Aug24 11:03] ceph: mds0 came back
>>> [  +8.803334] libceph: mon2 10.15.0.3:6789 socket closed (con state OPEN)
>>> [  +0.18] libceph: mon2 10.15.0.3:6789 session lost, hunting for new mon
>>> [Aug24 11:04] ceph: mds0 reconnect start
>>> [  +0.084938] libceph: mon2 10.15.0.3:6789 session established
>>> [  +0.008475] ceph: mds0 reconnect denied
>>
>> Oh, this might be a kernel bug, failing to ask for mdsmap updates when
>> the connection goes away. Zheng, does that sound familiar?
>> -Greg
>>
>
> I reproduced this locally (use SIGSTOP to stop the monitor) . I think
> the root cause is that kernel client does not implement
> CEPH_FEATURE_MSGR_KEEPALIVE2. So the kernel client couldn't reliably
> detect the event that network cable got unplugged. It kept waiting for
> new events from the disconnected connection.

Yeah, the userspace client maintains an ongoing MDSMap subscription
from the monitors in order to hear about this. It puts more load on
the monitors but right now that's the solution we're going with: the
monitor times out the MDS, publishes a series of new maps (pushed to
the clients) in order to activate a standby, and the clients see that
they need to connect to the new MDS instance.
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Testing CephFS

2015-08-28 Thread Simon Hallam 
After further testing, it does appear to be completely random (eg, not the same 
clients are affected).

I have one machine running the latest kernel/client which hasn't hung yet.
[root@f23-alpha ~]# uname -a
Linux f23-alpha.npm.ac.uk 4.2.0-0.rc5.git0.2.fc23.x86_64 #1 SMP Tue Aug 4 
01:37:40 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@f23-alpha ~]# ceph -v
ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)

Cheers,

Simon


 -Original Message-
 From: Gregory Farnum [mailto:gfar...@redhat.com]
 Sent: 27 August 2015 14:24
 To: Yan, Zheng
 Cc: Simon Hallam; ceph-users@lists.ceph.com
 Subject: Re: [ceph-users] Testing CephFS
 
 On Mon, Aug 24, 2015 at 12:53 PM, Simon  Hallam s...@pml.ac.uk wrote:
  The clients are:
  [root@gridnode50 ~]# uname -a
  Linux gridnode50 4.0.8-200.fc21.x86_64 #1 SMP Fri Jul 10 21:09:54 UTC 2015
 x86_64 x86_64 x86_64 GNU/Linux
  [root@gridnode50 ~]# ceph -v
  ceph version 0.80.10 (ea6c958c38df1216bf95c927f143d8b13c4a9e70)
 
  I don't think it is a reconnect timeout, as they don't even attempt to
 reconnect until I plug the Ethernet cable back into the original MDS?
 
 Right, the default timeout is going to be 15 seconds. But they clients
 should be getting MDSMap updates from the monitor that tell them the
 MDS has failed over. It looks like they're not timing out, then when
 the MDS *does* come back it tells them about its own death. Is that
 possible, Zheng?
 -Greg
 
 
  Cheers,
 
  Simon
 
  -Original Message-
  From: Yan, Zheng [mailto:z...@redhat.com]
  Sent: 24 August 2015 12:28
  To: Simon Hallam
  Cc: ceph-users@lists.ceph.com; Gregory Farnum
  Subject: Re: [ceph-users] Testing CephFS
 
 
   On Aug 24, 2015, at 18:38, Gregory Farnum gfar...@redhat.com
 wrote:
  
   On Mon, Aug 24, 2015 at 11:35 AM, Simon  Hallam s...@pml.ac.uk
 wrote:
   Hi Greg,
  
   The MDS' detect that the other one went down and started the replay.
  
   I did some further testing with 20 client machines. Of the 20 client
  machines, 5 hung with the following error:
  
   [Aug24 10:53] ceph: mds0 caps stale
   [Aug24 10:54] ceph: mds0 caps stale
   [Aug24 10:58] ceph: mds0 hung
   [Aug24 11:03] ceph: mds0 came back
   [  +8.803334] libceph: mon2 10.15.0.3:6789 socket closed (con state
 OPEN)
   [  +0.18] libceph: mon2 10.15.0.3:6789 session lost, hunting for
 new
  mon
   [Aug24 11:04] ceph: mds0 reconnect start
   [  +0.084938] libceph: mon2 10.15.0.3:6789 session established
   [  +0.008475] ceph: mds0 reconnect denied
  
   Oh, this might be a kernel bug, failing to ask for mdsmap updates when
   the connection goes away. Zheng, does that sound familiar?
   -Greg
 
  This seems like reconnect timeout. you can try enlarging
  mds_reconnect_timeout config option.
 
  Which version of kernel are you using?
 
  Yan, Zheng
 
  
  
   10.15.0.3 was the active MDS at the time I unplugged the Ethernet
 cable.
  
  
   This was the output of ceph -w as I ran the test (I've removed a lot of
 the
  pg remapping):
  
   2015-08-24 11:02:39.547529 mon.1 [INF] mon.ceph2 calling new
 monitor
  election
   2015-08-24 11:02:40.011995 mon.0 [INF] mon.ceph1 calling new
 monitor
  election
   2015-08-24 11:02:45.245869 mon.0 [INF] mon.ceph1@0 won leader
  election with quorum 0,1
   2015-08-24 11:02:45.257440 mon.0 [INF] HEALTH_WARN; 1 mons down,
  quorum 0,1 ceph1,ceph2
   2015-08-24 11:02:45.535369 mon.0 [INF] monmap e1: 3 mons at
  {ceph1=10.15.0.1:6789/0,ceph2=10.15.0.2:6789/0,ceph3=10.15.0.3:6789/0}
   2015-08-24 11:02:45.535444 mon.0 [INF] pgmap v15803: 8256 pgs: 8256
  active+clean; 1248 GB data, 2503 GB used, 193 TB / 196 TB avail; 47 B/s wr,
 0
  op/s
   2015-08-24 11:02:45.535541 mon.0 [INF] mdsmap e38: 1/1/1 up
  {0=ceph3=up:active}, 2 up:standby
   2015-08-24 11:02:45.535629 mon.0 [INF] osdmap e197: 36 osds: 36 up,
 36
  in
   2015-08-24 11:03:01.946397 mon.0 [INF] mdsmap e39: 1/1/1 up
  {0=ceph2=up:replay}, 1 up:standby
   2015-08-24 11:03:02.993880 mon.0 [INF] mds.0 10.15.0.2:6849/17644
  up:reconnect
   2015-08-24 11:03:02.993930 mon.0 [INF] mdsmap e40: 1/1/1 up
  {0=ceph2=up:reconnect}, 1 up:standby
   2015-08-24 11:03:51.461248 mon.0 [INF] mds.0 10.15.0.2:6849/17644
  up:rejoin
   2015-08-24 11:03:55.807131 mon.0 [INF] mds.0 10.15.0.2:6849/17644
  up:active
   2015-08-24 11:03:55.807195 mon.0 [INF] mdsmap e42: 1/1/1 up
  {0=ceph2=up:active}, 1 up:standby
   2015-08-24 11:06:48.036736 mon.0 [INF] mds.0 10.15.0.2:6849/17644
  up:active
   2015-08-24 11:06:48.036799 mon.0 [INF] mdsmap e43: 1/1/1 up
  {0=ceph2=up:active}, 1 up:standby
   *cable plugged back in*
   2015-08-24 11:13:13.230714 mon.0 [INF] osd.32 10.15.0.3:6832/11565
 boot
   2015-08-24 11:13:13.230765 mon.0 [INF] osdmap e212: 36 osds: 25 up,
 25
  in
   2015-08-24 11:13:13.230809 mon.0 [INF] mds.? 10.15.0.3:6833/16993
  up:boot
   2015-08-24 11:13:13.230837 mon.0 [INF] mdsmap e47: 1/1/1 up
  {0=ceph2=up:active}, 2 up:standby
   2015-08-24 11:13:30.799429 mon.2 [INF] mon.ceph3 calling new
 monitor
  election

Re: [ceph-users] Testing CephFS

2015-08-27 Thread Gregory Farnum 
On Mon, Aug 24, 2015 at 12:53 PM, Simon  Hallam s...@pml.ac.uk wrote:
 The clients are:
 [root@gridnode50 ~]# uname -a
 Linux gridnode50 4.0.8-200.fc21.x86_64 #1 SMP Fri Jul 10 21:09:54 UTC 2015 
 x86_64 x86_64 x86_64 GNU/Linux
 [root@gridnode50 ~]# ceph -v
 ceph version 0.80.10 (ea6c958c38df1216bf95c927f143d8b13c4a9e70)

 I don't think it is a reconnect timeout, as they don't even attempt to 
 reconnect until I plug the Ethernet cable back into the original MDS?

Right, the default timeout is going to be 15 seconds. But they clients
should be getting MDSMap updates from the monitor that tell them the
MDS has failed over. It looks like they're not timing out, then when
the MDS *does* come back it tells them about its own death. Is that
possible, Zheng?
-Greg


 Cheers,

 Simon

 -Original Message-
 From: Yan, Zheng [mailto:z...@redhat.com]
 Sent: 24 August 2015 12:28
 To: Simon Hallam
 Cc: ceph-users@lists.ceph.com; Gregory Farnum
 Subject: Re: [ceph-users] Testing CephFS


  On Aug 24, 2015, at 18:38, Gregory Farnum gfar...@redhat.com wrote:
 
  On Mon, Aug 24, 2015 at 11:35 AM, Simon  Hallam s...@pml.ac.uk wrote:
  Hi Greg,
 
  The MDS' detect that the other one went down and started the replay.
 
  I did some further testing with 20 client machines. Of the 20 client
 machines, 5 hung with the following error:
 
  [Aug24 10:53] ceph: mds0 caps stale
  [Aug24 10:54] ceph: mds0 caps stale
  [Aug24 10:58] ceph: mds0 hung
  [Aug24 11:03] ceph: mds0 came back
  [  +8.803334] libceph: mon2 10.15.0.3:6789 socket closed (con state OPEN)
  [  +0.18] libceph: mon2 10.15.0.3:6789 session lost, hunting for new
 mon
  [Aug24 11:04] ceph: mds0 reconnect start
  [  +0.084938] libceph: mon2 10.15.0.3:6789 session established
  [  +0.008475] ceph: mds0 reconnect denied
 
  Oh, this might be a kernel bug, failing to ask for mdsmap updates when
  the connection goes away. Zheng, does that sound familiar?
  -Greg

 This seems like reconnect timeout. you can try enlarging
 mds_reconnect_timeout config option.

 Which version of kernel are you using?

 Yan, Zheng

 
 
  10.15.0.3 was the active MDS at the time I unplugged the Ethernet cable.
 
 
  This was the output of ceph -w as I ran the test (I've removed a lot of 
  the
 pg remapping):
 
  2015-08-24 11:02:39.547529 mon.1 [INF] mon.ceph2 calling new monitor
 election
  2015-08-24 11:02:40.011995 mon.0 [INF] mon.ceph1 calling new monitor
 election
  2015-08-24 11:02:45.245869 mon.0 [INF] mon.ceph1@0 won leader
 election with quorum 0,1
  2015-08-24 11:02:45.257440 mon.0 [INF] HEALTH_WARN; 1 mons down,
 quorum 0,1 ceph1,ceph2
  2015-08-24 11:02:45.535369 mon.0 [INF] monmap e1: 3 mons at
 {ceph1=10.15.0.1:6789/0,ceph2=10.15.0.2:6789/0,ceph3=10.15.0.3:6789/0}
  2015-08-24 11:02:45.535444 mon.0 [INF] pgmap v15803: 8256 pgs: 8256
 active+clean; 1248 GB data, 2503 GB used, 193 TB / 196 TB avail; 47 B/s wr, 0
 op/s
  2015-08-24 11:02:45.535541 mon.0 [INF] mdsmap e38: 1/1/1 up
 {0=ceph3=up:active}, 2 up:standby
  2015-08-24 11:02:45.535629 mon.0 [INF] osdmap e197: 36 osds: 36 up, 36
 in
  2015-08-24 11:03:01.946397 mon.0 [INF] mdsmap e39: 1/1/1 up
 {0=ceph2=up:replay}, 1 up:standby
  2015-08-24 11:03:02.993880 mon.0 [INF] mds.0 10.15.0.2:6849/17644
 up:reconnect
  2015-08-24 11:03:02.993930 mon.0 [INF] mdsmap e40: 1/1/1 up
 {0=ceph2=up:reconnect}, 1 up:standby
  2015-08-24 11:03:51.461248 mon.0 [INF] mds.0 10.15.0.2:6849/17644
 up:rejoin
  2015-08-24 11:03:55.807131 mon.0 [INF] mds.0 10.15.0.2:6849/17644
 up:active
  2015-08-24 11:03:55.807195 mon.0 [INF] mdsmap e42: 1/1/1 up
 {0=ceph2=up:active}, 1 up:standby
  2015-08-24 11:06:48.036736 mon.0 [INF] mds.0 10.15.0.2:6849/17644
 up:active
  2015-08-24 11:06:48.036799 mon.0 [INF] mdsmap e43: 1/1/1 up
 {0=ceph2=up:active}, 1 up:standby
  *cable plugged back in*
  2015-08-24 11:13:13.230714 mon.0 [INF] osd.32 10.15.0.3:6832/11565 boot
  2015-08-24 11:13:13.230765 mon.0 [INF] osdmap e212: 36 osds: 25 up, 25
 in
  2015-08-24 11:13:13.230809 mon.0 [INF] mds.? 10.15.0.3:6833/16993
 up:boot
  2015-08-24 11:13:13.230837 mon.0 [INF] mdsmap e47: 1/1/1 up
 {0=ceph2=up:active}, 2 up:standby
  2015-08-24 11:13:30.799429 mon.2 [INF] mon.ceph3 calling new monitor
 election
  2015-08-24 11:13:30.826158 mon.0 [INF] mon.ceph1 calling new monitor
 election
  2015-08-24 11:13:30.926331 mon.0 [INF] mon.ceph1@0 won leader
 election with quorum 0,1,2
  2015-08-24 11:13:30.968739 mon.0 [INF] mdsmap e47: 1/1/1 up
 {0=ceph2=up:active}, 2 up:standby
  2015-08-24 11:13:28.383203 mds.0 [INF] denied reconnect attempt (mds is
 up:active) from client.24155 10.10.10.95:0/3238635414 after 625.375507
 (allowed interval 45)
  2015-08-24 11:13:29.721653 mds.0 [INF] denied reconnect attempt (mds is
 up:active) from client.24146 10.10.10.99:0/3454703638 after 626.713952
 (allowed interval 45)
  2015-08-24 11:13:31.113004 mds.0 [INF] denied reconnect attempt (mds is
 up:active) from client.24140 10.10.10.60:0/359606080 after 628.105302
 (allowed interval 45

Re: [ceph-users] Testing CephFS

2015-08-24 Thread Shinobu 
Need to be more careful but probably you're right -;

./net/ceph/messenger.c

Shinobu

On Mon, Aug 24, 2015 at 8:53 PM, Simon Hallam s...@pml.ac.uk wrote:

 The clients are:
 [root@gridnode50 ~]# uname -a
 Linux gridnode50 4.0.8-200.fc21.x86_64 #1 SMP Fri Jul 10 21:09:54 UTC 2015
 x86_64 x86_64 x86_64 GNU/Linux
 [root@gridnode50 ~]# ceph -v
 ceph version 0.80.10 (ea6c958c38df1216bf95c927f143d8b13c4a9e70)

 I don't think it is a reconnect timeout, as they don't even attempt to
 reconnect until I plug the Ethernet cable back into the original MDS?

 Cheers,

 Simon

  -Original Message-
  From: Yan, Zheng [mailto:z...@redhat.com]
  Sent: 24 August 2015 12:28
  To: Simon Hallam
  Cc: ceph-users@lists.ceph.com; Gregory Farnum
  Subject: Re: [ceph-users] Testing CephFS
 
 
   On Aug 24, 2015, at 18:38, Gregory Farnum gfar...@redhat.com wrote:
  
   On Mon, Aug 24, 2015 at 11:35 AM, Simon  Hallam s...@pml.ac.uk wrote:
   Hi Greg,
  
   The MDS' detect that the other one went down and started the replay.
  
   I did some further testing with 20 client machines. Of the 20 client
  machines, 5 hung with the following error:
  
   [Aug24 10:53] ceph: mds0 caps stale
   [Aug24 10:54] ceph: mds0 caps stale
   [Aug24 10:58] ceph: mds0 hung
   [Aug24 11:03] ceph: mds0 came back
   [  +8.803334] libceph: mon2 10.15.0.3:6789 socket closed (con state
 OPEN)
   [  +0.18] libceph: mon2 10.15.0.3:6789 session lost, hunting for
 new
  mon
   [Aug24 11:04] ceph: mds0 reconnect start
   [  +0.084938] libceph: mon2 10.15.0.3:6789 session established
   [  +0.008475] ceph: mds0 reconnect denied
  
   Oh, this might be a kernel bug, failing to ask for mdsmap updates when
   the connection goes away. Zheng, does that sound familiar?
   -Greg
 
  This seems like reconnect timeout. you can try enlarging
  mds_reconnect_timeout config option.
 
  Which version of kernel are you using?
 
  Yan, Zheng
 
  
  
   10.15.0.3 was the active MDS at the time I unplugged the Ethernet
 cable.
  
  
   This was the output of ceph -w as I ran the test (I've removed a lot
 of the
  pg remapping):
  
   2015-08-24 11:02:39.547529 mon.1 [INF] mon.ceph2 calling new monitor
  election
   2015-08-24 11:02:40.011995 mon.0 [INF] mon.ceph1 calling new monitor
  election
   2015-08-24 11:02:45.245869 mon.0 [INF] mon.ceph1@0 won leader
  election with quorum 0,1
   2015-08-24 11:02:45.257440 mon.0 [INF] HEALTH_WARN; 1 mons down,
  quorum 0,1 ceph1,ceph2
   2015-08-24 11:02:45.535369 mon.0 [INF] monmap e1: 3 mons at
  {ceph1=10.15.0.1:6789/0,ceph2=10.15.0.2:6789/0,ceph3=10.15.0.3:6789/0}
   2015-08-24 11:02:45.535444 mon.0 [INF] pgmap v15803: 8256 pgs: 8256
  active+clean; 1248 GB data, 2503 GB used, 193 TB / 196 TB avail; 47 B/s
 wr, 0
  op/s
   2015-08-24 11:02:45.535541 mon.0 [INF] mdsmap e38: 1/1/1 up
  {0=ceph3=up:active}, 2 up:standby
   2015-08-24 11:02:45.535629 mon.0 [INF] osdmap e197: 36 osds: 36 up, 36
  in
   2015-08-24 11:03:01.946397 mon.0 [INF] mdsmap e39: 1/1/1 up
  {0=ceph2=up:replay}, 1 up:standby
   2015-08-24 11:03:02.993880 mon.0 [INF] mds.0 10.15.0.2:6849/17644
  up:reconnect
   2015-08-24 11:03:02.993930 mon.0 [INF] mdsmap e40: 1/1/1 up
  {0=ceph2=up:reconnect}, 1 up:standby
   2015-08-24 11:03:51.461248 mon.0 [INF] mds.0 10.15.0.2:6849/17644
  up:rejoin
   2015-08-24 11:03:55.807131 mon.0 [INF] mds.0 10.15.0.2:6849/17644
  up:active
   2015-08-24 11:03:55.807195 mon.0 [INF] mdsmap e42: 1/1/1 up
  {0=ceph2=up:active}, 1 up:standby
   2015-08-24 11:06:48.036736 mon.0 [INF] mds.0 10.15.0.2:6849/17644
  up:active
   2015-08-24 11:06:48.036799 mon.0 [INF] mdsmap e43: 1/1/1 up
  {0=ceph2=up:active}, 1 up:standby
   *cable plugged back in*
   2015-08-24 11:13:13.230714 mon.0 [INF] osd.32 10.15.0.3:6832/11565
 boot
   2015-08-24 11:13:13.230765 mon.0 [INF] osdmap e212: 36 osds: 25 up, 25
  in
   2015-08-24 11:13:13.230809 mon.0 [INF] mds.? 10.15.0.3:6833/16993
  up:boot
   2015-08-24 11:13:13.230837 mon.0 [INF] mdsmap e47: 1/1/1 up
  {0=ceph2=up:active}, 2 up:standby
   2015-08-24 11:13:30.799429 mon.2 [INF] mon.ceph3 calling new monitor
  election
   2015-08-24 11:13:30.826158 mon.0 [INF] mon.ceph1 calling new monitor
  election
   2015-08-24 11:13:30.926331 mon.0 [INF] mon.ceph1@0 won leader
  election with quorum 0,1,2
   2015-08-24 11:13:30.968739 mon.0 [INF] mdsmap e47: 1/1/1 up
  {0=ceph2=up:active}, 2 up:standby
   2015-08-24 11:13:28.383203 mds.0 [INF] denied reconnect attempt (mds
 is
  up:active) from client.24155 10.10.10.95:0/3238635414 after 625.375507
  (allowed interval 45)
   2015-08-24 11:13:29.721653 mds.0 [INF] denied reconnect attempt (mds
 is
  up:active) from client.24146 10.10.10.99:0/3454703638 after 626.713952
  (allowed interval 45)
   2015-08-24 11:13:31.113004 mds.0 [INF] denied reconnect attempt (mds
 is
  up:active) from client.24140 10.10.10.60:0/359606080 after 628.105302
  (allowed interval 45)
   2015-08-24 11:13:50.933020 mds.0 [INF] denied reconnect attempt (mds
 is
  up:active) from client

Re: [ceph-users] Testing CephFS

2015-08-24 Thread Simon Hallam 
Hi Greg,

The MDS' detect that the other one went down and started the replay. 

I did some further testing with 20 client machines. Of the 20 client machines, 
5 hung with the following error:

[Aug24 10:53] ceph: mds0 caps stale
[Aug24 10:54] ceph: mds0 caps stale
[Aug24 10:58] ceph: mds0 hung
[Aug24 11:03] ceph: mds0 came back
[  +8.803334] libceph: mon2 10.15.0.3:6789 socket closed (con state OPEN)
[  +0.18] libceph: mon2 10.15.0.3:6789 session lost, hunting for new mon
[Aug24 11:04] ceph: mds0 reconnect start
[  +0.084938] libceph: mon2 10.15.0.3:6789 session established
[  +0.008475] ceph: mds0 reconnect denied

10.15.0.3 was the active MDS at the time I unplugged the Ethernet cable.


This was the output of ceph -w as I ran the test (I've removed a lot of the pg 
remapping):

2015-08-24 11:02:39.547529 mon.1 [INF] mon.ceph2 calling new monitor election
2015-08-24 11:02:40.011995 mon.0 [INF] mon.ceph1 calling new monitor election
2015-08-24 11:02:45.245869 mon.0 [INF] mon.ceph1@0 won leader election with 
quorum 0,1
2015-08-24 11:02:45.257440 mon.0 [INF] HEALTH_WARN; 1 mons down, quorum 0,1 
ceph1,ceph2
2015-08-24 11:02:45.535369 mon.0 [INF] monmap e1: 3 mons at 
{ceph1=10.15.0.1:6789/0,ceph2=10.15.0.2:6789/0,ceph3=10.15.0.3:6789/0}
2015-08-24 11:02:45.535444 mon.0 [INF] pgmap v15803: 8256 pgs: 8256 
active+clean; 1248 GB data, 2503 GB used, 193 TB / 196 TB avail; 47 B/s wr, 0 
op/s
2015-08-24 11:02:45.535541 mon.0 [INF] mdsmap e38: 1/1/1 up 
{0=ceph3=up:active}, 2 up:standby
2015-08-24 11:02:45.535629 mon.0 [INF] osdmap e197: 36 osds: 36 up, 36 in
2015-08-24 11:03:01.946397 mon.0 [INF] mdsmap e39: 1/1/1 up 
{0=ceph2=up:replay}, 1 up:standby
2015-08-24 11:03:02.993880 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:reconnect
2015-08-24 11:03:02.993930 mon.0 [INF] mdsmap e40: 1/1/1 up 
{0=ceph2=up:reconnect}, 1 up:standby
2015-08-24 11:03:51.461248 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:rejoin
2015-08-24 11:03:55.807131 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:active
2015-08-24 11:03:55.807195 mon.0 [INF] mdsmap e42: 1/1/1 up 
{0=ceph2=up:active}, 1 up:standby
2015-08-24 11:06:48.036736 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:active
2015-08-24 11:06:48.036799 mon.0 [INF] mdsmap e43: 1/1/1 up 
{0=ceph2=up:active}, 1 up:standby
*cable plugged back in*
2015-08-24 11:13:13.230714 mon.0 [INF] osd.32 10.15.0.3:6832/11565 boot
2015-08-24 11:13:13.230765 mon.0 [INF] osdmap e212: 36 osds: 25 up, 25 in
2015-08-24 11:13:13.230809 mon.0 [INF] mds.? 10.15.0.3:6833/16993 up:boot
2015-08-24 11:13:13.230837 mon.0 [INF] mdsmap e47: 1/1/1 up 
{0=ceph2=up:active}, 2 up:standby
2015-08-24 11:13:30.799429 mon.2 [INF] mon.ceph3 calling new monitor election
2015-08-24 11:13:30.826158 mon.0 [INF] mon.ceph1 calling new monitor election
2015-08-24 11:13:30.926331 mon.0 [INF] mon.ceph1@0 won leader election with 
quorum 0,1,2
2015-08-24 11:13:30.968739 mon.0 [INF] mdsmap e47: 1/1/1 up 
{0=ceph2=up:active}, 2 up:standby
2015-08-24 11:13:28.383203 mds.0 [INF] denied reconnect attempt (mds is 
up:active) from client.24155 10.10.10.95:0/3238635414 after 625.375507 (allowed 
interval 45)
2015-08-24 11:13:29.721653 mds.0 [INF] denied reconnect attempt (mds is 
up:active) from client.24146 10.10.10.99:0/3454703638 after 626.713952 (allowed 
interval 45)
2015-08-24 11:13:31.113004 mds.0 [INF] denied reconnect attempt (mds is 
up:active) from client.24140 10.10.10.60:0/359606080 after 628.105302 (allowed 
interval 45)
2015-08-24 11:13:50.933020 mds.0 [INF] denied reconnect attempt (mds is 
up:active) from client.24152 10.10.10.67:0/3475305031 after 647.925323 (allowed 
interval 45)
2015-08-24 11:13:51.037681 mds.0 [INF] denied reconnect attempt (mds is 
up:active) from client.24149 10.10.10.68:0/22416725 after 648.029988 (allowed 
interval 45)

I did just notice that none of the times match up. So may try again once I fix 
ntp/chrony and see if that makes a difference.

Cheers,

Simon

 -Original Message-
 From: Gregory Farnum [mailto:gfar...@redhat.com]
 Sent: 21 August 2015 12:16
 To: Simon Hallam
 Cc: ceph-users@lists.ceph.com
 Subject: Re: [ceph-users] Testing CephFS
 
 On Thu, Aug 20, 2015 at 11:07 AM, Simon  Hallam s...@pml.ac.uk wrote:
  Hey all,
 
 
 
  We are currently testing CephFS on a small (3 node) cluster.
 
 
 
  The setup is currently:
 
 
 
  Each server has 12 OSDs, 1 Monitor and 1 MDS running on it:
 
  The servers are running: 0.94.2-0.el7
 
  The clients are running: Ceph: 0.80.10-1.fc21, Kernel: 4.0.6-200.fc21.x86_64
 
 
 
  ceph -s
 
  cluster 4ed5ecdd-0c5b-4422-9d99-c9e42c6bd4cd
 
   health HEALTH_OK
 
   monmap e1: 3 mons at
  {ceph1=10.15.0.1:6789/0,ceph2=10.15.0.2:6789/0,ceph3=10.15.0.3:6789/0}
 
  election epoch 20, quorum 0,1,2 ceph1,ceph2,ceph3
 
   mdsmap e12: 1/1/1 up {0=ceph3=up:active}, 2 up:standby
 
   osdmap e389: 36 osds: 36 up, 36 in
 
pgmap v19370: 8256 pgs, 3 pools, 51217 MB data, 14035 objects
 
  95526 MB used, 196 TB / 196 TB

Re: [ceph-users] Testing CephFS

2015-08-24 Thread Gregory Farnum 
On Mon, Aug 24, 2015 at 11:35 AM, Simon  Hallam s...@pml.ac.uk wrote:
 Hi Greg,

 The MDS' detect that the other one went down and started the replay.

 I did some further testing with 20 client machines. Of the 20 client 
 machines, 5 hung with the following error:

 [Aug24 10:53] ceph: mds0 caps stale
 [Aug24 10:54] ceph: mds0 caps stale
 [Aug24 10:58] ceph: mds0 hung
 [Aug24 11:03] ceph: mds0 came back
 [  +8.803334] libceph: mon2 10.15.0.3:6789 socket closed (con state OPEN)
 [  +0.18] libceph: mon2 10.15.0.3:6789 session lost, hunting for new mon
 [Aug24 11:04] ceph: mds0 reconnect start
 [  +0.084938] libceph: mon2 10.15.0.3:6789 session established
 [  +0.008475] ceph: mds0 reconnect denied

Oh, this might be a kernel bug, failing to ask for mdsmap updates when
the connection goes away. Zheng, does that sound familiar?
-Greg


 10.15.0.3 was the active MDS at the time I unplugged the Ethernet cable.


 This was the output of ceph -w as I ran the test (I've removed a lot of the 
 pg remapping):

 2015-08-24 11:02:39.547529 mon.1 [INF] mon.ceph2 calling new monitor election
 2015-08-24 11:02:40.011995 mon.0 [INF] mon.ceph1 calling new monitor election
 2015-08-24 11:02:45.245869 mon.0 [INF] mon.ceph1@0 won leader election with 
 quorum 0,1
 2015-08-24 11:02:45.257440 mon.0 [INF] HEALTH_WARN; 1 mons down, quorum 0,1 
 ceph1,ceph2
 2015-08-24 11:02:45.535369 mon.0 [INF] monmap e1: 3 mons at 
 {ceph1=10.15.0.1:6789/0,ceph2=10.15.0.2:6789/0,ceph3=10.15.0.3:6789/0}
 2015-08-24 11:02:45.535444 mon.0 [INF] pgmap v15803: 8256 pgs: 8256 
 active+clean; 1248 GB data, 2503 GB used, 193 TB / 196 TB avail; 47 B/s wr, 0 
 op/s
 2015-08-24 11:02:45.535541 mon.0 [INF] mdsmap e38: 1/1/1 up 
 {0=ceph3=up:active}, 2 up:standby
 2015-08-24 11:02:45.535629 mon.0 [INF] osdmap e197: 36 osds: 36 up, 36 in
 2015-08-24 11:03:01.946397 mon.0 [INF] mdsmap e39: 1/1/1 up 
 {0=ceph2=up:replay}, 1 up:standby
 2015-08-24 11:03:02.993880 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:reconnect
 2015-08-24 11:03:02.993930 mon.0 [INF] mdsmap e40: 1/1/1 up 
 {0=ceph2=up:reconnect}, 1 up:standby
 2015-08-24 11:03:51.461248 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:rejoin
 2015-08-24 11:03:55.807131 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:active
 2015-08-24 11:03:55.807195 mon.0 [INF] mdsmap e42: 1/1/1 up 
 {0=ceph2=up:active}, 1 up:standby
 2015-08-24 11:06:48.036736 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:active
 2015-08-24 11:06:48.036799 mon.0 [INF] mdsmap e43: 1/1/1 up 
 {0=ceph2=up:active}, 1 up:standby
 *cable plugged back in*
 2015-08-24 11:13:13.230714 mon.0 [INF] osd.32 10.15.0.3:6832/11565 boot
 2015-08-24 11:13:13.230765 mon.0 [INF] osdmap e212: 36 osds: 25 up, 25 in
 2015-08-24 11:13:13.230809 mon.0 [INF] mds.? 10.15.0.3:6833/16993 up:boot
 2015-08-24 11:13:13.230837 mon.0 [INF] mdsmap e47: 1/1/1 up 
 {0=ceph2=up:active}, 2 up:standby
 2015-08-24 11:13:30.799429 mon.2 [INF] mon.ceph3 calling new monitor election
 2015-08-24 11:13:30.826158 mon.0 [INF] mon.ceph1 calling new monitor election
 2015-08-24 11:13:30.926331 mon.0 [INF] mon.ceph1@0 won leader election with 
 quorum 0,1,2
 2015-08-24 11:13:30.968739 mon.0 [INF] mdsmap e47: 1/1/1 up 
 {0=ceph2=up:active}, 2 up:standby
 2015-08-24 11:13:28.383203 mds.0 [INF] denied reconnect attempt (mds is 
 up:active) from client.24155 10.10.10.95:0/3238635414 after 625.375507 
 (allowed interval 45)
 2015-08-24 11:13:29.721653 mds.0 [INF] denied reconnect attempt (mds is 
 up:active) from client.24146 10.10.10.99:0/3454703638 after 626.713952 
 (allowed interval 45)
 2015-08-24 11:13:31.113004 mds.0 [INF] denied reconnect attempt (mds is 
 up:active) from client.24140 10.10.10.60:0/359606080 after 628.105302 
 (allowed interval 45)
 2015-08-24 11:13:50.933020 mds.0 [INF] denied reconnect attempt (mds is 
 up:active) from client.24152 10.10.10.67:0/3475305031 after 647.925323 
 (allowed interval 45)
 2015-08-24 11:13:51.037681 mds.0 [INF] denied reconnect attempt (mds is 
 up:active) from client.24149 10.10.10.68:0/22416725 after 648.029988 (allowed 
 interval 45)

 I did just notice that none of the times match up. So may try again once I 
 fix ntp/chrony and see if that makes a difference.

 Cheers,

 Simon

 -Original Message-
 From: Gregory Farnum [mailto:gfar...@redhat.com]
 Sent: 21 August 2015 12:16
 To: Simon Hallam
 Cc: ceph-users@lists.ceph.com
 Subject: Re: [ceph-users] Testing CephFS

 On Thu, Aug 20, 2015 at 11:07 AM, Simon  Hallam s...@pml.ac.uk wrote:
  Hey all,
 
 
 
  We are currently testing CephFS on a small (3 node) cluster.
 
 
 
  The setup is currently:
 
 
 
  Each server has 12 OSDs, 1 Monitor and 1 MDS running on it:
 
  The servers are running: 0.94.2-0.el7
 
  The clients are running: Ceph: 0.80.10-1.fc21, Kernel: 
  4.0.6-200.fc21.x86_64
 
 
 
  ceph -s
 
  cluster 4ed5ecdd-0c5b-4422-9d99-c9e42c6bd4cd
 
   health HEALTH_OK
 
   monmap e1: 3 mons at
  {ceph1=10.15.0.1:6789/0,ceph2=10.15.0.2:6789/0,ceph3=10.15.0.3:6789/0

Re: [ceph-users] Testing CephFS

2015-08-24 Thread Yan, Zheng 

 On Aug 24, 2015, at 18:38, Gregory Farnum gfar...@redhat.com wrote:
 
 On Mon, Aug 24, 2015 at 11:35 AM, Simon  Hallam s...@pml.ac.uk wrote:
 Hi Greg,
 
 The MDS' detect that the other one went down and started the replay.
 
 I did some further testing with 20 client machines. Of the 20 client 
 machines, 5 hung with the following error:
 
 [Aug24 10:53] ceph: mds0 caps stale
 [Aug24 10:54] ceph: mds0 caps stale
 [Aug24 10:58] ceph: mds0 hung
 [Aug24 11:03] ceph: mds0 came back
 [  +8.803334] libceph: mon2 10.15.0.3:6789 socket closed (con state OPEN)
 [  +0.18] libceph: mon2 10.15.0.3:6789 session lost, hunting for new mon
 [Aug24 11:04] ceph: mds0 reconnect start
 [  +0.084938] libceph: mon2 10.15.0.3:6789 session established
 [  +0.008475] ceph: mds0 reconnect denied
 
 Oh, this might be a kernel bug, failing to ask for mdsmap updates when
 the connection goes away. Zheng, does that sound familiar?
 -Greg

This seems like reconnect timeout. you can try enlarging mds_reconnect_timeout 
config option.

Which version of kernel are you using?

Yan, Zheng

 
 
 10.15.0.3 was the active MDS at the time I unplugged the Ethernet cable.
 
 
 This was the output of ceph -w as I ran the test (I've removed a lot of the 
 pg remapping):
 
 2015-08-24 11:02:39.547529 mon.1 [INF] mon.ceph2 calling new monitor election
 2015-08-24 11:02:40.011995 mon.0 [INF] mon.ceph1 calling new monitor election
 2015-08-24 11:02:45.245869 mon.0 [INF] mon.ceph1@0 won leader election with 
 quorum 0,1
 2015-08-24 11:02:45.257440 mon.0 [INF] HEALTH_WARN; 1 mons down, quorum 0,1 
 ceph1,ceph2
 2015-08-24 11:02:45.535369 mon.0 [INF] monmap e1: 3 mons at 
 {ceph1=10.15.0.1:6789/0,ceph2=10.15.0.2:6789/0,ceph3=10.15.0.3:6789/0}
 2015-08-24 11:02:45.535444 mon.0 [INF] pgmap v15803: 8256 pgs: 8256 
 active+clean; 1248 GB data, 2503 GB used, 193 TB / 196 TB avail; 47 B/s wr, 
 0 op/s
 2015-08-24 11:02:45.535541 mon.0 [INF] mdsmap e38: 1/1/1 up 
 {0=ceph3=up:active}, 2 up:standby
 2015-08-24 11:02:45.535629 mon.0 [INF] osdmap e197: 36 osds: 36 up, 36 in
 2015-08-24 11:03:01.946397 mon.0 [INF] mdsmap e39: 1/1/1 up 
 {0=ceph2=up:replay}, 1 up:standby
 2015-08-24 11:03:02.993880 mon.0 [INF] mds.0 10.15.0.2:6849/17644 
 up:reconnect
 2015-08-24 11:03:02.993930 mon.0 [INF] mdsmap e40: 1/1/1 up 
 {0=ceph2=up:reconnect}, 1 up:standby
 2015-08-24 11:03:51.461248 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:rejoin
 2015-08-24 11:03:55.807131 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:active
 2015-08-24 11:03:55.807195 mon.0 [INF] mdsmap e42: 1/1/1 up 
 {0=ceph2=up:active}, 1 up:standby
 2015-08-24 11:06:48.036736 mon.0 [INF] mds.0 10.15.0.2:6849/17644 up:active
 2015-08-24 11:06:48.036799 mon.0 [INF] mdsmap e43: 1/1/1 up 
 {0=ceph2=up:active}, 1 up:standby
 *cable plugged back in*
 2015-08-24 11:13:13.230714 mon.0 [INF] osd.32 10.15.0.3:6832/11565 boot
 2015-08-24 11:13:13.230765 mon.0 [INF] osdmap e212: 36 osds: 25 up, 25 in
 2015-08-24 11:13:13.230809 mon.0 [INF] mds.? 10.15.0.3:6833/16993 up:boot
 2015-08-24 11:13:13.230837 mon.0 [INF] mdsmap e47: 1/1/1 up 
 {0=ceph2=up:active}, 2 up:standby
 2015-08-24 11:13:30.799429 mon.2 [INF] mon.ceph3 calling new monitor election
 2015-08-24 11:13:30.826158 mon.0 [INF] mon.ceph1 calling new monitor election
 2015-08-24 11:13:30.926331 mon.0 [INF] mon.ceph1@0 won leader election with 
 quorum 0,1,2
 2015-08-24 11:13:30.968739 mon.0 [INF] mdsmap e47: 1/1/1 up 
 {0=ceph2=up:active}, 2 up:standby
 2015-08-24 11:13:28.383203 mds.0 [INF] denied reconnect attempt (mds is 
 up:active) from client.24155 10.10.10.95:0/3238635414 after 625.375507 
 (allowed interval 45)
 2015-08-24 11:13:29.721653 mds.0 [INF] denied reconnect attempt (mds is 
 up:active) from client.24146 10.10.10.99:0/3454703638 after 626.713952 
 (allowed interval 45)
 2015-08-24 11:13:31.113004 mds.0 [INF] denied reconnect attempt (mds is 
 up:active) from client.24140 10.10.10.60:0/359606080 after 628.105302 
 (allowed interval 45)
 2015-08-24 11:13:50.933020 mds.0 [INF] denied reconnect attempt (mds is 
 up:active) from client.24152 10.10.10.67:0/3475305031 after 647.925323 
 (allowed interval 45)
 2015-08-24 11:13:51.037681 mds.0 [INF] denied reconnect attempt (mds is 
 up:active) from client.24149 10.10.10.68:0/22416725 after 648.029988 
 (allowed interval 45)
 
 I did just notice that none of the times match up. So may try again once I 
 fix ntp/chrony and see if that makes a difference.
 
 Cheers,
 
 Simon
 
 -Original Message-
 From: Gregory Farnum [mailto:gfar...@redhat.com]
 Sent: 21 August 2015 12:16
 To: Simon Hallam
 Cc: ceph-users@lists.ceph.com
 Subject: Re: [ceph-users] Testing CephFS
 
 On Thu, Aug 20, 2015 at 11:07 AM, Simon  Hallam s...@pml.ac.uk wrote:
 Hey all,
 
 
 
 We are currently testing CephFS on a small (3 node) cluster.
 
 
 
 The setup is currently:
 
 
 
 Each server has 12 OSDs, 1 Monitor and 1 MDS running on it:
 
 The servers are running: 0.94.2-0.el7
 
 The clients are running: Ceph: 0.80.10-1.fc21, Kernel

Re: [ceph-users] Testing CephFS

2015-08-24 Thread Simon Hallam 
The clients are:
[root@gridnode50 ~]# uname -a
Linux gridnode50 4.0.8-200.fc21.x86_64 #1 SMP Fri Jul 10 21:09:54 UTC 2015 
x86_64 x86_64 x86_64 GNU/Linux
[root@gridnode50 ~]# ceph -v
ceph version 0.80.10 (ea6c958c38df1216bf95c927f143d8b13c4a9e70)

I don't think it is a reconnect timeout, as they don't even attempt to 
reconnect until I plug the Ethernet cable back into the original MDS?

Cheers,

Simon

 -Original Message-
 From: Yan, Zheng [mailto:z...@redhat.com]
 Sent: 24 August 2015 12:28
 To: Simon Hallam
 Cc: ceph-users@lists.ceph.com; Gregory Farnum
 Subject: Re: [ceph-users] Testing CephFS
 
 
  On Aug 24, 2015, at 18:38, Gregory Farnum gfar...@redhat.com wrote:
 
  On Mon, Aug 24, 2015 at 11:35 AM, Simon  Hallam s...@pml.ac.uk wrote:
  Hi Greg,
 
  The MDS' detect that the other one went down and started the replay.
 
  I did some further testing with 20 client machines. Of the 20 client
 machines, 5 hung with the following error:
 
  [Aug24 10:53] ceph: mds0 caps stale
  [Aug24 10:54] ceph: mds0 caps stale
  [Aug24 10:58] ceph: mds0 hung
  [Aug24 11:03] ceph: mds0 came back
  [  +8.803334] libceph: mon2 10.15.0.3:6789 socket closed (con state OPEN)
  [  +0.18] libceph: mon2 10.15.0.3:6789 session lost, hunting for new
 mon
  [Aug24 11:04] ceph: mds0 reconnect start
  [  +0.084938] libceph: mon2 10.15.0.3:6789 session established
  [  +0.008475] ceph: mds0 reconnect denied
 
  Oh, this might be a kernel bug, failing to ask for mdsmap updates when
  the connection goes away. Zheng, does that sound familiar?
  -Greg
 
 This seems like reconnect timeout. you can try enlarging
 mds_reconnect_timeout config option.
 
 Which version of kernel are you using?
 
 Yan, Zheng
 
 
 
  10.15.0.3 was the active MDS at the time I unplugged the Ethernet cable.
 
 
  This was the output of ceph -w as I ran the test (I've removed a lot of the
 pg remapping):
 
  2015-08-24 11:02:39.547529 mon.1 [INF] mon.ceph2 calling new monitor
 election
  2015-08-24 11:02:40.011995 mon.0 [INF] mon.ceph1 calling new monitor
 election
  2015-08-24 11:02:45.245869 mon.0 [INF] mon.ceph1@0 won leader
 election with quorum 0,1
  2015-08-24 11:02:45.257440 mon.0 [INF] HEALTH_WARN; 1 mons down,
 quorum 0,1 ceph1,ceph2
  2015-08-24 11:02:45.535369 mon.0 [INF] monmap e1: 3 mons at
 {ceph1=10.15.0.1:6789/0,ceph2=10.15.0.2:6789/0,ceph3=10.15.0.3:6789/0}
  2015-08-24 11:02:45.535444 mon.0 [INF] pgmap v15803: 8256 pgs: 8256
 active+clean; 1248 GB data, 2503 GB used, 193 TB / 196 TB avail; 47 B/s wr, 0
 op/s
  2015-08-24 11:02:45.535541 mon.0 [INF] mdsmap e38: 1/1/1 up
 {0=ceph3=up:active}, 2 up:standby
  2015-08-24 11:02:45.535629 mon.0 [INF] osdmap e197: 36 osds: 36 up, 36
 in
  2015-08-24 11:03:01.946397 mon.0 [INF] mdsmap e39: 1/1/1 up
 {0=ceph2=up:replay}, 1 up:standby
  2015-08-24 11:03:02.993880 mon.0 [INF] mds.0 10.15.0.2:6849/17644
 up:reconnect
  2015-08-24 11:03:02.993930 mon.0 [INF] mdsmap e40: 1/1/1 up
 {0=ceph2=up:reconnect}, 1 up:standby
  2015-08-24 11:03:51.461248 mon.0 [INF] mds.0 10.15.0.2:6849/17644
 up:rejoin
  2015-08-24 11:03:55.807131 mon.0 [INF] mds.0 10.15.0.2:6849/17644
 up:active
  2015-08-24 11:03:55.807195 mon.0 [INF] mdsmap e42: 1/1/1 up
 {0=ceph2=up:active}, 1 up:standby
  2015-08-24 11:06:48.036736 mon.0 [INF] mds.0 10.15.0.2:6849/17644
 up:active
  2015-08-24 11:06:48.036799 mon.0 [INF] mdsmap e43: 1/1/1 up
 {0=ceph2=up:active}, 1 up:standby
  *cable plugged back in*
  2015-08-24 11:13:13.230714 mon.0 [INF] osd.32 10.15.0.3:6832/11565 boot
  2015-08-24 11:13:13.230765 mon.0 [INF] osdmap e212: 36 osds: 25 up, 25
 in
  2015-08-24 11:13:13.230809 mon.0 [INF] mds.? 10.15.0.3:6833/16993
 up:boot
  2015-08-24 11:13:13.230837 mon.0 [INF] mdsmap e47: 1/1/1 up
 {0=ceph2=up:active}, 2 up:standby
  2015-08-24 11:13:30.799429 mon.2 [INF] mon.ceph3 calling new monitor
 election
  2015-08-24 11:13:30.826158 mon.0 [INF] mon.ceph1 calling new monitor
 election
  2015-08-24 11:13:30.926331 mon.0 [INF] mon.ceph1@0 won leader
 election with quorum 0,1,2
  2015-08-24 11:13:30.968739 mon.0 [INF] mdsmap e47: 1/1/1 up
 {0=ceph2=up:active}, 2 up:standby
  2015-08-24 11:13:28.383203 mds.0 [INF] denied reconnect attempt (mds is
 up:active) from client.24155 10.10.10.95:0/3238635414 after 625.375507
 (allowed interval 45)
  2015-08-24 11:13:29.721653 mds.0 [INF] denied reconnect attempt (mds is
 up:active) from client.24146 10.10.10.99:0/3454703638 after 626.713952
 (allowed interval 45)
  2015-08-24 11:13:31.113004 mds.0 [INF] denied reconnect attempt (mds is
 up:active) from client.24140 10.10.10.60:0/359606080 after 628.105302
 (allowed interval 45)
  2015-08-24 11:13:50.933020 mds.0 [INF] denied reconnect attempt (mds is
 up:active) from client.24152 10.10.10.67:0/3475305031 after 647.925323
 (allowed interval 45)
  2015-08-24 11:13:51.037681 mds.0 [INF] denied reconnect attempt (mds is
 up:active) from client.24149 10.10.10.68:0/22416725 after 648.029988
 (allowed interval 45)
 
  I did just notice that none

Re: [ceph-users] Testing CephFS

2015-08-21 Thread Gregory Farnum 
On Thu, Aug 20, 2015 at 11:07 AM, Simon  Hallam s...@pml.ac.uk wrote:
 Hey all,



 We are currently testing CephFS on a small (3 node) cluster.



 The setup is currently:



 Each server has 12 OSDs, 1 Monitor and 1 MDS running on it:

 The servers are running: 0.94.2-0.el7

 The clients are running: Ceph: 0.80.10-1.fc21, Kernel: 4.0.6-200.fc21.x86_64



 ceph -s

 cluster 4ed5ecdd-0c5b-4422-9d99-c9e42c6bd4cd

  health HEALTH_OK

  monmap e1: 3 mons at
 {ceph1=10.15.0.1:6789/0,ceph2=10.15.0.2:6789/0,ceph3=10.15.0.3:6789/0}

 election epoch 20, quorum 0,1,2 ceph1,ceph2,ceph3

  mdsmap e12: 1/1/1 up {0=ceph3=up:active}, 2 up:standby

  osdmap e389: 36 osds: 36 up, 36 in

   pgmap v19370: 8256 pgs, 3 pools, 51217 MB data, 14035 objects

 95526 MB used, 196 TB / 196 TB avail

 8256 active+clean



 Our Ceph.conf is relatively simple at the moment:



 cat /etc/ceph/ceph.conf

 [global]

 fsid = 4ed5ecdd-0c5b-4422-9d99-c9e42c6bd4cd

 mon_initial_members = ceph1, ceph2, ceph3

 mon_host = 10.15.0.1,10.15.0.2,10.15.0.3

 mon_pg_warn_max_per_osd = 1000

 auth_cluster_required = cephx

 auth_service_required = cephx

 auth_client_required = cephx

 filestore_xattr_use_omap = true

 osd_pool_default_size = 2



 When I pulled the plug on the master MDS last time (ceph1), it stopped all
 IO until I plugged it back in. I was under the assumption that the MDS would
 fail over the other 2 MDS’s and IO would continue?



 Is there something I need to do to allow the MDS’s to failover from each
 other without too much interruption? Or is this because the clients ceph
 version?

That's quite strange. How long did you wait for it to fail over? Did
the output of ceph -s (or ceph -w, whichever) change during that
time?
By default the monitors should have detected the MDS was dead after 30
seconds and put one of the other MDS nodes into replay and active.

...I wonder if this is because you lost a monitor at the same time as
the MDS. What kind of logging do you have available from during your
test?
-Greg




 Cheers,



 Simon Hallam

 Linux Support  Development Officer
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com