date:20170107

Re: [ceph-users] Ceph Monitor cephx issues

2017-01-07 Thread Shinobu Kinjo

Good to know.
Sorry for such an inconvenient for you anyway.

Regards,

On Sun, Jan 8, 2017 at 2:19 PM, Alex Evonosky 
wrote:

> Since this was a test lab, I totally purged the whole cluster and
> re-deployed..  working good now, thank you.
>
>
>
> Alex F. Evonosky
>
>  
>
> On Sat, Jan 7, 2017 at 9:14 PM, Alex Evonosky 
> wrote:
>
>> Thank you..
>>
>>
>> After sending the post, I totally removed the mon and issued the build
>> with ceph-deploy:
>>
>>
>> In the logs now:
>>
>> 2017-01-07 21:12:38.113534 7fa9613fd700  0 cephx: verify_reply couldn't
>> decrypt with error: error decoding block for decryption
>> 2017-01-07 21:12:38.113546 7fa9613fd700  0 -- 10.10.10.138:6789/0 >>
>> 10.10.10.103:6789/0 pipe(0x55feb2e9 sd=12 :50266 s=1 pgs=0 cs=0 l=0
>> c=0x55feb2ca0a80).failed verifying authorize reply
>> 2017-01-07 21:12:38.114529 7fa95787b700  0 cephx: verify_reply couldn't
>> decrypt with error: error decoding block for decryption
>> 2017-01-07 21:12:38.114567 7fa95787b700  0 -- 10.10.10.138:6789/0 >>
>> 10.10.10.252:6789/0 pipe(0x55feb2e91400 sd=11 :38690 s=1 pgs=0 cs=0 l=0
>> c=0x55feb2ca0c00).failed verifying authorize reply
>> 2017-01-07 21:12:40.114522 7fa9613fd700  0 cephx: verify_reply couldn't
>> decrypt with error: error decoding block for decryption
>> 2017-01-07 21:12:40.114542 7fa9613fd700  0 -- 10.10.10.138:6789/0 >>
>> 10.10.10.103:6789/0 pipe(0x55feb2e9 sd=11 :50278 s=1 pgs=0 cs=0 l=0
>> c=0x55feb2ca0a80).failed verifying authorize reply
>> 2017-01-07 21:12:40.115706 7fa95787b700  0 cephx: verify_reply couldn't
>> decrypt with error: error decoding block for decryption
>> 2017-01-07 21:12:40.115721 7fa95787b700  0 -- 10.10.10.138:6789/0 >>
>> 10.10.10.252:6789/0 pipe(0x55feb2e91400 sd=12 :38702 s=1 pgs=0 cs=0 l=0
>> c=0x55feb2ca0c00).failed verifying authorize reply
>> 2017-01-07 21:12:41.621916 7fa956f79700  0 cephx: verify_authorizer could
>> not decrypt ticket info: error: NSS AES final round failed: -8190
>> 2017-01-07 21:12:41.621929 7fa956f79700  0 mon.alex-desktop@1(probing)
>> e0 ms_verify_authorizer bad authorizer from mon 10.10.10.103:6789/0
>> 2017-01-07 21:12:41.621944 7fa956f79700  0 -- 10.10.10.138:6789/0 >>
>> 10.10.10.103:6789/0 pipe(0x55feb2fb5400 sd=21 :6789 s=0 pgs=0 cs=0 l=0
>> c=0x55feb2ca1500).accept: got bad authorizer
>>
>>
>>
>> $ sudo ceph -s
>> cluster f5aba719-4856-4ae2-a5d4-f9ff0f614b60
>>  health HEALTH_WARN
>> 512 pgs degraded
>> 348 pgs stale
>> 512 pgs stuck unclean
>> 512 pgs undersized
>> 6 requests are blocked > 32 sec
>> recovery 25013/50026 objects degraded (50.000%)
>> mds cluster is degraded
>> 1 mons down, quorum 0,2 alpha,toshiba-laptop
>>  monmap e17: 3 mons at {alex-desktop=10.10.10.138:678
>> 9/0,alpha=10.10.10.103:6789/0,toshiba-laptop=10.10.10.252:6789/0}
>> election epoch 806, quorum 0,2 alpha,toshiba-laptop
>>   fsmap e201858: 1/1/1 up {0=1=up:replay}
>>  osdmap e200229: 3 osds: 2 up, 2 in; 85 remapped pgs
>> flags sortbitwise
>>   pgmap v4088774: 512 pgs, 4 pools, 50883 MB data, 25013 objects
>> 59662 MB used, 476 GB / 563 GB avail
>> 25013/50026 objects degraded (50.000%)
>>  348 stale+active+undersized+degraded
>>  164 active+undersized+degraded
>>
>>
>>
>> root@alex-desktop:/var/lib/ceph/mon/ceph-alex-desktop# ls -ls
>> total 8
>> 0 -rw-r--r-- 1 ceph ceph0 Jan  7 21:11 done
>> 4 -rw--- 1 ceph ceph   77 Jan  7 21:05 keyring
>> 4 drwxr-xr-x 2 ceph ceph 4096 Jan  7 21:10 store.db
>> 0 -rw-r--r-- 1 ceph ceph0 Jan  7 21:05 systemd
>>
>>
>>
>>
>> Very odd...  never seen this issue on the other monitor deployments...
>>
>>
>>
>>
>>
>>
>>
>>
>> Alex F. Evonosky
>>
>>  
>>
>> On Sat, Jan 7, 2017 at 8:54 PM, Shinobu Kinjo  wrote:
>>
>>> Using ``ceph-deploy`` will save your life:
>>>
>>>  # https://github.com/ceph/ceph/blob/master/doc/start/quick-cep
>>> h-deploy.rst
>>>   * Please look at: Adding Monitors
>>>
>>> If you are using centos or similar, the latest package is available here:
>>>
>>>  # http://download.ceph.com/rpm-jewel/el7/noarch/ceph-deploy-1.
>>> 5.37-0.noarch.rpm
>>>
>>> Regards,
>>>
>>>
>>> On Sun, Jan 8, 2017 at 9:53 AM, Alex Evonosky 
>>> wrote:
>>>
 Thank you for the reply!

 I followed this article:

 http://docs.ceph.com/docs/jewel/rados/operations/add-or-rm-mons/


 Under the section: ADDING A MONITOR (MANUAL)



 Alex F. Evonosky

 
 

 On Sat, Jan 7, 2017 at 6:36 PM, Shinobu Kinjo 
 wrote:

> How did you add

Re: [ceph-users] Ceph Monitor cephx issues

2017-01-07 Thread Alex Evonosky

Since this was a test lab, I totally purged the whole cluster and
re-deployed..  working good now, thank you.



Alex F. Evonosky

 

On Sat, Jan 7, 2017 at 9:14 PM, Alex Evonosky 
wrote:

> Thank you..
>
>
> After sending the post, I totally removed the mon and issued the build
> with ceph-deploy:
>
>
> In the logs now:
>
> 2017-01-07 21:12:38.113534 7fa9613fd700  0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2017-01-07 21:12:38.113546 7fa9613fd700  0 -- 10.10.10.138:6789/0 >>
> 10.10.10.103:6789/0 pipe(0x55feb2e9 sd=12 :50266 s=1 pgs=0 cs=0 l=0
> c=0x55feb2ca0a80).failed verifying authorize reply
> 2017-01-07 21:12:38.114529 7fa95787b700  0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2017-01-07 21:12:38.114567 7fa95787b700  0 -- 10.10.10.138:6789/0 >>
> 10.10.10.252:6789/0 pipe(0x55feb2e91400 sd=11 :38690 s=1 pgs=0 cs=0 l=0
> c=0x55feb2ca0c00).failed verifying authorize reply
> 2017-01-07 21:12:40.114522 7fa9613fd700  0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2017-01-07 21:12:40.114542 7fa9613fd700  0 -- 10.10.10.138:6789/0 >>
> 10.10.10.103:6789/0 pipe(0x55feb2e9 sd=11 :50278 s=1 pgs=0 cs=0 l=0
> c=0x55feb2ca0a80).failed verifying authorize reply
> 2017-01-07 21:12:40.115706 7fa95787b700  0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2017-01-07 21:12:40.115721 7fa95787b700  0 -- 10.10.10.138:6789/0 >>
> 10.10.10.252:6789/0 pipe(0x55feb2e91400 sd=12 :38702 s=1 pgs=0 cs=0 l=0
> c=0x55feb2ca0c00).failed verifying authorize reply
> 2017-01-07 21:12:41.621916 7fa956f79700  0 cephx: verify_authorizer could
> not decrypt ticket info: error: NSS AES final round failed: -8190
> 2017-01-07 21:12:41.621929 7fa956f79700  0 mon.alex-desktop@1(probing) e0
> ms_verify_authorizer bad authorizer from mon 10.10.10.103:6789/0
> 2017-01-07 21:12:41.621944 7fa956f79700  0 -- 10.10.10.138:6789/0 >>
> 10.10.10.103:6789/0 pipe(0x55feb2fb5400 sd=21 :6789 s=0 pgs=0 cs=0 l=0
> c=0x55feb2ca1500).accept: got bad authorizer
>
>
>
> $ sudo ceph -s
> cluster f5aba719-4856-4ae2-a5d4-f9ff0f614b60
>  health HEALTH_WARN
> 512 pgs degraded
> 348 pgs stale
> 512 pgs stuck unclean
> 512 pgs undersized
> 6 requests are blocked > 32 sec
> recovery 25013/50026 objects degraded (50.000%)
> mds cluster is degraded
> 1 mons down, quorum 0,2 alpha,toshiba-laptop
>  monmap e17: 3 mons at {alex-desktop=10.10.10.138:
> 6789/0,alpha=10.10.10.103:6789/0,toshiba-laptop=10.10.10.252:6789/0}
> election epoch 806, quorum 0,2 alpha,toshiba-laptop
>   fsmap e201858: 1/1/1 up {0=1=up:replay}
>  osdmap e200229: 3 osds: 2 up, 2 in; 85 remapped pgs
> flags sortbitwise
>   pgmap v4088774: 512 pgs, 4 pools, 50883 MB data, 25013 objects
> 59662 MB used, 476 GB / 563 GB avail
> 25013/50026 objects degraded (50.000%)
>  348 stale+active+undersized+degraded
>  164 active+undersized+degraded
>
>
>
> root@alex-desktop:/var/lib/ceph/mon/ceph-alex-desktop# ls -ls
> total 8
> 0 -rw-r--r-- 1 ceph ceph0 Jan  7 21:11 done
> 4 -rw--- 1 ceph ceph   77 Jan  7 21:05 keyring
> 4 drwxr-xr-x 2 ceph ceph 4096 Jan  7 21:10 store.db
> 0 -rw-r--r-- 1 ceph ceph0 Jan  7 21:05 systemd
>
>
>
>
> Very odd...  never seen this issue on the other monitor deployments...
>
>
>
>
>
>
>
>
> Alex F. Evonosky
>
>  
>
> On Sat, Jan 7, 2017 at 8:54 PM, Shinobu Kinjo  wrote:
>
>> Using ``ceph-deploy`` will save your life:
>>
>>  # https://github.com/ceph/ceph/blob/master/doc/start/quick-cep
>> h-deploy.rst
>>   * Please look at: Adding Monitors
>>
>> If you are using centos or similar, the latest package is available here:
>>
>>  # http://download.ceph.com/rpm-jewel/el7/noarch/ceph-deploy-1.
>> 5.37-0.noarch.rpm
>>
>> Regards,
>>
>>
>> On Sun, Jan 8, 2017 at 9:53 AM, Alex Evonosky 
>> wrote:
>>
>>> Thank you for the reply!
>>>
>>> I followed this article:
>>>
>>> http://docs.ceph.com/docs/jewel/rados/operations/add-or-rm-mons/
>>>
>>>
>>> Under the section: ADDING A MONITOR (MANUAL)
>>>
>>>
>>>
>>> Alex F. Evonosky
>>>
>>> 
>>> 
>>>
>>> On Sat, Jan 7, 2017 at 6:36 PM, Shinobu Kinjo  wrote:
>>>
 How did you add a third MON?

 Regards,

 On Sun, Jan 8, 2017 at 7:01 AM, Alex Evonosky 
 wrote:
 > Anyone see this before?
 >
 >
 > 2017-01-07 16:55:11.406047 7f095b379700  0 cephx: verify_reply
 couldn't
 > decrypt with error: error decoding

Re: [ceph-users] Ceph Monitor cephx issues

2017-01-07 Thread Alex Evonosky

Thank you..


After sending the post, I totally removed the mon and issued the build with
ceph-deploy:


In the logs now:

2017-01-07 21:12:38.113534 7fa9613fd700  0 cephx: verify_reply couldn't
decrypt with error: error decoding block for decryption
2017-01-07 21:12:38.113546 7fa9613fd700  0 -- 10.10.10.138:6789/0 >>
10.10.10.103:6789/0 pipe(0x55feb2e9 sd=12 :50266 s=1 pgs=0 cs=0 l=0
c=0x55feb2ca0a80).failed verifying authorize reply
2017-01-07 21:12:38.114529 7fa95787b700  0 cephx: verify_reply couldn't
decrypt with error: error decoding block for decryption
2017-01-07 21:12:38.114567 7fa95787b700  0 -- 10.10.10.138:6789/0 >>
10.10.10.252:6789/0 pipe(0x55feb2e91400 sd=11 :38690 s=1 pgs=0 cs=0 l=0
c=0x55feb2ca0c00).failed verifying authorize reply
2017-01-07 21:12:40.114522 7fa9613fd700  0 cephx: verify_reply couldn't
decrypt with error: error decoding block for decryption
2017-01-07 21:12:40.114542 7fa9613fd700  0 -- 10.10.10.138:6789/0 >>
10.10.10.103:6789/0 pipe(0x55feb2e9 sd=11 :50278 s=1 pgs=0 cs=0 l=0
c=0x55feb2ca0a80).failed verifying authorize reply
2017-01-07 21:12:40.115706 7fa95787b700  0 cephx: verify_reply couldn't
decrypt with error: error decoding block for decryption
2017-01-07 21:12:40.115721 7fa95787b700  0 -- 10.10.10.138:6789/0 >>
10.10.10.252:6789/0 pipe(0x55feb2e91400 sd=12 :38702 s=1 pgs=0 cs=0 l=0
c=0x55feb2ca0c00).failed verifying authorize reply
2017-01-07 21:12:41.621916 7fa956f79700  0 cephx: verify_authorizer could
not decrypt ticket info: error: NSS AES final round failed: -8190
2017-01-07 21:12:41.621929 7fa956f79700  0 mon.alex-desktop@1(probing) e0
ms_verify_authorizer bad authorizer from mon 10.10.10.103:6789/0
2017-01-07 21:12:41.621944 7fa956f79700  0 -- 10.10.10.138:6789/0 >>
10.10.10.103:6789/0 pipe(0x55feb2fb5400 sd=21 :6789 s=0 pgs=0 cs=0 l=0
c=0x55feb2ca1500).accept: got bad authorizer



$ sudo ceph -s
cluster f5aba719-4856-4ae2-a5d4-f9ff0f614b60
 health HEALTH_WARN
512 pgs degraded
348 pgs stale
512 pgs stuck unclean
512 pgs undersized
6 requests are blocked > 32 sec
recovery 25013/50026 objects degraded (50.000%)
mds cluster is degraded
1 mons down, quorum 0,2 alpha,toshiba-laptop
 monmap e17: 3 mons at {alex-desktop=
10.10.10.138:6789/0,alpha=10.10.10.103:6789/0,toshiba-laptop=10.10.10.252:6789/0
}
election epoch 806, quorum 0,2 alpha,toshiba-laptop
  fsmap e201858: 1/1/1 up {0=1=up:replay}
 osdmap e200229: 3 osds: 2 up, 2 in; 85 remapped pgs
flags sortbitwise
  pgmap v4088774: 512 pgs, 4 pools, 50883 MB data, 25013 objects
59662 MB used, 476 GB / 563 GB avail
25013/50026 objects degraded (50.000%)
 348 stale+active+undersized+degraded
 164 active+undersized+degraded



root@alex-desktop:/var/lib/ceph/mon/ceph-alex-desktop# ls -ls
total 8
0 -rw-r--r-- 1 ceph ceph0 Jan  7 21:11 done
4 -rw--- 1 ceph ceph   77 Jan  7 21:05 keyring
4 drwxr-xr-x 2 ceph ceph 4096 Jan  7 21:10 store.db
0 -rw-r--r-- 1 ceph ceph0 Jan  7 21:05 systemd




Very odd...  never seen this issue on the other monitor deployments...








Alex F. Evonosky

 

On Sat, Jan 7, 2017 at 8:54 PM, Shinobu Kinjo  wrote:

> Using ``ceph-deploy`` will save your life:
>
>  # https://github.com/ceph/ceph/blob/master/doc/start/quick-
> ceph-deploy.rst
>   * Please look at: Adding Monitors
>
> If you are using centos or similar, the latest package is available here:
>
>  # http://download.ceph.com/rpm-jewel/el7/noarch/ceph-deploy-
> 1.5.37-0.noarch.rpm
>
> Regards,
>
>
> On Sun, Jan 8, 2017 at 9:53 AM, Alex Evonosky 
> wrote:
>
>> Thank you for the reply!
>>
>> I followed this article:
>>
>> http://docs.ceph.com/docs/jewel/rados/operations/add-or-rm-mons/
>>
>>
>> Under the section: ADDING A MONITOR (MANUAL)
>>
>>
>>
>> Alex F. Evonosky
>>
>>  
>>
>> On Sat, Jan 7, 2017 at 6:36 PM, Shinobu Kinjo  wrote:
>>
>>> How did you add a third MON?
>>>
>>> Regards,
>>>
>>> On Sun, Jan 8, 2017 at 7:01 AM, Alex Evonosky 
>>> wrote:
>>> > Anyone see this before?
>>> >
>>> >
>>> > 2017-01-07 16:55:11.406047 7f095b379700  0 cephx: verify_reply couldn't
>>> > decrypt with error: error decoding block for decryption
>>> > 2017-01-07 16:55:11.406053 7f095b379700  0 -- 10.10.10.138:6789/0 >>
>>> > 10.10.10.252:6789/0 pipe(0x55cf8d028000 sd=11 :47548 s=1 pgs=0 cs=0
>>> l=0
>>> > c=0x55cf8ce28f00).failed verifying authorize reply
>>> >
>>> >
>>> >
>>> > Two monitors are up just fine, just trying to add a third and a quorum
>>> > cannot be met.  NTP is running and no iptables running at all on
>>> internal
>>> > cluster.
>>> >
>>> >
>>> > Thank you.
>>> > -Alex
>>> >
>>> >
>>> >
>>> >

Re: [ceph-users] Ceph Monitor cephx issues

2017-01-07 Thread Shinobu Kinjo

Using ``ceph-deploy`` will save your life:

 # https://github.com/ceph/ceph/blob/master/doc/start/quick-ceph-deploy.rst
  * Please look at: Adding Monitors

If you are using centos or similar, the latest package is available here:

 #
http://download.ceph.com/rpm-jewel/el7/noarch/ceph-deploy-1.5.37-0.noarch.rpm

Regards,


On Sun, Jan 8, 2017 at 9:53 AM, Alex Evonosky 
wrote:

> Thank you for the reply!
>
> I followed this article:
>
> http://docs.ceph.com/docs/jewel/rados/operations/add-or-rm-mons/
>
>
> Under the section: ADDING A MONITOR (MANUAL)
>
>
>
> Alex F. Evonosky
>
>  
>
> On Sat, Jan 7, 2017 at 6:36 PM, Shinobu Kinjo  wrote:
>
>> How did you add a third MON?
>>
>> Regards,
>>
>> On Sun, Jan 8, 2017 at 7:01 AM, Alex Evonosky 
>> wrote:
>> > Anyone see this before?
>> >
>> >
>> > 2017-01-07 16:55:11.406047 7f095b379700  0 cephx: verify_reply couldn't
>> > decrypt with error: error decoding block for decryption
>> > 2017-01-07 16:55:11.406053 7f095b379700  0 -- 10.10.10.138:6789/0 >>
>> > 10.10.10.252:6789/0 pipe(0x55cf8d028000 sd=11 :47548 s=1 pgs=0 cs=0 l=0
>> > c=0x55cf8ce28f00).failed verifying authorize reply
>> >
>> >
>> >
>> > Two monitors are up just fine, just trying to add a third and a quorum
>> > cannot be met.  NTP is running and no iptables running at all on
>> internal
>> > cluster.
>> >
>> >
>> > Thank you.
>> > -Alex
>> >
>> >
>> >
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Monitor cephx issues

2017-01-07 Thread Alex Evonosky

Thank you for the reply!

I followed this article:

http://docs.ceph.com/docs/jewel/rados/operations/add-or-rm-mons/


Under the section: ADDING A MONITOR (MANUAL)



Alex F. Evonosky

 

On Sat, Jan 7, 2017 at 6:36 PM, Shinobu Kinjo  wrote:

> How did you add a third MON?
>
> Regards,
>
> On Sun, Jan 8, 2017 at 7:01 AM, Alex Evonosky 
> wrote:
> > Anyone see this before?
> >
> >
> > 2017-01-07 16:55:11.406047 7f095b379700  0 cephx: verify_reply couldn't
> > decrypt with error: error decoding block for decryption
> > 2017-01-07 16:55:11.406053 7f095b379700  0 -- 10.10.10.138:6789/0 >>
> > 10.10.10.252:6789/0 pipe(0x55cf8d028000 sd=11 :47548 s=1 pgs=0 cs=0 l=0
> > c=0x55cf8ce28f00).failed verifying authorize reply
> >
> >
> >
> > Two monitors are up just fine, just trying to add a third and a quorum
> > cannot be met.  NTP is running and no iptables running at all on internal
> > cluster.
> >
> >
> > Thank you.
> > -Alex
> >
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Monitor cephx issues

2017-01-07 Thread Shinobu Kinjo

How did you add a third MON?

Regards,

On Sun, Jan 8, 2017 at 7:01 AM, Alex Evonosky  wrote:
> Anyone see this before?
>
>
> 2017-01-07 16:55:11.406047 7f095b379700  0 cephx: verify_reply couldn't
> decrypt with error: error decoding block for decryption
> 2017-01-07 16:55:11.406053 7f095b379700  0 -- 10.10.10.138:6789/0 >>
> 10.10.10.252:6789/0 pipe(0x55cf8d028000 sd=11 :47548 s=1 pgs=0 cs=0 l=0
> c=0x55cf8ce28f00).failed verifying authorize reply
>
>
>
> Two monitors are up just fine, just trying to add a third and a quorum
> cannot be met.  NTP is running and no iptables running at all on internal
> cluster.
>
>
> Thank you.
> -Alex
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Analysing ceph performance with SSD journal, 10gbe NIC and 2 replicas -Hammer release

2017-01-07 Thread Nick Fisk

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of kevin 
parrikar
Sent: 07 January 2017 13:11
To: Lionel Bouton 
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Analysing ceph performance with SSD journal, 10gbe 
NIC and 2 replicas -Hammer release

Thanks for your valuable input.
We were using these SSD in our NAS box(synology)  and it was giving 13k iops 
for our fileserver in raid1.We had a few spare disks which we added to our ceph 
nodes hoping that it will give good performance same as that of NAS box.(i am 
not comparing NAS with ceph ,just the reason why we decided to use these SSD)

We dont have S3520 or S3610 at the moment but can order one of these to see how 
it performs in ceph .We have 4xS3500  80Gb handy.
If i create a 2 node cluster with 2xS3500 each and with replica of 2,do you 
think it can deliver 24MB/s of 4k writes .
We bought S3500 because last time when we tried ceph, people were suggesting 
this model :) :) 

Thanks alot for your help

Was is your application/use case for Ceph? 24MB/s of 4k writes will be quite a 
hard number to hit at lower queue depths. I saw your benchmark in your other 
post showing you were testing at a queue depth of 32, is this representative of 
your real life workload?

On Sat, Jan 7, 2017 at 6:01 PM, Lionel Bouton  > wrote:

Hi,

Le 07/01/2017 à 04:48, kevin parrikar a écrit :

i really need some help here :(

replaced all 7.2 rpm SAS disks with new Samsung 840 evo 512Gb SSD with no 
seperate journal Disk .Now both OSD nodes are with 2 ssd disks  with a replica 
of 2 . 
Total number of OSD process in the cluster is 4.with all SSD.

These SSDs are not designed for the kind of usage you are putting them through. 
The Evo and even the Pro line from Samsung can't write both fast and securely 
(ie : you can write fast and lose data if you get a power outage or you can 
write slow and keep your data, Ceph always makes sure your data is recoverable 
before completing a write : it is slow with these SSDs).

Christian already warned you about endurance and reliability, you just 
discovered the third problem : speed.

Lionel

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs AND rbds

2017-01-07 Thread Nick Fisk

Technically I think there is no reason why you couldn’t do this, but I think it 
is unadvisable. There was a similar thread a while back where somebody had done 
this and it caused problems when he was trying to do maintenance/recovery 
further down the line. 

 

I’m assuming you want to do this because you have already created a pool with 
the max number of PG’s per OSD and extra pools would take you further over this 
limit? If it’s the case I would just bump up the limit, it’s not worth the risk.

 

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of David 
Turner
Sent: 07 January 2017 00:54
To: ceph-users@lists.ceph.com
Subject: [ceph-users] cephfs AND rbds

 

Can cephfs and rbds use the same pool to store data?  I know you would need a 
separate metadata pool for cephfs, but could they share the same data pool?

  _  


  

David Turner | Cloud Operations Engineer |   
StorageCraft Technology Corporation
380 Data Drive Suite 300 | Draper | Utah | 84020
Office: 801.871.2760 | Mobile: 385.224.2943

  _  


If you are not the intended recipient of this message or received it 
erroneously, please notify the sender and delete it, together with any 
attachments, and be advised that any dissemination or copying of this message 
is prohibited.

  _  

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph Monitor cephx issues

2017-01-07 Thread Alex Evonosky

Anyone see this before?


2017-01-07 16:55:11.406047 7f095b379700  0 cephx: verify_reply couldn't
decrypt with error: error decoding block for decryption
2017-01-07 16:55:11.406053 7f095b379700  0 -- 10.10.10.138:6789/0 >>
10.10.10.252:6789/0 pipe(0x55cf8d028000 sd=11 :47548 s=1 pgs=0 cs=0 l=0
c=0x55cf8ce28f00).failed verifying authorize reply



Two monitors are up just fine, just trying to add a third and a quorum
cannot be met.  NTP is running and no iptables running at all on internal
cluster.


Thank you.
-Alex
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Analysing ceph performance with SSD journal, 10gbe NIC and 2 replicas -Hammer release

2017-01-07 Thread Jake Young

I use 2U servers with 9x 3.5" spinning disks in each. This has scaled well
for me, in both performance and  budget.

I may add 3 more spinning disks to each server at a later time if I need to
maximize storage, or I may add 3 SSDs for journals/cache tier if we need
better performance.

Another consideration is failure domain. If you had a server crash, how
much of your cluster will go down?  Some good advice I've read on this
forum is no single OSD server should be more than 10% of the cluster.

I had taken a week off and one of my 12 OSD servers had an OS SD card fail,
which took down the server. No one even noticed it went down. None of the
VM clients had any performance issues and no data was lost (3x
replication). I have the recovery settings turned down as low as possible,
and even so it only took about 6 hours to rebuild.

Speaking of rebuilding, do your performance measurements during a rebuild.
This has been the time when the cluster is the most stressed and when
performance is the most important.

There's a lot to think about. Read through the archives of this mailing
list, there is a lot of useful advice!

Jake


On Sat, Jan 7, 2017 at 1:38 PM Maged Mokhtar  wrote:

>
>
> Adding more nodes is best if you have unlimited budget :)
> You should add more osds per node until you start hitting cpu or network
> bottlenecks. Use a perf tool like atop/sysstat to know when this happens.
>
>
>
>
>  Original message 
> From: kevin parrikar 
> Date: 07/01/2017 19:56 (GMT+02:00)
> To: Lionel Bouton 
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Analysing ceph performance with SSD journal,
> 10gbe NIC and 2 replicas -Hammer release
>
> Wow thats a lot of good information. I wish i knew about all these before
> investing on all these devices.Since i dont have any other option,will get
> better SSD and faster HDD .
> I have one more generic question about Ceph.
> To increase the throughput of a cluster what is the standard practice is
> it more osd "per" node or more osd "nodes".
>
> Thanks alot for all your help.Learned so many new things thanks again
>
> Kevin
>
> On Sat, Jan 7, 2017 at 7:33 PM, Lionel Bouton <
> lionel-subscript...@bouton.name> wrote:
>
>
>
>
>
>
> Le 07/01/2017 à 14:11, kevin parrikar a
> écrit :
>
>
>
> Thanks for your valuable input.
>
> We were using these SSD in our NAS box(synology)  and it was
> giving 13k iops for our fileserver in raid1.We had a few spare
> disks which we added to our ceph nodes hoping that it will give
> good performance same as that of NAS box.(i am not comparing NAS
> with ceph ,just the reason why we decided to use these SSD)
>
>
>
> We dont have S3520 or S3610 at
> the moment but can order one of these to see how it performs
> in ceph .We have 4xS3500  80Gb handy.
>
> If i create a 2 node cluster with 2xS3500 each and with
> replica of 2,do you think it can deliver 24MB/s of 4k writes .
>
>
>
>
>
> Probably not. See
>
> http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/
>
>
>
> According to the page above the DC S3500 reaches 39MB/s. Its
> capacity isn't specified, yours are 80GB only which is the lowest
> capacity I'm aware of and for all DC models I know of the speed goes
> down with the capacity so you probably will get lower than that.
>
> If you put both data and journal on the same device you cut your
> bandwidth in half : so this would give you an average <20MB/s per
> OSD (with occasional peaks above that if you don't have a sustained
> 20MB/s). With 4 OSDs and size=2, your total write bandwidth is
> <40MB/s. For a single stream of data you will only get <20MB/s
> though (you won't benefit from parallel writes to the 4 OSDs and
> will only write on 2 at a time).
>
>
>
> Not that by comparison the 250GB 840 EVO only reaches 1.9MB/s.
>
>
>
> But even if you reach the 40MB/s, these models are not designed for
> heavy writes, you will probably kill them long before their warranty
> is expired (IIRC these are rated for ~24GB writes per day over the
> warranty period). In your configuration you only have to write 24G
> each day (as you have 4 of them, write both to data and journal and
> size=2) to be in this situation (this is an average of only 0.28
> MB/s compared to your 24 MB/s target).
>
>
>
>
> We bought S3500
> because last time when we tried ceph, people were suggesting
> this model :) :)
>
>
>
>
>
> The 3500 series might be enough with the higher capacities in some
> rare cases but the 80GB model is almost useless.
>
>
>
> You have to do the math considering :
>
> - how much you will write to the cluster (guess high if you have to
> guess),
>
> - if you will use the SSD for both journals and data (which means
> writing twice on them),
>
> - your replication level (which means you will write multiple times
> the same data),
>
> - when you expect to replace the hardware,
>
> - the amount

Re: [ceph-users] Analysing ceph performance with SSD journal, 10gbe NIC and 2 replicas -Hammer release

2017-01-07 Thread Maged Mokhtar



Adding more nodes is best if you have unlimited budget :)You should add more 
osds per node until you start hitting cpu or network bottlenecks. Use a perf 
tool like atop/sysstat to know when this happens.



 Original message 
From: kevin parrikar  
Date: 07/01/2017  19:56  (GMT+02:00) 
To: Lionel Bouton  
Cc: ceph-users@lists.ceph.com 
Subject: Re: [ceph-users] Analysing ceph performance with SSD journal, 10gbe 
NIC and 2 replicas -Hammer release 

Wow thats a lot of good information. I wish i knew about all these before 
investing on all these devices.Since i dont have any other option,will get 
better SSD and faster HDD .
I have one more generic question about Ceph.
To increase the throughput of a cluster what is the standard practice is it 
more osd "per" node or more osd "nodes".

Thanks alot for all your help.Learned so many new things thanks again

Kevin
On Sat, Jan 7, 2017 at 7:33 PM, Lionel Bouton  
wrote:

  

  
  
Le 07/01/2017 à 14:11, kevin parrikar a
  écrit :



  Thanks for your valuable input.

We were using these SSD in our NAS box(synology)  and it was
giving 13k iops for our fileserver in raid1.We had a few spare
disks which we added to our ceph nodes hoping that it will give
good performance same as that of NAS box.(i am not comparing NAS
with ceph ,just the reason why we decided to use these SSD)



We dont have S3520 or S3610 at
  the moment but can order one of these to see how it performs
  in ceph .We have 4xS3500  80Gb handy.

  If i create a 2 node cluster with 2xS3500 each and with
  replica of 2,do you think it can deliver 24MB/s of 4k writes .





Probably not. See
http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/



According to the page above the DC S3500 reaches 39MB/s. Its
capacity isn't specified, yours are 80GB only which is the lowest
capacity I'm aware of and for all DC models I know of the speed goes
down with the capacity so you probably will get lower than that.

If you put both data and journal on the same device you cut your
bandwidth in half : so this would give you an average <20MB/s per
OSD (with occasional peaks above that if you don't have a sustained
20MB/s). With 4 OSDs and size=2, your total write bandwidth is
<40MB/s. For a single stream of data you will only get <20MB/s
though (you won't benefit from parallel writes to the 4 OSDs and
will only write on 2 at a time).



Not that by comparison the 250GB 840 EVO only reaches 1.9MB/s.



But even if you reach the 40MB/s, these models are not designed for
heavy writes, you will probably kill them long before their warranty
is expired (IIRC these are rated for ~24GB writes per day over the
warranty period). In your configuration you only have to write 24G
each day (as you have 4 of them, write both to data and journal and
size=2) to be in this situation (this is an average of only 0.28
MB/s compared to your 24 MB/s target).




  We bought S3500
  because last time when we tried ceph, people were suggesting
  this model :) :) 





The 3500 series might be enough with the higher capacities in some
rare cases but the 80GB model is almost useless.



You have to do the math considering :

- how much you will write to the cluster (guess high if you have to
guess),

- if you will use the SSD for both journals and data (which means
writing twice on them),

- your replication level (which means you will write multiple times
the same data),

- when you expect to replace the hardware,

- the amount of writes per day they support under warranty (if the
manufacturer doesn't present this number prominently they probably
are trying to sell you a fast car headed for a brick wall)



If your hardware can't handle the amount of write you expect to put
in it then you are screwed. There were reports of new Ceph users not
aware of this and using cheap SSDs that failed in a matter of months
all at the same time. You definitely don't want to be in their
position.

In fact as problems happen (hardware failure leading to cluster
storage rebalancing for example) you should probably get a system
able to handle 10x the amount of writes you expect it to handle and
then monitor the SSD SMART attributes to be alerted long before they
die and replace them before problems happen. You definitely want a
controller allowing access to this information. If you can't you
will have to monitor the writes and guess this value which is risky
as write amplification inside SSDs is not easy to guess...

Re: [ceph-users] Analysing ceph performance with SSD journal, 10gbe NIC and 2 replicas -Hammer release

2017-01-07 Thread Lionel Bouton

Le 07/01/2017 à 14:11, kevin parrikar a écrit :
> Thanks for your valuable input.
> We were using these SSD in our NAS box(synology)  and it was giving
> 13k iops for our fileserver in raid1.We had a few spare disks which we
> added to our ceph nodes hoping that it will give good performance same
> as that of NAS box.(i am not comparing NAS with ceph ,just the reason
> why we decided to use these SSD)
>
> We dont have S3520 or S3610 at the moment but can order one of these
> to see how it performs in ceph .We have 4xS3500  80Gb handy.
> If i create a 2 node cluster with 2xS3500 each and with replica of
> 2,do you think it can deliver 24MB/s of 4k writes .

Probably not. See
http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/

According to the page above the DC S3500 reaches 39MB/s. Its capacity
isn't specified, yours are 80GB only which is the lowest capacity I'm
aware of and for all DC models I know of the speed goes down with the
capacity so you probably will get lower than that.
If you put both data and journal on the same device you cut your
bandwidth in half : so this would give you an average <20MB/s per OSD
(with occasional peaks above that if you don't have a sustained 20MB/s).
With 4 OSDs and size=2, your total write bandwidth is <40MB/s. For a
single stream of data you will only get <20MB/s though (you won't
benefit from parallel writes to the 4 OSDs and will only write on 2 at a
time).

Not that by comparison the 250GB 840 EVO only reaches 1.9MB/s.

But even if you reach the 40MB/s, these models are not designed for
heavy writes, you will probably kill them long before their warranty is
expired (IIRC these are rated for ~24GB writes per day over the warranty
period). In your configuration you only have to write 24G each day (as
you have 4 of them, write both to data and journal and size=2) to be in
this situation (this is an average of only 0.28 MB/s compared to your 24
MB/s target).

> We bought S3500 because last time when we tried ceph, people were
> suggesting this model :) :)

The 3500 series might be enough with the higher capacities in some rare
cases but the 80GB model is almost useless.

You have to do the math considering :
- how much you will write to the cluster (guess high if you have to guess),
- if you will use the SSD for both journals and data (which means
writing twice on them),
- your replication level (which means you will write multiple times the
same data),
- when you expect to replace the hardware,
- the amount of writes per day they support under warranty (if the
manufacturer doesn't present this number prominently they probably are
trying to sell you a fast car headed for a brick wall)

If your hardware can't handle the amount of write you expect to put in
it then you are screwed. There were reports of new Ceph users not aware
of this and using cheap SSDs that failed in a matter of months all at
the same time. You definitely don't want to be in their position.
In fact as problems happen (hardware failure leading to cluster storage
rebalancing for example) you should probably get a system able to handle
10x the amount of writes you expect it to handle and then monitor the
SSD SMART attributes to be alerted long before they die and replace them
before problems happen. You definitely want a controller allowing access
to this information. If you can't you will have to monitor the writes
and guess this value which is risky as write amplification inside SSDs
is not easy to guess...

Lionel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Analysing ceph performance with SSD journal, 10gbe NIC and 2 replicas -Hammer release

2017-01-07 Thread kevin parrikar

Thanks for your valuable input.
We were using these SSD in our NAS box(synology)  and it was giving 13k
iops for our fileserver in raid1.We had a few spare disks which we added to
our ceph nodes hoping that it will give good performance same as that of
NAS box.(i am not comparing NAS with ceph ,just the reason why we decided
to use these SSD)

We dont have S3520 or S3610 at the moment but can order one of these to see
how it performs in ceph .We have 4xS3500  80Gb handy.
If i create a 2 node cluster with 2xS3500 each and with replica of 2,do you
think it can deliver 24MB/s of 4k writes .
We bought S3500 because last time when we tried ceph, people were
suggesting this model :) :)

Thanks alot for your help

On Sat, Jan 7, 2017 at 6:01 PM, Lionel Bouton <
lionel-subscript...@bouton.name> wrote:

> Hi,
>
> Le 07/01/2017 à 04:48, kevin parrikar a écrit :
>
> i really need some help here :(
>
> replaced all 7.2 rpm SAS disks with new Samsung 840 evo 512Gb SSD with no
> seperate journal Disk .Now both OSD nodes are with 2 ssd disks  with a
> replica of *2* .
> Total number of OSD process in the cluster is *4*.with all SSD.
>
>
> These SSDs are not designed for the kind of usage you are putting them
> through. The Evo and even the Pro line from Samsung can't write both fast
> and securely (ie : you can write fast and lose data if you get a power
> outage or you can write slow and keep your data, Ceph always makes sure
> your data is recoverable before completing a write : it is slow with these
> SSDs).
>
> Christian already warned you about endurance and reliability, you just
> discovered the third problem : speed.
>
> Lionel
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD mirroring

2017-01-07 Thread Klemen Pogacnik

Yes, disaster recovery can be solved by application layer, but I think it
would be nice openstack feature too. Specially when the replication is
already solved by Ceph. I'll ask on other forums if something is doing on
that feature. Thanks again for pointing me to the right direction.
Kemo

On Fri, Jan 6, 2017 at 2:59 PM, Jason Dillaman  wrote:

> In all honesty, this unfortunately isn't one of my areas of expertise.
>
> OpenStack has such a large umbrella and is moving too quickly for me
> to stay 100% up-to-date. I don't think, from a cloud point-of-view,
> this is a problem too many purists are concerned about since a
> cloud-native app should be able to survive failures -- and any
> necessary replication of data should be handled by the
> application-layer instead of the infrastructure.  Also -- it's
> definitely a hard problem to solve in a generic fashion.
>
> On Fri, Jan 6, 2017 at 8:27 AM, Klemen Pogacnik  wrote:
> > That I was afraid of. So there isn't any commands available and I must
> > somehow synchronize Cinder DB, to get access to volumes also on second
> site.
> > Do you maybe know, if somebody is already thinking or even working on
> that?
> > On presentation Kingbird project was mentioned, but I'm not sure if their
> > work will solve this problem.
> > Kemo
> >
> > On Thu, Jan 5, 2017 at 4:45 PM, Jason Dillaman 
> wrote:
> >>
> >> On Thu, Jan 5, 2017 at 7:24 AM, Klemen Pogacnik 
> wrote:
> >> > I'm playing with rbd mirroring with openstack. The final idea is to
> use
> >> > it
> >> > for disaster recovery of DB server running on Openstack cluster, but
> >> > would
> >> > like to test this functionality first.
> >> > I've prepared this configuration:
> >> > - 2 openstack clusters (devstacks)
> >> > - 2 ceph clusters (one node clusters)
> >> > Remote Ceph is used as a backend for Cinder service. Each devstack has
> >> > its
> >> > own Ceph cluster. Mirroring was enabled for volumes pool, and
> rbd-mirror
> >> > daemon was started.
> >> > When I create new cinder volume on devstack1, the same rbd storage
> >> > appeared
> >> > on both Ceph clusters, so it seems, mirroring is working.
> >> > Now I would like to see this storage as a Cinder volume on devstack2
> >> > too. Is
> >> > it somehow possible to do that?
> >>
> >> This level is HA/DR is not currently built-in to OpenStack (it's
> >> outside the scope of Ceph). There are several strategies you could use
> >> to try to replicate the devstack1 database to devstack2. Here is a
> >> presentation from OpenStack Summit Austin [1] re: this subject.
> >>
> >> > The next question is, how to make switchover. On Ceph it can easily be
> >> > done
> >> > by demote and promote commands, but then the volumes are still not
> seen
> >> > on
> >> > Devstack2, so I can't use it.
> >> > On open stack there is cinder failover-host command, which is, as I
> can
> >> > understand, only useful for configuration with one openstack and two
> >> > ceph
> >> > clusters. Any idea how to make switchover with my configuration.
> >> > Thanks a lot for help!
> >>
> >> Correct -- Cinder's built-in volume replication feature is just a set
> >> of hooks available for backends that already support
> >> replication/mirroring. The hooks for Ceph RBD are scheduled to be
> >> included in the next release of OpenStack, but as you have discovered,
> >> it really only protects against a storage failure (where you can
> >> switch from Ceph cluster A to Ceph cluster B), but does not help with
> >> losing your OpenStack data center.
> >>
> >> > Kemo
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > ___
> >> > ceph-users mailing list
> >> > ceph-users@lists.ceph.com
> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >
> >>
> >> [1]
> >> https://www.openstack.org/videos/video/protecting-the-
> galaxy-multi-region-disaster-recovery-with-openstack-and-ceph
> >>
> >> --
> >> Jason
> >
> >
>
>
>
> --
> Jason
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Analysing ceph performance with SSD journal, 10gbe NIC and 2 replicas -Hammer release

2017-01-07 Thread kevin parrikar

Thanks Maged for your suggestion.

I have executed rbd bench and here is the result,please have a look at it


rbd bench-write image01  --pool=rbd --io-threads=32 --io-size 4096
--io-pattern rand --rbd_cache=false

bench-write  io_size 4096 io_threads 32 bytes 1073741824 pattern rand
  SEC   OPS   OPS/SEC   BYTES/SEC
1  4750   4750.19  19456758.28
2  7152   3068.49  12568516.09
4  7220   1564.41  6407837.20
5  8941   1794.35  7349666.74
6 11938   1994.94  8171294.61
7 12932   1365.21  5591891.85
^C

not sure why it skipped "3" from SEC .

I suppose this also shows slow performance.


Any idea where could be the issue?

I use LSI 9260-4i controller (firmware 12.13.0.-0154) on both the nodes
with write back enabled . i am not sure if this controller is suitable for
ceph.

Regards,
Kevin

On Sat, Jan 7, 2017 at 1:23 PM, Maged Mokhtar  wrote:

> The numbers are very low. I would first benchmark the system without the
> vm client using rbd 4k test such as:
>
> rbd bench-write image01  --pool=rbd --io-threads=32 --io-size 4096
> --io-pattern rand --rbd_cache=false
>
>
>
>  Original message 
> From: kevin parrikar 
> Date: 07/01/2017 05:48 (GMT+02:00)
> To: Christian Balzer 
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Analysing ceph performance with SSD journal,
> 10gbe NIC and 2 replicas -Hammer release
>
> i really need some help here :(
>
> replaced all 7.2 rpm SAS disks with new Samsung 840 evo 512Gb SSD with no
> seperate journal Disk .Now both OSD nodes are with 2 ssd disks  with a
> replica of *2* .
> Total number of OSD process in the cluster is *4*.with all SSD.
>
>
> But throughput has gone down from 1.4 MB/s to 1.3 MB/s for 4k writes and
> for 4M it has gone down from 140MB/s to 126MB/s .
>
> now atop no longer shows OSD device as 100% busy..
>
> How ever i can see both ceph-osd process in atop with 53% and 47% disk
> utilization.
>
>  PID RDDSK  WRDSK   WCANCL
> DSK CMD1/2
> 20771  0K648.8M 0K
>   53%ceph-osd
> 19547  0K576.7M 0K
>   47%ceph-osd
>
>
> OSD disks(ssd) utilization from atop
>
> DSK |  sdc | busy  6%  | read  0  | write  517  | KiB/r   0  | KiB/w  293
> | MBr/s 0.00  | MBw/s 148.18  | avq   9.44  | avio 0.12 ms  |
>
> DSK |  sdd | busy   5% | read   0 | write   336 | KiB/r   0  | KiB/w   292
> | MBr/s 0.00 | MBw/s  96.12  | avq 7.62  | avio 0.15 ms  |
>
>
> Queue Depth of OSD disks
>  cat /sys/block/sdd/device//queue_depth
> 256
>
> atop inside virtual machine:[4 CPU/3Gb RAM]
> DSK |   vdc  | busy 96%  | read 0  | write  256  | KiB/r   0  |
> KiB/w  512  | MBr/s   0.00  | MBw/s 128.00  | avq7.96  | avio 3.77 ms  |
>
>
> Both Guest and Host are using deadline I/O scheduler
>
>
> Virtual Machine Configuration:
>
>  
> 
>   
>   
> 
>   
>   
> 
> 
> 
>   
>   
>   449da0e7-6223-457c-b2c6-b5e112099212
>function='0x0'/>
> 
>
>
>
> ceph.conf
>
>  cat /etc/ceph/ceph.conf
>
> [global]
> fsid = c4e1a523-9017-492e-9c30-8350eba1bd51
> mon_initial_members = node-16 node-30 node-31
> mon_host = 172.16.1.11 172.16.1.12 172.16.1.8
> auth_cluster_required = cephx
> auth_service_required = cephx
> auth_client_required = cephx
> filestore_xattr_use_omap = true
> log_to_syslog_level = info
> log_to_syslog = True
> osd_pool_default_size = 2
> osd_pool_default_min_size = 1
> osd_pool_default_pg_num = 64
> public_network = 172.16.1.0/24
> log_to_syslog_facility = LOG_LOCAL0
> osd_journal_size = 2048
> auth_supported = cephx
> osd_pool_default_pgp_num = 64
> osd_mkfs_type = xfs
> cluster_network = 172.16.1.0/24
> osd_recovery_max_active = 1
> osd_max_backfills = 1
>
>
> [client]
> rbd_cache_writethrough_until_flush = True
> rbd_cache = True
>
> [client.radosgw.gateway]
> rgw_keystone_accepted_roles = _member_, Member, admin, swiftoperator
> keyring = /etc/ceph/keyring.radosgw.gateway
> rgw_frontends = fastcgi socket_port=9000 socket_host=127.0.0.1
> rgw_socket_path = /tmp/radosgw.sock
> rgw_keystone_revocation_interval = 100
>
> Any guidance on where to look for issues.
>
> Regards,
> Kevin
>
> On Fri, Jan 6, 2017 at 4:42 PM, kevin parrikar 
> wrote:
>
>> Thanks Christian for your valuable comments,each comment is a new
>> learning for me.
>> Please see inline
>>
>> On Fri, Jan 6, 2017 at 9:32 AM, Christian Balzer  wrote:
>>
>>>
>>> Hello,
>>>
>>> On Fri, 6 Jan 2017 08:40:36 +0530 kevin parrikar wrote:
>>>
>>> > Hello All,
>>> >
>>> > I have setup a ceph cluster based on 0.94.6 release in  2 servers each
>>> with
>>> > 80Gb intel s3510 and 2x3 Tb 7.2 SATA disks,16 CPU,24G RAM
>>> > which is connected to a 10G switch with a replica of 2 [ i will add 3

Re: [ceph-users] Ceph Monitor cephx issues

Re: [ceph-users] Ceph Monitor cephx issues

Re: [ceph-users] Ceph Monitor cephx issues

Re: [ceph-users] Ceph Monitor cephx issues

Re: [ceph-users] Ceph Monitor cephx issues

Re: [ceph-users] Ceph Monitor cephx issues

Re: [ceph-users] Analysing ceph performance with SSD journal, 10gbe NIC and 2 replicas -Hammer release

Re: [ceph-users] cephfs AND rbds

[ceph-users] Ceph Monitor cephx issues

Re: [ceph-users] Analysing ceph performance with SSD journal, 10gbe NIC and 2 replicas -Hammer release

Re: [ceph-users] Analysing ceph performance with SSD journal, 10gbe NIC and 2 replicas -Hammer release

Re: [ceph-users] Analysing ceph performance with SSD journal, 10gbe NIC and 2 replicas -Hammer release

Re: [ceph-users] Analysing ceph performance with SSD journal, 10gbe NIC and 2 replicas -Hammer release

Re: [ceph-users] RBD mirroring

Re: [ceph-users] Analysing ceph performance with SSD journal, 10gbe NIC and 2 replicas -Hammer release

15 matches

Site Navigation

Mail list logo

Footer information