Re: [ceph-users] RadosGW performance s3 many objects

[Vickey Singh]

Hello Guys

Is several millions of object with Ceph ( for RGW use case ) still an issue
?  Or is it fixed ?

Thnx
Vickey

On Thu, Jan 28, 2016 at 12:55 AM, Krzysztof Księżyk 
wrote:

> Stefan Rogge  writes:
>
> >
> >
> > Hi,
> > we are using the Ceph with RadosGW and S3 setting.
> > With more and more objects in the storage the writing speed slows down
> significantly. With 5 million object in the storage we had a writing speed
> of 10MS/s. With 10 million objects in the storage its only 5MB/s.
> > Is this a common issue?
> > Is the RadosGW suitable for a large amount of objects or would you
> recommend to not use the RadosGW with these amount of objects?
> >
> > Thank you.
> >
> > Stefan
> >
> > I found also a ticket at the ceph tracker with the same issue:
> >
> >
> http://tracker.ceph.com/projects/ceph/wiki/Rgw_-_bucket_index_scalability
> >
> > ___
> > ceph-users mailing list
> > ceph-users@...
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
> Hi,
>
> I'm struggling with the same issue on Ceph 9.2.0. Unfortunately I wasn't
> aware of it and now the only way to improve things is create new bucket
> with bucket index shrading or change way our apps store data into buckets.
> And of course copy tons of data :( In my case also sth happened to leveldb
> files and now I cannot even run some radosgw-admin commands like:
>
> radosgw-admin bucket check -b 
>
> what causes osd daemon flapping and process timeout messages in logs. PGS
> containing  .rgw.bucket.index  can't be even backfilled to other osd as osd
> process dies with messages:
>
> [...]
> > 2016-01-25 15:47:22.700737 7f79fc66d700  1 heartbeat_map is_healthy
> 'OSD::osd_op_tp thread 0x7f7992c86700' had suicide timed out after 150
> > 2016-01-25 15:47:22.702619 7f79fc66d700 -1 common/HeartbeatMap.cc: In
> function 'bool ceph::HeartbeatMap::_check(const ceph::heartbeat_handle_d*,
> const char*, time_t)' thread 7f79fc66d700 time 2016-01-25 15:47:22.700751
> > common/HeartbeatMap.cc: 81: FAILED assert(0 == "hit suicide timeout")
> >
> >  ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299)
> >  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x85) [0x7f7a019f4be5]
> >  2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char
> const*, long)+0x2d9) [0x7f7a019343b9]
> >  3: (ceph::HeartbeatMap::is_healthy()+0xd6) [0x7f7a01934bf6]
> >  4: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0x7f7a019353bc]
> >  5: (CephContextServiceThread::entry()+0x15b) [0x7f7a01a10dcb]
> >  6: (()+0x7df5) [0x7f79ffa8fdf5]
> >  7: (clone()+0x6d) [0x7f79fe3381ad]
> >
> >
> I don't know - maybe it's because number of leveldb files in omap folder
> (total 5.1GB). Read somewhere that things can be improved by setting
> 'leveldb_compression' to false and leveldb_compact_on_mount to true but I
> don't know if these options have any effect in 9.2.0 as they are not
> documented for this release. Tried with 'leveldb_compression' but without
> visible effect and wasn't brave enough with trying leveldb_compact_on_mount
> on production env. But setting it to true on my test 0.94.5 makes osd
> failing on restart.
>
> Kind regards -
> Krzysztof Księżyk
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Crush map customization for production use

[Vickey Singh]

Hello Geeks

Can someone please review and comment on my custom crush maps. I would
really appreciate your help


My setup :  1 Rack , 4 chassis , 3 storage nodes each chassis ( so total 12
storage nodes ) , pool size = 3

What i want to achieve is:
- Survive chassis failures , even if i loose 2 complete chassis (containing
3 nodes each) , data should not be lost
- The crush ruleset should store each copy on a unique chassis and host

For example :
copy 1 ---> c1-node1
copy 2 ---> c2-node3
copy 3 ---> c4-node2



Here is my crushmap
=

chassis block_storage_chassis_4 {
id -17 # do not change unnecessarily
# weight weight 163.350
alg straw
hash 0 # rjenkins1
item c4-node1 weight 54.450
item c4-node2 weight 54.450
item c4-node3 weight 54.450

}

chassis block_storage_chassis_3 {
id -16 # do not change unnecessarily
# weight weight 163.350
alg straw
hash 0 # rjenkins1
item c3-node1 weight 54.450
item c3-node2 weight 54.450
item c3-node3 weight 54.450

}

chassis block_storage_chassis_2 {
id -15 # do not change unnecessarily
# weight weight 163.350
alg straw
hash 0 # rjenkins1
item c2-node1 weight 54.450
item c2-node2 weight 54.450
item c3-node3 weight 54.450

}

chassis block_storage_chassis_1 {
id -14 # do not change unnecessarily
# weight 163.350
alg straw
hash 0 # rjenkins1
item c1-node1 weight 54.450
item c1-node2 weight 54.450
item c1-node3 weight 54.450

}

rack block_storage_rack_1 {
id -10 # do not change unnecessarily
# weight 174.240
alg straw
hash 0 # rjenkins1
item block_storage_chassis_1 weight 163.350
item block_storage_chassis_2 weight 163.350
item block_storage_chassis_3 weight 163.350
item block_storage_chassis_4 weight 163.350

}

class block_storage {
id -6 # do not change unnecessarily
# weight 210.540
alg straw
hash 0 # rjenkins1
item block_storage_rack_1 weight 656.400
}

rule ruleset_block_storage {
ruleset 1
type replicated
min_size 1
max_size 10
step take block_storage
step chooseleaf firstn 0 type chassis
step emit
}
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Incorrect output from ceph osd map command

[Vickey Singh]

Adding community for further help on this.

On Tue, Feb 23, 2016 at 10:57 PM, Vickey Singh <vickey.singh22...@gmail.com>
wrote:

>
>
> On Tue, Feb 23, 2016 at 9:53 PM, Gregory Farnum <gfar...@redhat.com>
> wrote:
>
>>
>>
>> On Tuesday, February 23, 2016, Vickey Singh <vickey.singh22...@gmail.com>
>> wrote:
>>
>>> Thanks Greg,
>>>
>>> Do you mean ceph osd map command is not displaying accurate information ?
>>>
>>> I guess, either of these things are happening with my cluster
>>> - ceph osd map is not printing true information
>>> - Object to PG mapping is not correct ( one object is mapped to multiple
>>> PG's )
>>>
>>> This is happening for several objects , but the cluster is Healthy.
>>>
>>
>> No, you're looking for the map command to do something it was not
>> designed for. If you want to see if an object exists, you will need to use
>> a RADOS client to fetch the object and see if it's there. "map" is a
>> mapping command: given an object name, which PG/OSD does CRUSH map that
>> name to?
>>
>
> well your 6th sense is amazing :)
>
> This is exactly i want to achieve , i wan to see my PG/OSD mapping for
> objects. ( basically i have changed my crush hierarchy , now i want to
> verify that no 2 objects should go to a single host / chassis / rack ) so
> to verify them i was using ceph osd map command.
>
> Is there a smarter way to achieve this ?
>
>
>
>
>
>>
>>
>>>
>>> Need expert suggestion.
>>>
>>>
>>> On Tue, Feb 23, 2016 at 7:20 PM, Gregory Farnum <gfar...@redhat.com>
>>> wrote:
>>>
>>>> This is not a bug. The map command just says which PG/OSD an object
>>>> maps to; it does not go out and query the osd to see if there actually is
>>>> such an object.
>>>> -Greg
>>>>
>>>>
>>>> On Tuesday, February 23, 2016, Vickey Singh <
>>>> vickey.singh22...@gmail.com> wrote:
>>>>
>>>>> Hello Guys
>>>>>
>>>>> I am getting wired output from osd map. The object does not exists on
>>>>> pool but osd map still shows its PG and OSD on which its stored.
>>>>>
>>>>> So i have rbd device coming from pool 'gold' , this image has an
>>>>> object 'rb.0.10f61.238e1f29.2ac5'
>>>>>
>>>>> The below commands verifies this
>>>>>
>>>>> *[root@ceph-node1 ~]# rados -p gold ls | grep -i
>>>>> rb.0.10f61.238e1f29.2ac5*
>>>>> *rb.0.10f61.238e1f29.2ac5*
>>>>> *[root@ceph-node1 ~]#*
>>>>>
>>>>> This object lives on pool gold and OSD 38,0,20 , which is correct
>>>>>
>>>>> *[root@ceph-node1 ~]# ceph osd map gold
>>>>> rb.0.10f61.238e1f29.2ac5*
>>>>> *osdmap e1357 pool 'gold' (1) object
>>>>> 'rb.0.10f61.238e1f29.2ac5' -> pg 1.11692600 (1.0) -> up 
>>>>> ([38,0,20],
>>>>> p38) acting ([38,0,20], p38)*
>>>>> *[root@ceph-node1 ~]#*
>>>>>
>>>>>
>>>>> Since i don't have object 'rb.0.10f61.238e1f29.2ac5' in data
>>>>> and rbd pools , rados ls will not list it. Which is expected.
>>>>>
>>>>> *[root@ceph-node1 ~]# rados -p data ls | grep -i
>>>>> rb.0.10f61.238e1f29.2ac5*
>>>>> *[root@ceph-node1 ~]# rados -p rbd ls | grep -i
>>>>> rb.0.10f61.238e1f29.2ac5*
>>>>>
>>>>>
>>>>> But , how come the object is showing in osd map of pool data and rbd.
>>>>>
>>>>> *[root@ceph-node1 ~]# ceph osd map data
>>>>> rb.0.10f61.238e1f29.2ac5*
>>>>> *osdmap e1357 pool 'data' (2) object
>>>>> 'rb.0.10f61.238e1f29.2ac5' -> pg 2.11692600 (2.0) -> up 
>>>>> ([3,51,29],
>>>>> p3) acting ([3,51,29], p3)*
>>>>> *[root@ceph-node1 ~]#*
>>>>>
>>>>> *[root@ceph-node1 ~]# ceph osd map rbd
>>>>> rb.0.10f61.238e1f29.2ac5*
>>>>> *osdmap e1357 pool 'rbd' (0) object 'rb.0.10f61.238e1f29.2ac5'
>>>>> -> pg 0.11692600 (0.0) -> up ([41,20,3], p41) acting ([41,20,3], p41)*
>>>>> *[root@ceph-node1 ~]#*
>>>>>
>>>>>
>>>>> In ceph, object is unique and belongs to only one pool. So why does it
>>>>> shows up in all pool's osd map.
>>>>>
>>>>> Is this some kind of BUG in Ceph
>>>>>
>>>>> Ceph Hammer 0.94.5
>>>>> CentOS 7.2
>>>>>
>>>>
>>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Incorrect output from ceph osd map command

[Vickey Singh]

Thanks Greg,

Do you mean ceph osd map command is not displaying accurate information ?

I guess, either of these things are happening with my cluster
- ceph osd map is not printing true information
- Object to PG mapping is not correct ( one object is mapped to multiple
PG's )

This is happening for several objects , but the cluster is Healthy.

Need expert suggestion.


On Tue, Feb 23, 2016 at 7:20 PM, Gregory Farnum <gfar...@redhat.com> wrote:

> This is not a bug. The map command just says which PG/OSD an object maps
> to; it does not go out and query the osd to see if there actually is such
> an object.
> -Greg
>
>
> On Tuesday, February 23, 2016, Vickey Singh <vickey.singh22...@gmail.com>
> wrote:
>
>> Hello Guys
>>
>> I am getting wired output from osd map. The object does not exists on
>> pool but osd map still shows its PG and OSD on which its stored.
>>
>> So i have rbd device coming from pool 'gold' , this image has an object
>> 'rb.0.10f61.238e1f29.2ac5'
>>
>> The below commands verifies this
>>
>> *[root@ceph-node1 ~]# rados -p gold ls | grep -i
>> rb.0.10f61.238e1f29.2ac5*
>> *rb.0.10f61.238e1f29.2ac5*
>> *[root@ceph-node1 ~]#*
>>
>> This object lives on pool gold and OSD 38,0,20 , which is correct
>>
>> *[root@ceph-node1 ~]# ceph osd map gold rb.0.10f61.238e1f29.2ac5*
>> *osdmap e1357 pool 'gold' (1) object 'rb.0.10f61.238e1f29.2ac5'
>> -> pg 1.11692600 (1.0) -> up ([38,0,20], p38) acting ([38,0,20], p38)*
>> *[root@ceph-node1 ~]#*
>>
>>
>> Since i don't have object 'rb.0.10f61.238e1f29.2ac5' in data and
>> rbd pools , rados ls will not list it. Which is expected.
>>
>> *[root@ceph-node1 ~]# rados -p data ls | grep -i
>> rb.0.10f61.238e1f29.2ac5*
>> *[root@ceph-node1 ~]# rados -p rbd ls | grep -i
>> rb.0.10f61.238e1f29.2ac5*
>>
>>
>> But , how come the object is showing in osd map of pool data and rbd.
>>
>> *[root@ceph-node1 ~]# ceph osd map data rb.0.10f61.238e1f29.2ac5*
>> *osdmap e1357 pool 'data' (2) object 'rb.0.10f61.238e1f29.2ac5'
>> -> pg 2.11692600 (2.0) -> up ([3,51,29], p3) acting ([3,51,29], p3)*
>> *[root@ceph-node1 ~]#*
>>
>> *[root@ceph-node1 ~]# ceph osd map rbd rb.0.10f61.238e1f29.2ac5*
>> *osdmap e1357 pool 'rbd' (0) object 'rb.0.10f61.238e1f29.2ac5' ->
>> pg 0.11692600 (0.0) -> up ([41,20,3], p41) acting ([41,20,3], p41)*
>> *[root@ceph-node1 ~]#*
>>
>>
>> In ceph, object is unique and belongs to only one pool. So why does it
>> shows up in all pool's osd map.
>>
>> Is this some kind of BUG in Ceph
>>
>> Ceph Hammer 0.94.5
>> CentOS 7.2
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Incorrect output from ceph osd map command

[Vickey Singh]

Hello Guys

I am getting wired output from osd map. The object does not exists on pool
but osd map still shows its PG and OSD on which its stored.

So i have rbd device coming from pool 'gold' , this image has an object
'rb.0.10f61.238e1f29.2ac5'

The below commands verifies this

*[root@ceph-node1 ~]# rados -p gold ls | grep -i
rb.0.10f61.238e1f29.2ac5*
*rb.0.10f61.238e1f29.2ac5*
*[root@ceph-node1 ~]#*

This object lives on pool gold and OSD 38,0,20 , which is correct

*[root@ceph-node1 ~]# ceph osd map gold rb.0.10f61.238e1f29.2ac5*
*osdmap e1357 pool 'gold' (1) object 'rb.0.10f61.238e1f29.2ac5' ->
pg 1.11692600 (1.0) -> up ([38,0,20], p38) acting ([38,0,20], p38)*
*[root@ceph-node1 ~]#*


Since i don't have object 'rb.0.10f61.238e1f29.2ac5' in data and
rbd pools , rados ls will not list it. Which is expected.

*[root@ceph-node1 ~]# rados -p data ls | grep -i
rb.0.10f61.238e1f29.2ac5*
*[root@ceph-node1 ~]# rados -p rbd ls | grep -i
rb.0.10f61.238e1f29.2ac5*


But , how come the object is showing in osd map of pool data and rbd.

*[root@ceph-node1 ~]# ceph osd map data rb.0.10f61.238e1f29.2ac5*
*osdmap e1357 pool 'data' (2) object 'rb.0.10f61.238e1f29.2ac5' ->
pg 2.11692600 (2.0) -> up ([3,51,29], p3) acting ([3,51,29], p3)*
*[root@ceph-node1 ~]#*

*[root@ceph-node1 ~]# ceph osd map rbd rb.0.10f61.238e1f29.2ac5*
*osdmap e1357 pool 'rbd' (0) object 'rb.0.10f61.238e1f29.2ac5' ->
pg 0.11692600 (0.0) -> up ([41,20,3], p41) acting ([41,20,3], p41)*
*[root@ceph-node1 ~]#*


In ceph, object is unique and belongs to only one pool. So why does it
shows up in all pool's osd map.

Is this some kind of BUG in Ceph

Ceph Hammer 0.94.5
CentOS 7.2
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Extra RAM to improve OSD write performance ?

[Vickey Singh]

Hello Community

Happy Valentines Day ;-)

I need some advice on using EXATA RAM on my OSD servers to improve Ceph's
write performance.

I have 20 OSD servers each with 256GB RAM and 6TB x 16 OSD's, so assuming
cluster is not recovering, most of the time system will have at least
~150GB RAM free. And for 20 machines its a lot ~3.0 TB RAM

Is there any way to use this free RAM to improve write performance of
cluster. Something like Linux page cache for OSD write operations.

I assume that by default Linux page cache can use free memory to improve
OSD read performance ( please correct me if i am wrong). But how about OSD
write improvement , How to improve that with free RAM.

PS : My Ceph cluster's workload is just OpenStack Cinder , Glance , Nova
for instance disk

- Vickey -
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Tips for faster openstack instance boot

[Vickey Singh]

Guys Thanks a lot for your response.

We are running OpenStack Juno + Ceph 94.5

@Jason Dillaman Can you please explain what do you mean by "Glance is
configured to cache your RBD image" ? This might give me some clue.

Many Thanks.


On Mon, Feb 8, 2016 at 10:33 PM, Jason Dillaman <dilla...@redhat.com> wrote:

> If Nova and Glance are properly configured, it should only require a quick
> clone of the Glance image to create your Nova ephemeral image.  Have you
> double-checked your configuration against the documentation [1]?  What
> version of OpenStack are you using?
>
> To answer your questions:
>
> > - From Ceph point of view. does COW works cross pool i.e. image from
> glance
> > pool ---> (cow) --> instance disk on nova pool
> Yes, cloning copy-on-write images works across pools
>
> > - Will a single pool for glance and nova instead of separate pool . will
> help
> > here ?
> Should be no change -- the creation of the clone is extremely lightweight
> (add the image to a directory, create a couple metadata objects)
>
> > - Is there any tunable parameter from Ceph or OpenStack side that should
> be
> > set ?
> I'd double-check your OpenStack configuration.  Perhaps Glance isn't
> configured with "show_image_direct_url = True", or Glance is configured to
> cache your RBD images, or you have an older OpenStack release that requires
> patches to fully support Nova+RBD.
>
> [1] http://docs.ceph.com/docs/master/rbd/rbd-openstack/
>
> --
>
> Jason Dillaman
>
>
> - Original Message -
>
> > From: "Vickey Singh" <vickey.singh22...@gmail.com>
> > To: ceph-users@lists.ceph.com, "ceph-users" <ceph-us...@ceph.com>
> > Sent: Monday, February 8, 2016 9:10:59 AM
> > Subject: [ceph-users] Tips for faster openstack instance boot
>
> > Hello Community
>
> > I need some guidance how can i reduce openstack instance boot time using
> Ceph
>
> > We are using Ceph Storage with openstack ( cinder, glance and nova ). All
> > OpenStack images and instances are being stored on Ceph in different
> pools
> > glance and nova pool respectively.
>
> > I assume that Ceph by default uses COW rbd , so for example if an
> instance is
> > launched using glance image (which is stored on Ceph) , Ceph should take
> COW
> > snapshot of glance image and map it as RBD disk for instance. And this
> whole
> > process should be very quick.
>
> > In our case , the instance launch is taking 90 seconds. Is this normal ?
> ( i
> > know this really depends one's infra , but still )
>
> > Is there any way , i can utilize Ceph's power and can launch instances
> ever
> > faster.
>
> > - From Ceph point of view. does COW works cross pool i.e. image from
> glance
> > pool ---> (cow) --> instance disk on nova pool
> > - Will a single pool for glance and nova instead of separate pool . will
> help
> > here ?
> > - Is there any tunable parameter from Ceph or OpenStack side that should
> be
> > set ?
>
> > Regards
> > Vickey
>
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Tips for faster openstack instance boot

[Vickey Singh]

Hello Community

I need some guidance how can i reduce openstack instance boot time using
Ceph

We are using Ceph Storage with openstack ( cinder, glance and nova ). All
OpenStack images and instances are being stored on Ceph in different pools
glance and nova pool respectively.

I assume that Ceph by default uses COW rbd , so for example if an instance
is launched using glance image (which is stored on Ceph) , Ceph should take
COW snapshot of glance image and map it as RBD disk for instance. And this
whole process should be very quick.

In our case , the instance launch is taking 90 seconds. Is this normal ? (
i know this really depends one's infra , but still )

Is there any way , i can utilize Ceph's power and can launch instances ever
faster.

- From Ceph point of view. does COW works cross pool i.e. image from glance
pool ---> (cow) --> instance disk on nova pool
- Will a single pool for glance and nova instead of separate pool . will
help here ?
- Is there any tunable parameter from Ceph or OpenStack side that should be
set ?

Regards
Vickey
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster + Ceph client upgrade path for production environment

[Vickey Singh]

Hello Guys

Need help with this , thanks

- vickey -

On Tue, Jan 12, 2016 at 12:10 PM, Vickey Singh <vickey.singh22...@gmail.com>
wrote:

> Hello Community , wishing you a great new year :)
>
> This is the recommended upgrade path
> http://docs.ceph.com/docs/master/install/upgrading-ceph/
>
> Ceph Deploy
> Ceph Monitors
> Ceph OSD Daemons
> Ceph Metadata Servers
> Ceph Object Gateways
>
> How about upgrading Ceph clients ( in my case openstack compute and
> controller nodes). Should i upgrade my ceph clients after upgrading entire
> ceph cluster ??
>
> Currently my Ceph cluster and Ceph client version is 0.80.8 and i am
> planning to upgrade it to  0.94.5
>
> How should i plan my Ceph client upgrade.
>
> Many Thanks in advance
>
> - vickey -
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph cluster + Ceph client upgrade path for production environment

[Vickey Singh]

Hello Community , wishing you a great new year :)

This is the recommended upgrade path
http://docs.ceph.com/docs/master/install/upgrading-ceph/

Ceph Deploy
Ceph Monitors
Ceph OSD Daemons
Ceph Metadata Servers
Ceph Object Gateways

How about upgrading Ceph clients ( in my case openstack compute and
controller nodes). Should i upgrade my ceph clients after upgrading entire
ceph cluster ??

Currently my Ceph cluster and Ceph client version is 0.80.8 and i am
planning to upgrade it to  0.94.5

How should i plan my Ceph client upgrade.

Many Thanks in advance

- vickey -
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Math behind : : OSD count vs OSD process vs OSD ports

[Vickey Singh]

A BIG Thanks Dmitry for your HELP.



On Wed, Nov 18, 2015 at 11:47 AM, Дмитрий Глушенок <gl...@jet.msk.su> wrote:

> Hi Vickey,
>
> 18 нояб. 2015 г., в 11:36, Vickey Singh <vickey.singh22...@gmail.com>
> написал(а):
>
> Can anyone please help me understand this.
>
> Thank You
>
>
> On Mon, Nov 16, 2015 at 5:55 PM, Vickey Singh <vickey.singh22...@gmail.com
> > wrote:
>
>> Hello Community
>>
>> Need your help in understanding this.
>>
>> I have the below node, which is hosting 60 physical disks, running 1 OSD
>> per disk so total 60 Ceph OSD daemons
>>
>> *[root@node01 ~]# service ceph status | grep -i osd | grep -i running |
>> wc -l*
>> *60*
>> *[root@node01 ~]#*
>>
>> However if i check OSD processes it shows that there are 120 OSD process
>> are running.
>>
>> *[root@node01 ~]# ps -ef | grep -i osd | grep -v grep | wc -l*
>> *120*
>> *[root@node01 ~]#*
>>
>> *Question 1 : why its 120 processes ? , i it should be 60 (because of 60
>> OSD on the system)*
>> My guess : is this because of multithreading ?
>>
>
> No, it is the way OSD processes were launched. Try following (quotes are
> necessary):
>
> $ bash -c "sleep 1; sleep 999" &
> $ ps -ef | grep sleep | grep -v grep
>
> You will see that bash process will wait for "sleep 999" process to
> finish. OSD processes are launched similarly.
>
>
>> Now if i check the number of ports used by OSD its comming out to be 240
>>
>> *[root@node01 ~]# netstat -plunt | grep -i osd | wc -l*
>> *240*
>> *[root@node01 ~]#*
>>
>> *Question 2 : Now why its 240 ports ? It should be 60 ( because of 60 OSD
>> on the system)*
>>
>
> It is because each OSD uses four ports:
> http://docs.ceph.com/docs/master/rados/configuration/network-config-ref/#osd-ip-tables
>
>
>> If i grep a specific OSD port , its shows 2 ports are occupied by OSD
>> process 260519
>>
>> *[root@node01 ~]# netstat -plunt | grep -i osd | grep -i 6819*
>> *tcp0  0 10.101.50.1:6819 <http://10.101.50.1:6819/>
>>  0.0.0.0:*   LISTEN  260519/ceph-osd*
>> *tcp0  0 10.102.50.1:6819 <http://10.102.50.1:6819/>
>>  0.0.0.0:*   LISTEN  260519/ceph-osd*
>> *[root@node01 ~]#*
>>
>> *Question 3 : Now based on the scenario 2 it should be 4 ( so 60 OSD x 4
>> ports = 240 ports in total)*
>>
>> I have two public and cluster network configured in ceph.conf , is all
>> these because of two different networks ?
>>
>
> If you will grep 260519 (PID) instead of 6819 (port) you should see four
> ports listened (two for private network and two for public).
>
>
>> *I would really appreciate if some knowledgeable person share his
>> understanding with me.*
>>
>> Thank you in advance.
>>
>>
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> --
> Dmitry Glushenok
> Jet Infosystems
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Math behind : : OSD count vs OSD process vs OSD ports

[Vickey Singh]

Can anyone please help me understand this.

Thank You


On Mon, Nov 16, 2015 at 5:55 PM, Vickey Singh <vickey.singh22...@gmail.com>
wrote:

> Hello Community
>
> Need your help in understanding this.
>
> I have the below node, which is hosting 60 physical disks, running 1 OSD
> per disk so total 60 Ceph OSD daemons
>
> *[root@node01 ~]# service ceph status | grep -i osd | grep -i running | wc
> -l*
> *60*
> *[root@node01 ~]#*
>
> However if i check OSD processes it shows that there are 120 OSD process
> are running.
>
> *[root@node01 ~]# ps -ef | grep -i osd | grep -v grep | wc -l*
> *120*
> *[root@node01 ~]#*
>
> *Question 1 : why its 120 processes ? , i it should be 60 (because of 60
> OSD on the system)*
> My guess : is this because of multithreading ?
>
> Now if i check the number of ports used by OSD its comming out to be 240
>
> *[root@node01 ~]# netstat -plunt | grep -i osd | wc -l*
> *240*
> *[root@node01 ~]#*
>
> *Question 2 : Now why its 240 ports ? It should be 60 ( because of 60 OSD
> on the system)*
>
> If i grep a specific OSD port , its shows 2 ports are occupied by OSD
> process 260519
>
> *[root@node01 ~]# netstat -plunt | grep -i osd | grep -i 6819*
> *tcp0  0 10.101.50.1:6819 <http://10.101.50.1:6819>
>  0.0.0.0:*   LISTEN  260519/ceph-osd*
> *tcp0  0 10.102.50.1:6819 <http://10.102.50.1:6819>
>  0.0.0.0:*   LISTEN  260519/ceph-osd*
> *[root@node01 ~]#*
>
> *Question 3 : Now based on the scenario 2 it should be 4 ( so 60 OSD x 4
> ports = 240 ports in total)*
>
> I have two public and cluster network configured in ceph.conf , is all
> these because of two different networks ?
>
> *I would really appreciate if some knowledgeable person share his
> understanding with me.*
>
> Thank you in advance.
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Math behind : : OSD count vs OSD process vs OSD ports

[Vickey Singh]

Hello Community

Need your help in understanding this.

I have the below node, which is hosting 60 physical disks, running 1 OSD
per disk so total 60 Ceph OSD daemons

*[root@node01 ~]# service ceph status | grep -i osd | grep -i running | wc
-l*
*60*
*[root@node01 ~]#*

However if i check OSD processes it shows that there are 120 OSD process
are running.

*[root@node01 ~]# ps -ef | grep -i osd | grep -v grep | wc -l*
*120*
*[root@node01 ~]#*

*Question 1 : why its 120 processes ? , i it should be 60 (because of 60
OSD on the system)*
My guess : is this because of multithreading ?

Now if i check the number of ports used by OSD its comming out to be 240

*[root@node01 ~]# netstat -plunt | grep -i osd | wc -l*
*240*
*[root@node01 ~]#*

*Question 2 : Now why its 240 ports ? It should be 60 ( because of 60 OSD
on the system)*

If i grep a specific OSD port , its shows 2 ports are occupied by OSD
process 260519

*[root@node01 ~]# netstat -plunt | grep -i osd | grep -i 6819*
*tcp0  0 10.101.50.1:6819 
 0.0.0.0:*   LISTEN  260519/ceph-osd*
*tcp0  0 10.102.50.1:6819 
 0.0.0.0:*   LISTEN  260519/ceph-osd*
*[root@node01 ~]#*

*Question 3 : Now based on the scenario 2 it should be 4 ( so 60 OSD x 4
ports = 240 ports in total)*

I have two public and cluster network configured in ceph.conf , is all
these because of two different networks ?

*I would really appreciate if some knowledgeable person share his
understanding with me.*

Thank you in advance.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Math behind : : OSD count vs OSD process vs OSD ports

[Vickey Singh]

Hello Community

Need your help in understanding this.

I have the below node, which is hosting 60 physical disks, running 1 OSD
per disk so total 60 Ceph OSD daemons

*[root@node01 ~]# service ceph status | grep -i osd | grep -i running | wc
-l*
*60*
*[root@node01 ~]#*

However if i check OSD processes it shows that there are 120 OSD process
are running.

*[root@node01 ~]# ps -ef | grep -i osd | grep -v grep | wc -l*
*120*
*[root@node01 ~]#*

*Question 1 : why its 120 processes ? , i it should be 60 (because of 60
OSD on the system)*
My guess : is this because of multithreading ?

Now if i check the number of ports used by OSD its comming out to be 240

*[root@node01 ~]# netstat -plunt | grep -i osd | wc -l*
*240*
*[root@node01 ~]#*

*Question 2 : Now why its 240 ports ? It should be 60 ( because of 60 OSD
on the system)*

If i grep a specific OSD port , its shows 2 ports are occupied by OSD
process 260519

*[root@node01 ~]# netstat -plunt | grep -i osd | grep -i 6819*
*tcp0  0 10.101.50.1:6819 
 0.0.0.0:*   LISTEN  260519/ceph-osd*
*tcp0  0 10.102.50.1:6819 
 0.0.0.0:*   LISTEN  260519/ceph-osd*
*[root@node01 ~]#*

*Question 3 : Now based on the scenario 2 it should be 4 ( so 60 OSD x 4
ports = 240 ports in total)*

I have two public and cluster network configured in ceph.conf , is all
these because of two different networks ?

*I would really appreciate if some knowledgeable person share his
understanding with me.*

Thank you in advance.*ble person share his understanding with me.*
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Using straw2 crush also with Hammer

[Vickey Singh]

On Mon, Nov 9, 2015 at 8:16 PM, Wido den Hollander <w...@42on.com> wrote:

> On 11/09/2015 05:27 PM, Vickey Singh wrote:
> > Hello Ceph Geeks
> >
> > Need your comments with my understanding on straw2.
> >
> >- Is Straw2 better than straw ?
>
> It is not persé better then straw(1).
>
> straw2 distributes data better when not all OSDs are equally
> sized/weighted.
>
> >- Is it straw2 recommended  for production usage ?
> >
>
> I'm using it in multiple clusters and it works fine.
>

Yes i followed your update on twitter  :)


>
> > I have a production Ceph Firefly cluster , that i am going to upgrade to
> > Ceph hammer pretty soon. Should i use straw2 for all my ceph pools ?
> >
>
> I would upgrade to Hammer first and make sure that ALL clients are updated.
>
> In case you are using KVM/Qemu, you will have to stop those processes
> first and start them again before they are using the new code.
>

Thanks a lot for this pointer, i didn't knew this. So restarting KVM / QEMU
process effect running VMs ? ( some downtime )


>
> > Also how can i change from straw to straw2. ( should i decompile crush
> map
> > and replace all straw with straw2) ?
> >
>
> Yes, that's it. Extract, decompile, replace, compile, inject.
>
> $ ceph osd getcrushmap -o crushmap
> $ crushtool -d crushmap -o crushmap.txt
> $ 
> $ crushtool -c crushmap.txt -o crushmap.new
> $ ceph osd setcrushmap -i crushmap.new
>
> In case stuff goes wrong, inject the original crushmap.
>
> Wido
>

Thank You so much for your time in replying to my queries :)


>
> > Please suggest
> >
> > Thank You in advance.
> >
> > - Vickey -
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
> --
> Wido den Hollander
> 42on B.V.
> Ceph trainer and consultant
>
> Phone: +31 (0)20 700 9902
> Skype: contact42on
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Using straw2 crush also with Hammer

[Vickey Singh]

Hello Ceph Geeks

Need your comments with my understanding on straw2.

   - Is Straw2 better than straw ?
   - Is it straw2 recommended  for production usage ?

I have a production Ceph Firefly cluster , that i am going to upgrade to
Ceph hammer pretty soon. Should i use straw2 for all my ceph pools ?

Also how can i change from straw to straw2. ( should i decompile crush map
and replace all straw with straw2) ?

Please suggest

Thank You in advance.

- Vickey -
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Software Raid 1 for system disks on storage nodes (not for OSD disks)

[Vickey Singh]

On Fri, Sep 18, 2015 at 6:33 PM, Robert LeBlanc 
wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Depends on how easy it is to rebuild an OS from scratch. If you have
> something like Puppet or Chef that configure a node completely for
> you, it may not be too much of a pain to forgo the RAID. We run our
> OSD nodes from a single SATADOM and use Puppet for configuration. We
> also don't use swap (not very effective on SATADOM), but have enough
> RAM that we feel comfortable enough with that decision.
>
> If you use ceph-disk or ceph-deploy to configure the OSDs, then they
> should automatically come back up when you lay down the new OS and set
> up the necessary ceph config items (ceph.conf and the OSD bootstrap
> keys).
>

Hello sir

This sounds really interesting , could you please elaborate how after
reinstalling OS and installing Ceph packages, how does Ceph detects OSD's
that were hosted earlier on this node.

I am using ceph-deploy to provision ceph , now what all changes i need to
do after reinstalling OS of a OSD node. So that it should detect my OSD
daemons. Please help me to know this step by step.

Thanks in advance.

Vickey



> - 
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
>
>
> On Fri, Sep 18, 2015 at 9:06 AM, Martin Palma  wrote:
> > Hi,
> >
> > Is it a good idea to use a software raid for the system disk (Operating
> > System) on a Ceph storage node? I mean only for the OS not for the OSD
> > disks.
> >
> > And what about a swap partition? Is that needed?
> >
> > Best,
> > Martin
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
> -BEGIN PGP SIGNATURE-
> Version: Mailvelope v1.1.0
> Comment: https://www.mailvelope.com
>
> wsFcBAEBCAAQBQJV/C7UCRDmVDuy+mK58QAAoTMQAMZBv4/lphmntC23b9/l
> JWUPjZfbXUtNgnfMvWcVyTSXsTtM5mY/4/iSZ4ZfCQ4YyqWWMpSlocHONHFz
> nFTtGupqV3vPCo4X8bl58/iv4J0H2iWUr2klk7jtTj+e+JjyWDo25l8V2ofP
> edt5g7qcMAwiWYrrpjxQBK4AFNiPJKSMxrzK1Mgic15nwX0OJu0DDNS5twzZ
> s8Y+UfS80+hZvyBTUGhsO8pkYoJQvYRGgyqYtCdxA+m1T8lWVe8SC0eLWOXy
> xoyGR7dqcvEXQadrqfmU618eNpNEECPoHeIkeCqpTohrUVsyRcfSGAtfM0YY
> Ixf2SCaDMAaRwvXGJUf5OP/3HHWps0m4YyLBOddPZ5XZb1utZiclh26KuOyw
> QdGkP7uoYEMO0v40dcsIbOVhtgTdX+HrpEGuqEtNEGe194sS1nluw+49aLxe
> eozHSRGq3GmRm/q3bR5f2p+WXwKqmdDRFhqII8H11bb5F7etU2PBo1JA2bTW
> hUFqu6+ST8eI34OeC7LbC9Txfw/iUhL62kiCm+gj8Rg+m+TZ7a1HEaVc8uyq
> Jw1+5hIgyTWFvKdIiW65k++8w9my6kUIsY8RT8p08DTSPzxuwGtHr7UJJ629
> K/tlpGdQTRf7PXgmea6sSodnmaF5HRIUdU0nhQpRRxjX/V+PENI8Qq45KyfX
> BovV
> =Gzvl
> -END PGP SIGNATURE-
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Thumb rule for selecting memory for Ceph OSD node

[Vickey Singh]

Hello Guys

Doing hardware planning / selection for a new production Ceph cluster. Just
wondering how should i select memory.

*I have found two different rules of selecting memory for Ceph OSD.( on
Internet / googling / presentations )*

*#11GB / Ceph OSD  or 2GB / Ceph OSD ( for more performance )*

For Example : 12 OSD system it will be 12GB or 24GB. In this case don't the
disk size matter ??

4TB Drive : 12 X 4 = 48TB Raw storage  (  is 24GB sufficient ?  )
6TB Drive : 12 X 6 = 72 TB Raw storage  ( is 24  GB Sufficient  ? )


*#2   1GB / 1TB of RAW capacity of system*

4TB Drive : 12 X 4 = 48TB Raw storage  (  is 48GB is more than enough ?  )
6TB Drive : 12 X 6 = 72 TB Raw storage  ( is 72GB is more than enough  ? )

In case of Dense node

72 x 6TB = 432TB Raw storage ( 432G memory seems to be HUGE investment ? )


So which rule should we considered that can stand true for a 12 OSD node
and even for 72 OSD node.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

[Vickey Singh]

Hey Lincoln



On Tue, Sep 8, 2015 at 7:26 PM, Lincoln Bryant <linco...@uchicago.edu>
wrote:

> For whatever it’s worth, my problem has returned and is very similar to
> yours. Still trying to figure out what’s going on over here.
>
> Performance is nice for a few seconds, then goes to 0. This is a similar
> setup to yours (12 OSDs per box, Scientific Linux 6, Ceph 0.94.3, etc)
>
>   384  16 29520 29504   307.287  1188 0.0492006  0.208259
>   385  16 29813 29797   309.532  1172 0.0469708  0.206731
>   386  16 30105 30089   311.756  1168 0.0375764  0.205189
>   387  16 30401 30385   314.009  1184  0.036142  0.203791
>   388  16 30695 30679   316.231  1176 0.0372316  0.202355
>   389  16 30987 30971318.42  1168 0.0660476  0.200962
>   390  16 31282 31266   320.628  1180 0.0358611  0.199548
>   391  16 31568 31552   322.734  1144 0.0405166  0.198132
>   392  16 31857 31841   324.859  1156 0.0360826  0.196679
>   393  16 32090 32074   326.404   932 0.0416869   0.19549
>   394  16 32205 32189   326.743   460 0.0251877  0.194896
>   395  16 32302 32286   326.897   388 0.0280574  0.194395
>   396  16 32348 32332   326.537   184 0.0256821  0.194157
>   397  16 32385 32369   326.087   148 0.0254342  0.193965
>   398  16 32424 32408   325.659   156 0.0263006  0.193763
>   399  16 32445 32429   325.05484 0.0233839  0.193655
> 2015-09-08 11:22:31.940164 min lat: 0.0165045 max lat: 67.6184 avg lat:
> 0.193655
>   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>   400  16 32445 32429   324.241 0 -  0.193655
>   401  16 32445 32429   323.433 0 -  0.193655
>   402  16 32445 32429   322.628 0 -  0.193655
>   403  16 32445 32429   321.828 0 -  0.193655
>   404  16 32445 32429   321.031 0 -  0.193655
>   405  16 32445 32429   320.238 0 -  0.193655
>   406  16 32445 32429319.45 0 -  0.193655
>   407  16 32445 32429   318.665 0 -  0.193655
>
> needless to say, very strange.
>

Its indeed very strange

( The solution that you gave me in the below email ) Have you tried
restarting all OSD's ?

By the way my problem got fixed ( but i am afraid , it can come back any
time ) by doing

# service ceph restart osd  on all OSD nodes ( this didn't helped )
# set noout,nodown,nobackfill,norecover and then reboot all OSD nodes ( It
worked )  After they all the rados bench write started to work.

[ i know its hilarious , feels like  i am watching *The IT Crowd* ' Hello
IT , Have you tried turning it OFF and ON again ' ]

It would be really helpful if someone provides a real solution.




>
> —Lincoln
>
>
> > On Sep 7, 2015, at 3:35 PM, Vickey Singh <vickey.singh22...@gmail.com>
> wrote:
> >
> > Adding ceph-users.
> >
> > On Mon, Sep 7, 2015 at 11:31 PM, Vickey Singh <
> vickey.singh22...@gmail.com> wrote:
> >
> >
> > On Mon, Sep 7, 2015 at 10:04 PM, Udo Lembke <ulem...@polarzone.de>
> wrote:
> > Hi Vickey,
> > Thanks for your time in replying to my problem.
> >
> > I had the same rados bench output after changing the motherboard of the
> monitor node with the lowest IP...
> > Due to the new mainboard, I assume the hw-clock was wrong during
> startup. Ceph health show no errors, but all VMs aren't able to do IO (very
> high load on the VMs - but no traffic).
> > I stopped the mon, but this don't changed anything. I had to restart all
> other mons to get IO again. After that I started the first mon also (with
> the right time now) and all worked fine again...
> >
> > Thanks i will try to restart all OSD / MONS and report back , if it
> solves my problem
> >
> > Another posibility:
> > Do you use journal on SSDs? Perhaps the SSDs can't write to garbage
> collection?
> >
> > No i don't have journals on SSD , they are on the same OSD disk.
> >
> >
> >
> > Udo
> >
> >
> > On 07.09.2015 16:36, Vickey Singh wrote:
> >> Dear Experts
> >>
> >> Can someone please help me , why my cluster is not able write data.
> >>
> >> See the below output  cur MB/S  is 0  and Avg MB/s is decreasing.
> >>
> >>
> >> Ceph Hammer  0.94.2
> >> CentOS 6 (3.10.69-1)
> >>
> >> The Ceph status says OPS are blocked , i have tried

Re: [ceph-users] rebalancing taking very long time

[Vickey Singh]

Agreed with Alphe , Ceph Hammer (0.94.2) sucks when it comes to recovery
and rebalancing.

Here is my Ceph Hammer cluster , which is like this for more than 30 hours.

You might be thinking about that one OSD which is down and not in.  Its
intentional, i want to remove that OSD.
I want the cluster to become healthy again before i remove that OSD.

Can someone help us with this problem

 cluster 86edf8b8-b353-49f1-ab0a-a4827a9ea5e8
 health HEALTH_WARN
14 pgs stuck unclean
5 requests are blocked > 32 sec
recovery 420/28358085 objects degraded (0.001%)
recovery 199941/28358085 objects misplaced (0.705%)
too few PGs per OSD (28 < min 30)
 monmap e3: 3 mons at {stor0201=10.100.1.201:6789/0,stor0202
=10.100.1.202:6789/0,stor0203=10.100.1.203:6789/0}
election epoch 1076, quorum 0,1,2 stor0201,stor0202,
stor0203
 osdmap e778879: 96 osds: 95 up, 95 in; 14 remapped pgs
  pgmap v2475334: 896 pgs, 4 pools, 51364 GB data, 9231 kobjects
150 TB used, 193 TB / 344 TB avail
420/28358085 objects degraded (0.001%)
199941/28358085 objects misplaced (0.705%)
 879 active+clean
  14 active+remapped
   3 active+clean+scrubbing+deep



On Tue, Sep 8, 2015 at 5:59 PM, Alphe Salas  wrote:

> I can say exactly the same I am using ceph sin 0.38 and I never get osd so
> laggy than with 0.94. rebalancing /rebuild algorithm is crap in 0.94
> serriously I have 2 osd serving 2 discs of 2TB and 4 GB of RAM osd takes
> 1.6GB each !!! serriously ! that makes avanche snow.
>
> Let me be straight and explain what changed.
>
> in 0.38 you ALWAYS could stop the ceph cluster and then start it up it
> would evaluate if everyone is back if there is enough replicas then start
> rebuilding /rebalancing what needed of course like 10 minutes was necesary
> to bring up ceph cluster but then the rebuilding /rebalancing process was
> smooth.
> With 0.94 first you have 2 osd too full at 95 % and 4 osd at 63% over 20
> osd. then you get a disc crash. so ceph starts automatically to rebuild and
> rebalance stuff. and there osd start to lag then to crash
> you stop ceph cluster you change the drive restart the ceph cluster stops
> all rebuild process setting no-backfill, norecovey noscrub nodeep-scrub you
> rm the old osd create a new one wait for all osd
> to be in and up and then starts rebuilding lag/rebalancing since it is
> automated not much a choice there.
>
> And again all osd are stuck in enless lag/down/recovery intent cycle...
>
> It is a pain serriously. 5 days after changing the faulty disc it is still
> locked in the lag/down/recovery cycle.
>
> Sur it can be argued that my machines are really ressource limited and
> that I should buy 3 thousand dollar worth server at least. But intil 0.72
> that rebalancing /rebuilding process was working smoothly on the same
> hardware.
>
> It seems to me that the rebalancing/rebuilding algorithm is more strict
> now than it was in the past. in the past only what really really needed to
> be rebuild or rebalance was rebalanced or rebuild.
>
> I can still delete all and go back to 0.72... like I should buy a cray
> T-90 to not have anymore problems and have ceph run smoothly. But this will
> not help making ceph a better product.
>
> for me ceph 0.94 is like windows vista...
>
> Alphe Salas
> I.T ingeneer
>
>
> On 09/08/2015 10:20 AM, Gregory Farnum wrote:
>
>> On Wed, Sep 2, 2015 at 9:34 PM, Bob Ababurko  wrote:
>>
>>> When I lose a disk OR replace a OSD in my POC ceph cluster, it takes a
>>> very
>>> long time to rebalance.  I should note that my cluster is slightly
>>> unique in
>>> that I am using cephfs(shouldn't matter?) and it currently contains about
>>> 310 million objects.
>>>
>>> The last time I replaced a disk/OSD was 2.5 days ago and it is still
>>> rebalancing.  This is on a cluster with no client load.
>>>
>>> The configurations is 5 hosts with 6 x 1TB 7200rpm SATA OSD's & 1 850 Pro
>>> SSD which contains the journals for said OSD's.  Thats means 30 OSD's in
>>> total.  System disk is on its own disk.  I'm also using a backend network
>>> with single Gb NIC.  THe rebalancing rate(objects/s) seems to be very
>>> slow
>>> when it is close to finishingsay <1% objects misplaced.
>>>
>>> It doesn't seem right that it would take 2+ days to rebalance a 1TB disk
>>> with no load on the cluster.  Are my expectations off?
>>>
>>
>> Possibly...Ceph basically needs to treat each object as a single IO.
>> If you're recovering from a failed disk then you've got to replicate
>> roughly 310 million * 3 / 30 = 31 million objects. If it's perfectly
>> balanced across 30 disks that get 80 IOPS that's 12916 seconds (~3.5
>> hours) worth of work just to read each file — and in reality it's
>> likely to take more than one IO to read the file, and then you have to
>> spend a bunch to write it as well.
>>
>>

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

[Vickey Singh]

Hello Jan

On Wed, Sep 9, 2015 at 11:59 AM, Jan Schermer <j...@schermer.cz> wrote:

> Just to recapitulate - the nodes are doing "nothing" when it drops to
> zero? Not flushing something to drives (iostat)? Not cleaning pagecache
> (kswapd and similiar)? Not out of any type of memory (slab,
> min_free_kbytes)? Not network link errors, no bad checksums (those are hard
> to spot, though)?
>
> Unless you find something I suggest you try disabling offloads on the NICs
> and see if the problem goes away.
>

Could you please elaborate this point , how do you disable / offload on the
NIC ? what does it mean ? how to do it ? how its gonna help.

Sorry i don't know about this.

- Vickey -



>
> Jan
>
> > On 08 Sep 2015, at 18:26, Lincoln Bryant <linco...@uchicago.edu> wrote:
> >
> > For whatever it’s worth, my problem has returned and is very similar to
> yours. Still trying to figure out what’s going on over here.
> >
> > Performance is nice for a few seconds, then goes to 0. This is a similar
> setup to yours (12 OSDs per box, Scientific Linux 6, Ceph 0.94.3, etc)
> >
> >  384  16 29520 29504   307.287  1188 0.0492006  0.208259
> >  385  16 29813 29797   309.532  1172 0.0469708  0.206731
> >  386  16 30105 30089   311.756  1168 0.0375764  0.205189
> >  387  16 30401 30385   314.009  1184  0.036142  0.203791
> >  388  16 30695 30679   316.231  1176 0.0372316  0.202355
> >  389  16 30987 30971318.42  1168 0.0660476  0.200962
> >  390  16 31282 31266   320.628  1180 0.0358611  0.199548
> >  391  16 31568 31552   322.734  1144 0.0405166  0.198132
> >  392  16 31857 31841   324.859  1156 0.0360826  0.196679
> >  393  16 32090 32074   326.404   932 0.0416869   0.19549
> >  394  16 32205 32189   326.743   460 0.0251877  0.194896
> >  395  16 32302 32286   326.897   388 0.0280574  0.194395
> >  396  16 32348 32332   326.537   184 0.0256821  0.194157
> >  397  16 32385 32369   326.087   148 0.0254342  0.193965
> >  398  16 32424 32408   325.659   156 0.0263006  0.193763
> >  399  16 32445 32429   325.05484 0.0233839  0.193655
> > 2015-09-08 11:22:31.940164 min lat: 0.0165045 max lat: 67.6184 avg lat:
> 0.193655
> >  sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
> >  400  16 32445 32429   324.241 0 -  0.193655
> >  401  16 32445 32429   323.433 0 -  0.193655
> >  402  16 32445 32429   322.628 0 -  0.193655
> >  403  16 32445 32429   321.828 0 -  0.193655
> >  404  16 32445 32429   321.031 0 -  0.193655
> >  405  16 32445 32429   320.238 0 -  0.193655
> >  406  16 32445 32429319.45 0 -  0.193655
> >  407  16 32445 32429   318.665 0 -  0.193655
> >
> > needless to say, very strange.
> >
> > —Lincoln
> >
> >
> >> On Sep 7, 2015, at 3:35 PM, Vickey Singh <vickey.singh22...@gmail.com>
> wrote:
> >>
> >> Adding ceph-users.
> >>
> >> On Mon, Sep 7, 2015 at 11:31 PM, Vickey Singh <
> vickey.singh22...@gmail.com> wrote:
> >>
> >>
> >> On Mon, Sep 7, 2015 at 10:04 PM, Udo Lembke <ulem...@polarzone.de>
> wrote:
> >> Hi Vickey,
> >> Thanks for your time in replying to my problem.
> >>
> >> I had the same rados bench output after changing the motherboard of the
> monitor node with the lowest IP...
> >> Due to the new mainboard, I assume the hw-clock was wrong during
> startup. Ceph health show no errors, but all VMs aren't able to do IO (very
> high load on the VMs - but no traffic).
> >> I stopped the mon, but this don't changed anything. I had to restart
> all other mons to get IO again. After that I started the first mon also
> (with the right time now) and all worked fine again...
> >>
> >> Thanks i will try to restart all OSD / MONS and report back , if it
> solves my problem
> >>
> >> Another posibility:
> >> Do you use journal on SSDs? Perhaps the SSDs can't write to garbage
> collection?
> >>
> >> No i don't have journals on SSD , they are on the same OSD disk.
> >>
> >>
> >>
> >> Udo
> >>
> >>
> >> On 07.09.2015 16:36, Vickey Sin

[ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

[Vickey Singh]

Dear Experts

Can someone please help me , why my cluster is not able write data.

See the below output  cur MB/S  is 0  and Avg MB/s is decreasing.


Ceph Hammer  0.94.2
CentOS 6 (3.10.69-1)

The Ceph status says OPS are blocked , i have tried checking , what all i
know

- System resources ( CPU , net, disk , memory )-- All normal
- 10G network for public and cluster network  -- no saturation
- Add disks are physically healthy
- No messages in /var/log/messages OR dmesg
- Tried restarting OSD which are blocking operation , but no luck
- Tried writing through RBD  and Rados bench , both are giving same problemm

Please help me to fix this problem.

#  rados bench -p rbd 60 write
 Maintaining 16 concurrent writes of 4194304 bytes for up to 60 seconds or
0 objects
 Object prefix: benchmark_data_stor1_1791844
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
 0   0 0 0 0 0 - 0
 1  16   125   109   435.873   436  0.022076 0.0697864
 2  16   139   123   245.94856  0.246578 0.0674407
 3  16   139   123   163.969 0 - 0.0674407
 4  16   139   123   122.978 0 - 0.0674407
 5  16   139   12398.383 0 - 0.0674407
 6  16   139   123   81.9865 0 - 0.0674407
 7  16   139   123   70.2747 0 - 0.0674407
 8  16   139   123   61.4903 0 - 0.0674407
 9  16   139   123   54.6582 0 - 0.0674407
10  16   139   123   49.1924 0 - 0.0674407
11  16   139   123   44.7201 0 - 0.0674407
12  16   139   123   40.9934 0 - 0.0674407
13  16   139   123   37.8401 0 - 0.0674407
14  16   139   123   35.1373 0 - 0.0674407
15  16   139   123   32.7949 0 - 0.0674407
16  16   139   123   30.7451 0 - 0.0674407
17  16   139   123   28.9364 0 - 0.0674407
18  16   139   123   27.3289 0 - 0.0674407
19  16   139   123   25.8905 0 - 0.0674407
2015-09-07 15:54:52.694071min lat: 0.022076 max lat: 0.46117 avg lat:
0.0674407
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
20  16   139   12324.596 0 - 0.0674407
21  16   139   123   23.4247 0 - 0.0674407
22  16   139   123 22.36 0 - 0.0674407
23  16   139   123   21.3878 0 - 0.0674407
24  16   139   123   20.4966 0 - 0.0674407
25  16   139   123   19.6768 0 - 0.0674407
26  16   139   123 18.92 0 - 0.0674407
27  16   139   123   18.2192 0 - 0.0674407
28  16   139   123   17.5686 0 - 0.0674407
29  16   139   123   16.9628 0 - 0.0674407
30  16   139   123   16.3973 0 - 0.0674407
31  16   139   123   15.8684 0 - 0.0674407
32  16   139   123   15.3725 0 - 0.0674407
33  16   139   123   14.9067 0 - 0.0674407
34  16   139   123   14.4683 0 - 0.0674407
35  16   139   123   14.0549 0 - 0.0674407
36  16   139   123   13.6645 0 - 0.0674407
37  16   139   123   13.2952 0 - 0.0674407
38  16   139   123   12.9453 0 - 0.0674407
39  16   139   123   12.6134 0 - 0.0674407
2015-09-07 15:55:12.697124min lat: 0.022076 max lat: 0.46117 avg lat:
0.0674407
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
40  16   139   123   12.2981 0 - 0.0674407
41  16   139   123   11.9981 0 - 0.0674407




cluster 86edf8b8-b353-49f1-ab0a-a4827a9ea5e8
 health HEALTH_WARN
1 requests are blocked > 32 sec
 monmap e3: 3 mons at {stor0111=
10.100.1.111:6789/0,stor0113=10.100.1.113:6789/0,stor011
5=10.100.1.115:6789/0}
election epoch 32, quorum 0,1,2 stor0111,stor0113,stor0115
 osdmap e19536: 50 osds: 50 up, 50 in
  pgmap v928610: 2752 pgs, 9 pools, 30476 GB data, 4183 kobjects
91513 GB used, 47642 GB / 135 TB avail
2752 active+clean


Tried using RBD


# dd if=/dev/zero of=file1 bs=4K count=1 oflag=direct
1+0 records in
1+0 records out
4096 bytes (41 

[ceph-users] Extra RAM use as Read Cache

[Vickey Singh]

Hello Experts ,

I want to increase my Ceph cluster's read performance.

I have several OSD nodes having 196G RAM. On my OSD nodes Ceph just uses
15-20 GB of RAM.

So, can i instruct Ceph to make use of the remaining 150GB+ RAM as read
cache. So that it should cache data in RAM and server to clients very fast.

I hope if this can be done, i can get a good read performance boost.


By the way we have a LUSTRE cluster , that uses extra RAM as read cache and
we can get upto 2.5GBps read performance.  I am looking someone to do with
Ceph.

- Vickey -
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

[Vickey Singh]

On Mon, Sep 7, 2015 at 7:39 PM, Lincoln Bryant <linco...@uchicago.edu>
wrote:

> Hi Vickey,
>
>
Thanks a lot for replying to my problem.


> I had this exact same problem last week, resolved by rebooting all of my
> OSD nodes. I have yet to figure out why it happened, though. I _suspect_ in
> my case it's due to a failing controller on a particular box I've had
> trouble with in the past.
>

Mine is a 5 node cluster 12 OSD each node and and in past there has never
been any hardware problems.


> I tried setting 'noout', stopping my OSDs one host at a time, then
> rerunning RADOS bench between to see if I could nail down the problematic
> machine. Depending on your # of hosts, this might work for you. Admittedly,
> I got impatient with this approach though and just ended up restarting
> everything (which worked!) :)
>

So do you mean you intentionally brought one node's OSD down ? so some OSD
were down but none of then were out (no out ) . Then you waited for some
time to make cluster healthy , and then you rerun rados bench ??


>
> If you have a bunch of blocked ops, you could maybe try a 'pg query' on
> the PGs involved and see if there's a common OSD with all of your blocked
> ops. In my experience, it's not necessarily the one reporting.
>

Yeah, i have 55 OSDs and every time any random OSD shows OPS blocked. So i
can't blame any specific OSD. After few minutes that blocked OSD becomes
clean and after sometime some other osd blocks ops.


Thanks, i will try to restart all osd / monitor daemons , and see if this
fixes. Is there any thing i need to keep in mind to restart osd  ( expect
nodown , noout )  ??


>
> Anecdotally, I've had trouble with Intel 10Gb NICs and custom kernels as
> well. I've seen a NIC appear to be happy (no message in dmesg, machine
> appears to be communicating normally, etc) but when I went to iperf it, I
> was getting super pitiful performance (like KB/s). I don't know what kind
> of NICs you're using, but you may want to iperf everything just in case.
>

Yeah i did that , iperf shows no problem.

Is there anything else i should do ??


>
> --Lincoln
>
>
> On 9/7/2015 9:36 AM, Vickey Singh wrote:
>
> Dear Experts
>
> Can someone please help me , why my cluster is not able write data.
>
> See the below output  cur MB/S  is 0  and Avg MB/s is decreasing.
>
>
> Ceph Hammer  0.94.2
> CentOS 6 (3.10.69-1)
>
> The Ceph status says OPS are blocked , i have tried checking , what all i
> know
>
> - System resources ( CPU , net, disk , memory )-- All normal
> - 10G network for public and cluster network  -- no saturation
> - Add disks are physically healthy
> - No messages in /var/log/messages OR dmesg
> - Tried restarting OSD which are blocking operation , but no luck
> - Tried writing through RBD  and Rados bench , both are giving same problemm
>
> Please help me to fix this problem.
>
> #  rados bench -p rbd 60 write
>  Maintaining 16 concurrent writes of 4194304 bytes for up to 60 seconds or
> 0 objects
>  Object prefix: benchmark_data_stor1_1791844
>sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>  0   0 0 0 0 0 - 0
>  1  16   125   109   435.873   436  0.022076 0.0697864
>  2  16   139   123   245.94856  0.246578 0.0674407
>  3  16   139   123   163.969 0 - 0.0674407
>  4  16   139   123   122.978 0 - 0.0674407
>  5  16   139   12398.383 0 - 0.0674407
>  6  16   139   123   81.9865 0 - 0.0674407
>  7  16   139   123   70.2747 0 - 0.0674407
>  8  16   139   123   61.4903 0 - 0.0674407
>  9  16   139   123   54.6582 0 - 0.0674407
> 10  16   139   123   49.1924 0 - 0.0674407
> 11  16   139   123   44.7201 0 - 0.0674407
> 12  16   139   123   40.9934 0 - 0.0674407
> 13  16   139   123   37.8401 0 - 0.0674407
> 14  16   139   123   35.1373 0 - 0.0674407
> 15  16   139   123   32.7949 0 - 0.0674407
> 16  16   139   123   30.7451 0 - 0.0674407
> 17  16   139   123   28.9364 0 - 0.0674407
> 18  16   139   123   27.3289 0 - 0.0674407
> 19  16   139   123   25.8905 0 - 0.0674407
> 2015-09-07 15:54:52.694071min lat: 0.022076 max lat: 0.46117 avg lat:
> 0.0674407
>sec Cur op

Re: [ceph-users] Ceph cluster NO read / write performance :: Ops are blocked

[Vickey Singh]

Adding ceph-users.

On Mon, Sep 7, 2015 at 11:31 PM, Vickey Singh <vickey.singh22...@gmail.com>
wrote:

>
>
> On Mon, Sep 7, 2015 at 10:04 PM, Udo Lembke <ulem...@polarzone.de> wrote:
>
>> Hi Vickey,
>>
> Thanks for your time in replying to my problem.
>
>
>> I had the same rados bench output after changing the motherboard of the
>> monitor node with the lowest IP...
>> Due to the new mainboard, I assume the hw-clock was wrong during startup.
>> Ceph health show no errors, but all VMs aren't able to do IO (very high
>> load on the VMs - but no traffic).
>> I stopped the mon, but this don't changed anything. I had to restart all
>> other mons to get IO again. After that I started the first mon also (with
>> the right time now) and all worked fine again...
>>
>
> Thanks i will try to restart all OSD / MONS and report back , if it solves
> my problem
>
>>
>> Another posibility:
>> Do you use journal on SSDs? Perhaps the SSDs can't write to garbage
>> collection?
>>
>
> No i don't have journals on SSD , they are on the same OSD disk.
>
>>
>>
>>
>> Udo
>>
>>
>> On 07.09.2015 16:36, Vickey Singh wrote:
>>
>> Dear Experts
>>
>> Can someone please help me , why my cluster is not able write data.
>>
>> See the below output  cur MB/S  is 0  and Avg MB/s is decreasing.
>>
>>
>> Ceph Hammer  0.94.2
>> CentOS 6 (3.10.69-1)
>>
>> The Ceph status says OPS are blocked , i have tried checking , what all i
>> know
>>
>> - System resources ( CPU , net, disk , memory )-- All normal
>> - 10G network for public and cluster network  -- no saturation
>> - Add disks are physically healthy
>> - No messages in /var/log/messages OR dmesg
>> - Tried restarting OSD which are blocking operation , but no luck
>> - Tried writing through RBD  and Rados bench , both are giving same
>> problemm
>>
>> Please help me to fix this problem.
>>
>> #  rados bench -p rbd 60 write
>>  Maintaining 16 concurrent writes of 4194304 bytes for up to 60 seconds
>> or 0 objects
>>  Object prefix: benchmark_data_stor1_1791844
>>sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>>  0   0 0 0 0 0 - 0
>>  1  16   125   109   435.873   436  0.022076 0.0697864
>>  2  16   139   123   245.94856  0.246578 0.0674407
>>  3  16   139   123   163.969 0 - 0.0674407
>>  4  16   139   123   122.978 0 - 0.0674407
>>  5  16   139   12398.383 0 - 0.0674407
>>  6  16   139   123   81.9865 0 - 0.0674407
>>  7  16   139   123   70.2747 0 - 0.0674407
>>  8  16   139   123   61.4903 0 - 0.0674407
>>  9  16   139   123   54.6582 0 - 0.0674407
>> 10  16   139   123   49.1924 0 - 0.0674407
>> 11  16   139   123   44.7201 0 - 0.0674407
>> 12  16   139   123   40.9934 0 - 0.0674407
>> 13  16   139   123   37.8401 0 - 0.0674407
>> 14  16   139   123   35.1373 0 - 0.0674407
>> 15  16   139   123   32.7949 0 - 0.0674407
>> 16  16   139   123   30.7451 0 - 0.0674407
>> 17  16   139   123   28.9364 0 - 0.0674407
>> 18  16   139   123   27.3289 0 - 0.0674407
>> 19  16   139   123   25.8905 0 - 0.0674407
>> 2015-09-07 15:54:52.694071min lat: 0.022076 max lat: 0.46117 avg lat:
>> 0.0674407
>>sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
>> 20  16   139   12324.596 0 - 0.0674407
>> 21  16   139   123   23.4247 0 - 0.0674407
>> 22  16   139   123 22.36 0 - 0.0674407
>> 23  16   139   123   21.3878 0 - 0.0674407
>> 24  16   139   123   20.4966 0 - 0.0674407
>> 25  16   139   123   19.6768 0 - 0.0674407
>> 26  16   139   123 18.92 0 - 0.0674407
>> 27  16   139   123   18.2192 0 - 0.0674407
>

Re: [ceph-users] Ceph read / write : Terrible performance

[Vickey Singh]

Hey Mark / Community

These are the sequences of changes that seems to have fixed the ceph problem

1#  Upgrading Disk controller firmware from 6.34 to 6.64  ( latest )
2# Rebooting all nodes in order to make new firmware into effect

Read and write operations are now normal as well as system load and
CPU utilization

- Vickey -


On Wed, Sep 2, 2015 at 11:28 PM, Vickey Singh <vickey.singh22...@gmail.com>
wrote:

> Thank You Mark , please see my response below.
>
> On Wed, Sep 2, 2015 at 5:23 PM, Mark Nelson <mnel...@redhat.com> wrote:
>
>> On 09/02/2015 08:51 AM, Vickey Singh wrote:
>>
>>> Hello Ceph Experts
>>>
>>> I have a strange problem , when i am reading or writing to Ceph pool ,
>>> its not writing properly. Please notice Cur MB/s which is going up and
>>> down
>>>
>>> --- Ceph Hammer 0.94.2
>>> -- CentOS 6, 2.6
>>> -- Ceph cluster is healthy
>>>
>>
>> You might find that CentOS7 gives you better performance.  In some cases
>> we were seeing nearly 2X.
>
>
> Wooo 2X , i would definitely plan for upgrade. Thanks
>
>
>>
>>
>>
>>>
>>> One interesting thing is when every i start rados bench command for read
>>> or write CPU Idle % goes down ~10 and System load is increasing like
>>> anything.
>>>
>>> Hardware
>>>
>>> HpSL4540
>>>
>>
>> Please make sure the controller is on the newest firmware.  There used to
>> be a bug that would cause sequential write performance to bottleneck when
>> writeback cache was enabled on the RAID controller.
>
>
> Last month i have upgraded the firmwares for this hardware , so i hope
> they are up to date.
>
>
>>
>>
>> 32Core CPU
>>> 196G Memory
>>> 10G Network
>>>
>>
>> Be sure to check the network too.  We've seen a lot of cases where folks
>> have been burned by one of the NICs acting funky.
>>
>
> At a first view , Interface looks good and they are pushing data nicely (
> what ever they are getting )
>
>
>>
>>
>>> I don't think hardware is a problem.
>>>
>>> Please give me clues / pointers , how should i troubleshoot this problem.
>>>
>>>
>>>
>>> # rados bench -p glance-test 60 write
>>>   Maintaining 16 concurrent writes of 4194304 bytes for up to 60 seconds
>>> or 0 objects
>>>   Object prefix: benchmark_data_pouta-s01.pouta.csc.fi_2173350
>>> sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg
>>> lat
>>>   0   0 0 0 0 0 -
>>>  0
>>>   1  1620 4 15.9916   0.12308
>>>  0.10001
>>>   2  163721   41.984168   1.79104
>>> 0.827021
>>>   3  166852   69.3122   124  0.084304
>>> 0.854829
>>>   4  16   11498   97.9746   184   0.12285
>>> 0.614507
>>>   5  16   188   172   137.568   296  0.210669
>>> 0.449784
>>>   6  16   248   232   154.634   240  0.090418
>>> 0.390647
>>>   7  16   305   289165.11   228  0.069769
>>> 0.347957
>>>   8  16   331   315   157.471   104  0.026247
>>> 0.3345
>>>   9  16   361   345   153.306   120  0.082861
>>> 0.320711
>>>  10  16   380   364   145.57576  0.027964
>>> 0.310004
>>>  11  16   393   377   137.06752   3.73332
>>> 0.393318
>>>  12  16   448   432   143.971   220  0.334664
>>> 0.415606
>>>  13  16   476   460   141.508   112  0.271096
>>> 0.406574
>>>  14  16   497   481   137.39984  0.257794
>>> 0.412006
>>>  15  16   507   491   130.90640   1.49351
>>> 0.428057
>>>  16  16   529   513   115.04288  0.399384
>>>  0.48009
>>>  17  16   533   517   94.628616   5.50641
>>> 0.507804
>>>  18  16   537   52183.40516   4.42682
>>> 0.549951
>>>  19  16   538   52280.349 4   11.2052
>>> 0.570363
>>> 2015-09-02 09:26:18.398641min lat: 0.023851 max lat: 11.2052 avg lat:
>>> 0.570363
>>> sec Cur ops   started  finished  avg MB/s  cur MB/s  l

Re: [ceph-users] Ceph read / write : Terrible performance

[Vickey Singh]

Thank You Mark , please see my response below.

On Wed, Sep 2, 2015 at 5:23 PM, Mark Nelson <mnel...@redhat.com> wrote:

> On 09/02/2015 08:51 AM, Vickey Singh wrote:
>
>> Hello Ceph Experts
>>
>> I have a strange problem , when i am reading or writing to Ceph pool ,
>> its not writing properly. Please notice Cur MB/s which is going up and
>> down
>>
>> --- Ceph Hammer 0.94.2
>> -- CentOS 6, 2.6
>> -- Ceph cluster is healthy
>>
>
> You might find that CentOS7 gives you better performance.  In some cases
> we were seeing nearly 2X.


Wooo 2X , i would definitely plan for upgrade. Thanks


>
>
>
>>
>> One interesting thing is when every i start rados bench command for read
>> or write CPU Idle % goes down ~10 and System load is increasing like
>> anything.
>>
>> Hardware
>>
>> HpSL4540
>>
>
> Please make sure the controller is on the newest firmware.  There used to
> be a bug that would cause sequential write performance to bottleneck when
> writeback cache was enabled on the RAID controller.


Last month i have upgraded the firmwares for this hardware , so i hope they
are up to date.


>
>
> 32Core CPU
>> 196G Memory
>> 10G Network
>>
>
> Be sure to check the network too.  We've seen a lot of cases where folks
> have been burned by one of the NICs acting funky.
>

At a first view , Interface looks good and they are pushing data nicely (
what ever they are getting )


>
>
>> I don't think hardware is a problem.
>>
>> Please give me clues / pointers , how should i troubleshoot this problem.
>>
>>
>>
>> # rados bench -p glance-test 60 write
>>   Maintaining 16 concurrent writes of 4194304 bytes for up to 60 seconds
>> or 0 objects
>>   Object prefix: benchmark_data_pouta-s01.pouta.csc.fi_2173350
>> sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg
>> lat
>>   0   0 0 0 0 0 -
>>  0
>>   1  1620 4 15.9916   0.12308
>>  0.10001
>>   2  163721   41.984168   1.79104
>> 0.827021
>>   3  166852   69.3122   124  0.084304
>> 0.854829
>>   4  16   11498   97.9746   184   0.12285
>> 0.614507
>>   5  16   188   172   137.568   296  0.210669
>> 0.449784
>>   6  16   248   232   154.634   240  0.090418
>> 0.390647
>>   7  16   305   289165.11   228  0.069769
>> 0.347957
>>   8  16   331   315   157.471   104  0.026247
>> 0.3345
>>   9  16   361   345   153.306   120  0.082861
>> 0.320711
>>  10  16   380   364   145.57576  0.027964
>> 0.310004
>>  11  16   393   377   137.06752   3.73332
>> 0.393318
>>  12  16   448   432   143.971   220  0.334664
>> 0.415606
>>  13  16   476   460   141.508   112  0.271096
>> 0.406574
>>  14  16   497   481   137.39984  0.257794
>> 0.412006
>>  15  16   507   491   130.90640   1.49351
>> 0.428057
>>  16  16   529   513   115.04288  0.399384
>>  0.48009
>>  17  16   533   517   94.628616   5.50641
>> 0.507804
>>  18  16   537   52183.40516   4.42682
>> 0.549951
>>  19  16   538   52280.349 4   11.2052
>> 0.570363
>> 2015-09-02 09:26:18.398641min lat: 0.023851 max lat: 11.2052 avg lat:
>> 0.570363
>> sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg
>> lat
>>  20  16   538   522   77.3611 0 -
>> 0.570363
>>  21  16   540   524   74.8825 4   8.88847
>> 0.591767
>>  22  16   542   526   72.5748 8   1.41627
>> 0.593555
>>  23  16   543   527   70.2873 48.0856
>> 0.607771
>>  24  16   555   539   69.567448  0.145199
>> 0.781685
>>  25  16   560   544   68.0177201.4342
>> 0.787017
>>  26  16   564   548   66.424116  0.451905
>>  0.78765
>>  27  16   566   550   64.7055 8  0.611129
>> 0.787898
>>  28  16   570   554   63.313816   2.51086
>> 0.797067
>>  29  16   570   

[ceph-users] Ceph read / write : Terrible performance

[Vickey Singh]

Hello Ceph Experts

I have a strange problem , when i am reading or writing to Ceph pool , its
not writing properly. Please notice Cur MB/s which is going up and down

--- Ceph Hammer 0.94.2
-- CentOS 6, 2.6
-- Ceph cluster is healthy


One interesting thing is when every i start rados bench command for read or
write CPU Idle % goes down ~10 and System load is increasing like anything.

Hardware

HpSL4540
32Core CPU
196G Memory
10G Network

I don't think hardware is a problem.

Please give me clues / pointers , how should i troubleshoot this problem.



# rados bench -p glance-test 60 write
 Maintaining 16 concurrent writes of 4194304 bytes for up to 60 seconds or
0 objects
 Object prefix: benchmark_data_pouta-s01.pouta.csc.fi_2173350
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
 0   0 0 0 0 0 - 0
 1  1620 4 15.9916   0.12308   0.10001
 2  163721   41.984168   1.79104  0.827021
 3  166852   69.3122   124  0.084304  0.854829
 4  16   11498   97.9746   184   0.12285  0.614507
 5  16   188   172   137.568   296  0.210669  0.449784
 6  16   248   232   154.634   240  0.090418  0.390647
 7  16   305   289165.11   228  0.069769  0.347957
 8  16   331   315   157.471   104  0.0262470.3345
 9  16   361   345   153.306   120  0.082861  0.320711
10  16   380   364   145.57576  0.027964  0.310004
11  16   393   377   137.06752   3.73332  0.393318
12  16   448   432   143.971   220  0.334664  0.415606
13  16   476   460   141.508   112  0.271096  0.406574
14  16   497   481   137.39984  0.257794  0.412006
15  16   507   491   130.90640   1.49351  0.428057
16  16   529   513   115.04288  0.399384   0.48009
17  16   533   517   94.628616   5.50641  0.507804
18  16   537   52183.40516   4.42682  0.549951
19  16   538   52280.349 4   11.2052  0.570363
2015-09-02 09:26:18.398641min lat: 0.023851 max lat: 11.2052 avg lat:
0.570363
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
20  16   538   522   77.3611 0 -  0.570363
21  16   540   524   74.8825 4   8.88847  0.591767
22  16   542   526   72.5748 8   1.41627  0.593555
23  16   543   527   70.2873 48.0856  0.607771
24  16   555   539   69.567448  0.145199  0.781685
25  16   560   544   68.0177201.4342  0.787017
26  16   564   548   66.424116  0.451905   0.78765
27  16   566   550   64.7055 8  0.611129  0.787898
28  16   570   554   63.313816   2.51086  0.797067
29  16   570   554   61.5549 0 -  0.797067
30  16   572   556   60.1071 4   7.71382  0.830697
31  16   577   561   59.051520   23.3501  0.916368
32  16   590   574   58.870552  0.336684  0.956958
33  16   591   575   57.4986 4   1.92811  0.958647
34  16   591   575   56.0961 0 -  0.958647
35  16   591   575   54.7603 0 -  0.958647
36  16   597   581   54.0447 8  0.187351   1.00313
37  16   625   609   52.8394   112   2.12256   1.09256
38  16   631   61552.22724   1.57413   1.10206
39  16   638   622   51.723228   4.41663   1.15086
2015-09-02 09:26:40.510623min lat: 0.023851 max lat: 27.6704 avg lat:
1.15657
   sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
40  16   652   636   51.810256  0.113345   1.15657
41  16   682   666   53.1443   120  0.041251   1.17813
42  16   685   669   52.339512  0.501285   1.17421
43  15   690   675   51.795524   2.26605   1.18357
44  16   728   712   53.6062   148  0.589826   1.17478
45  16   728   712   52.6158 0 -   1.17478
46  16   728   712   51.6613 0 -   1.17478
47  16   728   712   50.7407 0 -   1.17478
48  16   772   756   52.933244  0.2348111.1946
49  16   835   819   56.3577   252   5.67087   1.12063
50  16   890   874   59.1252   220  0.230806   1.06778
51  16   896   880   58.5409   

[ceph-users] Opensource plugin for pulling out cluster recovery and client IO metric

[Vickey Singh]

Hello Ceph Geeks

I am planning to develop a python plugin that pulls out cluster *recovery
IO* and *client IO* operation metrics , that can be further used with
collectd.

*For example , i need to take out these values*

*recovery io 814 MB/s, 101 objects/s*
*client io 85475 kB/s rd, 1430 kB/s wr, 32 op/s*


Could you please help me in understanding how *ceph -s*  and *ceph -w*
outputs *prints cluster recovery IO and client IO information*.
Where this information is coming from. *Is it coming from perf dump* ? If
yes then which section of perf dump output is should focus on. If not then
how can i get this values.

I tried *ceph --admin-daemon /var/run/ceph/ceph-osd.48.asok perf dump* ,
but it generates hell lot of information and i am confused which section of
output should i use.


Please help

Thanks in advance
Vickey
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Opensource plugin for pulling out cluster recovery and client IO metric

[Vickey Singh]

Hello Ceph Geeks

I am planning to develop a python plugin that pulls out cluster *recovery
IO* and *client IO* operation metrics , that can be further used with
collectd.

*For example , i need to take out these values*

*recovery io 814 MB/s, 101 objects/s*
*client io 85475 kB/s rd, 1430 kB/s wr, 32 op/s*


Could you please help me in understanding how *ceph -s*  and *ceph -w*
 outputs *prints cluster recovery IO and client IO information*.
Where this information is coming from. *Is it coming from perf dump* ? If
yes then which section of perf dump output is should focus on. If not then
how can i get this values.

I tried *ceph --admin-daemon /var/run/ceph/ceph-osd.48.asok perf dump* ,
but it generates hell lot of information and i am confused which section of
output should i use.


Please help

Thanks in advance
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cache tier best practices

[Vickey Singh]

Thank you guys , this answers my query


Cheers

Vickey

On Thu, Aug 13, 2015 at 8:02 PM, Bill Sanders billysand...@gmail.com
wrote:

 I think you're looking for this.

 http://ceph.com/docs/master/man/8/rbd/#cmdoption-rbd--order

 It's used when you create the RBD images.  1MB is order=20, 512 is
 order=19.

 Thanks,
 Bill Sanders


 On Thu, Aug 13, 2015 at 1:31 AM, Vickey Singh vickey.singh22...@gmail.com
  wrote:

 Thanks Nick for your suggestion.

 Can you also tell how i can reduce RBD block size to 512K or 1M , do i
 need to put something in clients ceph.conf  ( what parameter i need to set )

 Thanks once again

 - Vickey

 On Wed, Aug 12, 2015 at 4:49 PM, Nick Fisk n...@fisk.me.uk wrote:

  -Original Message-
  From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf
 Of
  Dominik Zalewski
  Sent: 12 August 2015 14:40
  To: ceph-us...@ceph.com
  Subject: [ceph-users] Cache tier best practices
 
  Hi,
 
  I would like to hear from people who use cache tier in Ceph about best
  practices and things I should avoid.
 
  I remember hearing that it wasn't that stable back then. Has it
 changed in
  Hammer release?

 It's not so much the stability, but the performance. If your working set
 will sit mostly in the cache tier and won't tend to change then you might
 be alright. Otherwise you will find that performance is very poor.

 Only tip I can really give is that I have found dropping the RBD block
 size down to 512kb-1MB helps quite a bit as it makes the cache more
 effective and also minimises the amount of data transferred on each
 promotion/flush.

 
  Any tips and tricks are much appreciated!
 
  Thanks
 
  Dominik




 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cache tier best practices

[Vickey Singh]

Thanks Nick for your suggestion.

Can you also tell how i can reduce RBD block size to 512K or 1M , do i need
to put something in clients ceph.conf  ( what parameter i need to set )

Thanks once again

- Vickey

On Wed, Aug 12, 2015 at 4:49 PM, Nick Fisk n...@fisk.me.uk wrote:

  -Original Message-
  From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
  Dominik Zalewski
  Sent: 12 August 2015 14:40
  To: ceph-us...@ceph.com
  Subject: [ceph-users] Cache tier best practices
 
  Hi,
 
  I would like to hear from people who use cache tier in Ceph about best
  practices and things I should avoid.
 
  I remember hearing that it wasn't that stable back then. Has it changed
 in
  Hammer release?

 It's not so much the stability, but the performance. If your working set
 will sit mostly in the cache tier and won't tend to change then you might
 be alright. Otherwise you will find that performance is very poor.

 Only tip I can really give is that I have found dropping the RBD block
 size down to 512kb-1MB helps quite a bit as it makes the cache more
 effective and also minimises the amount of data transferred on each
 promotion/flush.

 
  Any tips and tricks are much appreciated!
 
  Thanks
 
  Dominik




 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] 160 Thousand ceph-client.admin.*.asok files : Wired problem , never seen before

[Vickey Singh]

Hello Community

I am facing a very wired problem with Ceph socket files.

For all monitor nodes under /var/run/ceph/ i can see ~160 Thousands asok
files , most of the file names are ceph-client.admin.*

*If i delete these files are the getting generated very quickly.*



*Could someone please answer*

*1# What are these file , i have never seen this many entries for
ceph-client.admin files*
*2# Why these files keep on generating*
*3# What should i do now *


*Necessary Outputs *


4 node cluster ( 3 monitors+osd and 1 osd)
Ceph Hammer , CentOS 6.6


[root@node1 ceph]# ls -l /var/run/ceph/*.asok | wc -l
159135
[root@node2 ceph]# ls -l /var/run/ceph/*.asok | wc -l
152051
[root@node3 ceph]# ls -l /var/run/ceph/*.asok | wc -l
164966
[root@node4 ceph]# ls -l /var/run/ceph/*.asok | wc -l
29


*160 Thousand entries like this *

srwxr-xr-x. 1 root root 0 Aug  2 22:39
ceph-client.admin.999606.140148473887088.asok
srwxr-xr-x. 1 root root 0 Jul 24 08:26
ceph-client.admin.999637.140148473887088.asok
srwxr-xr-x. 1 root root 0 Jul 30 18:39
ceph-client.admin.999656.140148473887088.asok
srwxr-xr-x. 1 root root 0 Jul 25 05:12
ceph-client.admin.999671.140148473887056.asok
srwxr-xr-x. 1 root root 0 Jul 24 08:26
ceph-client.admin.999732.140148473887088.asok
srwxr-xr-x. 1 root root 0 Jul 21 15:23
ceph-client.admin.999780.140148473887088.asok
srwxr-xr-x. 1 root root 0 Jul 24 16:41
ceph-client.admin.999784.140148473887088.asok
srwxr-xr-x. 1 root root 0 Jul 24 08:26
ceph-client.admin.999788.140148473887088.asok
srwxr-xr-x. 1 root root 0 Aug  1 05:40
ceph-client.admin.999790.140148473887088.asok
srwxr-xr-x. 1 root root 0 Jul 28 14:22
ceph-client.admin.999800.140148473887088.asok
srwxr-xr-x. 1 root root 0 Jul 21 15:23
ceph-client.admin.999805.140148473887088.asok
srwxr-xr-x. 1 root root 0 Jul 25 17:44
ceph-client.admin.999820.140148473887088.asok
srwxr-xr-x. 1 root root 0 Jul 24 16:41
ceph-client.admin.999823.140148473887088.asok
srwxr-xr-x. 1 root root 0 Jul 24 20:51
ceph-client.admin.999835.140148473887056.asok
srwxr-xr-x. 1 root root 0 Jul 21 15:23
ceph-client.admin.999857.140148473887088.asok
srwxr-xr-x. 1 root root 0 Jul 25 13:34
ceph-client.admin.999859.140148473887056.asok
srwxr-xr-x. 1 root root 0 Jul 24 20:51
ceph-client.admin.999870.140148473887056.asok
srwxr-xr-x. 1 root root 0 Jul 24 16:41
ceph-client.admin.20.140148473887088.asok
srwxr-xr-x. 1 root root 0 Jul 24 20:51
ceph-client.admin.23.140148473887056.asok
srwxr-xr-x. 1 root root 0 Jul 23 20:05
ceph-client.admin.78.140148473887088.asok

Regards
Vickey
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Redhat Storage Ceph Storage 1.3 released

[Vickey Singh]

Hello Ceph lovers

You would have noticed that recently RedHat has released RedHat Ceph
Storage 1.3

http://redhatstorage.redhat.com/2015/06/25/announcing-red-hat-ceph-storage-1-3/

My question is

- What's the exact version number of OpenSource Ceph is provided with this
Product
- RHCS 1.3 Features that are mentioned in the blog , will all of them
present in open source Ceph.



Regards
Vickey
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Beginners ceph journal question

[Vickey Singh]

Thanks Michael for your response.

Could you also please help in understanding

#1  On my ceph cluster , how can i confirm if journal is on block device
partition or on file ?

#2  Is it true that by default ceph-deploy creates journal on dedicated
partition and data on another partition if i use the command  ceph-deploy
osd create ceph-node1:/dev/sdb

I want to understand the concept of journals creation in Ceph. Hope you
will help me.

- vicky

On Tue, Jun 9, 2015 at 5:28 PM, Michael Kuriger mk7...@yp.com wrote:

   You could mount /dev/sdb to a filesystem, such as /ceph-disk, and then
 do this:
 ceph-deploy osd create ceph-node1:/ceph-disk

  Your journal would be a file doing it this way.



 [image: yp]



 Michael Kuriger

 Sr. Unix Systems Engineer

 * mk7...@yp.com |( 818-649-7235

   From: Vickey Singh vickey.singh22...@gmail.com
 Date: Tuesday, June 9, 2015 at 12:21 AM
 To: ceph-users@lists.ceph.com ceph-users@lists.ceph.com
 Subject: [ceph-users] Beginners ceph journal question

   Hello Cephers

  Beginners question on Ceph Journals creation. Need answers from experts.

  - Is it true that by default ceph-deploy creates journal on dedicated
 partition and data on another partition. It does not creates journal on
 file ??

  ceph-deploy osd create ceph-node1:/dev/sdb

  This commands is creating
 data partition : /dev/sdb2
 Journal Partition : /dev/sdb1

  In ceph-deploy command i have not specified journal partition but still
 it creates a journal on sdb1 ?

  - How can i confirm if journal is on block device partition or on file ?

  - How can i create journal on a file ? command would be helpful ?

  Regards
 Vicky

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Radosgw multi zone data replication failure

[Vickey Singh]

Hello Geeks

Need your help and advice in this problem.

- VS -

On Tue, Apr 28, 2015 at 12:48 AM, Vickey Singh vickey.singh22...@gmail.com
wrote:

 Hello Alfredo / Craig

 First of all Thank You So much for replying and giving your precious time
 to this problem.

 @Alfredo : I tried version radosgw-agent version 1.2.2 and the case has
 progressed a lot. ( below are some the logs )


 I am now getting

 *2015-04-28 00:35:14,781 5132 [radosgw_agent][INFO  ]
 http://us-east-1.crosslogic.com:7480 http://us-east-1.crosslogic.com:7480
 endpoint does not support versioning*

 *2015-04-28 00:35:14,781 5132 [radosgw_agent][WARNIN] encountered issues
 reaching to endpoint http://us-east-1.crosslogic.com:7480
 http://us-east-1.crosslogic.com:7480*

 *2015-04-28 00:35:14,782 5132 [radosgw_agent][WARNIN] HTTP Error 403:
 Forbidden*

 I am using CIVETWEB , any further help in this would be really helpful.


 [root@us-east-1 ceph]#

 [root@us-east-1 ceph]# radosgw-agent -c cluster-data-sync.conf

 2015-04-28 00:35:14,750 5132 [radosgw_agent][INFO  ]  ____
   __   ___  ___

 2015-04-28 00:35:14,750 5132 [radosgw_agent][INFO  ] /__` \ / |\ | /  `
   /\  / _` |__  |\ |  |

 2015-04-28 00:35:14,751 5132 [radosgw_agent][INFO  ] .__/  |  | \| \__,
 /~~\ \__ |___ | \|  |

 2015-04-28 00:35:14,751 5132 [radosgw_agent][INFO  ]
 v1.2.2

 2015-04-28 00:35:14,751 5132 [radosgw_agent][INFO  ] agent options:

 2015-04-28 00:35:14,752 5132 [radosgw_agent][INFO  ]  args:

 2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO  ]conf
 : None

 2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO  ]dest_access_key
 : 

 2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO  ]dest_secret_key
 : 

 2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO  ]destination
 : http://us-west-1.crosslogic.com:7480

 2015-04-28 00:35:14,753 5132 [radosgw_agent][INFO  ]
 incremental_sync_delay: 30

 2015-04-28 00:35:14,754 5132 [radosgw_agent][INFO  ]lock_timeout
 : 60

 2015-04-28 00:35:14,754 5132 [radosgw_agent][INFO  ]log_file
 : /var/log/radosgw/radosgw-sync-us-east-west.log

 2015-04-28 00:35:14,756 5132 [radosgw_agent][INFO  ]log_lock_time
 : 20

 2015-04-28 00:35:14,756 5132 [radosgw_agent][INFO  ]max_entries
 : 1000

 2015-04-28 00:35:14,757 5132 [radosgw_agent][INFO  ]metadata_only
 : False

 2015-04-28 00:35:14,757 5132 [radosgw_agent][INFO  ]num_workers
 : 1

 2015-04-28 00:35:14,758 5132 [radosgw_agent][INFO  ]
 object_sync_timeout   : 216000

 2015-04-28 00:35:14,758 5132 [radosgw_agent][INFO  ]
 prepare_error_delay   : 10

 2015-04-28 00:35:14,758 5132 [radosgw_agent][INFO  ]quiet
 : False

 2015-04-28 00:35:14,758 5132 [radosgw_agent][INFO  ]
 rgw_data_log_window   : 30

 2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO  ]source
 : None

 2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO  ]src_access_key
 : 

 2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO  ]src_secret_key
 : 

 2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO  ]src_zone
 : None

 2015-04-28 00:35:14,759 5132 [radosgw_agent][INFO  ]sync_scope
 : incremental

 2015-04-28 00:35:14,760 5132 [radosgw_agent][INFO  ]test_server_host
 : None

 2015-04-28 00:35:14,760 5132 [radosgw_agent][INFO  ]test_server_port
 : 8080

 2015-04-28 00:35:14,761 5132 [radosgw_agent][INFO  ]verbose
 : False

 2015-04-28 00:35:14,761 5132 [radosgw_agent][INFO  ]versioned
 : False

 2015-04-28 00:35:14,761 5132 [radosgw_agent.client][INFO  ] creating
 connection to endpoint: http://us-west-1.crosslogic.com:7480

 region map is: {u'us': [u'us-west', u'us-east']}

 *2015-04-28 00:35:14,781 5132 [radosgw_agent][INFO  ]
 http://us-east-1.crosslogic.com:7480 http://us-east-1.crosslogic.com:7480
 endpoint does not support versioning*

 *2015-04-28 00:35:14,781 5132 [radosgw_agent][WARNIN] encountered issues
 reaching to endpoint http://us-east-1.crosslogic.com:7480
 http://us-east-1.crosslogic.com:7480*

 *2015-04-28 00:35:14,782 5132 [radosgw_agent][WARNIN] HTTP Error 403:
 Forbidden*

 2015-04-28 00:35:14,782 5132 [radosgw_agent.client][INFO  ] creating
 connection to endpoint: http://us-east-1.crosslogic.com:7480

 2015-04-28 00:35:14,784 5132 [radosgw_agent.client][INFO  ] creating
 connection to endpoint: http://us-west-1.crosslogic.com:7480

 2015-04-28 00:35:14,785 5132 [radosgw_agent.client][INFO  ] creating
 connection to endpoint: http://us-east-1.crosslogic.com:7480

 2015-04-28 00:35:14,787 5132 [radosgw_agent.client][INFO  ] creating
 connection to endpoint: http://us-west-1.crosslogic.com:7480

 *2015-04-28 00:35:14,807 5132 [radosgw_agent.sync][ERROR

Re: [ceph-users] Ceph Radosgw multi zone data replication failure

[Vickey Singh]

Any help with related to this problem would be highly appreciated.

-VS-

On Sun, Apr 26, 2015 at 6:01 PM, Vickey Singh vickey.singh22...@gmail.com
wrote:

 Hello Geeks


 I am trying to setup Ceph Radosgw multi site data replication using
 official documentation
 http://ceph.com/docs/master/radosgw/federated-config/#multi-site-data-replication


 Everything seems to work except radosgw-agent sync , Request you to please
 check the below outputs and help me in any possible way.


 *Environment : *


 CentOS 7.0.1406

 Ceph Versino 0.87.1

 Rados Gateway configured using Civetweb



 *Radosgw zone list : Works nicely *


 [root@us-east-1 ceph]# radosgw-admin zone list --name
 client.radosgw.us-east-1

 { zones: [

 us-west,

 us-east]}

 [root@us-east-1 ceph]#


 *Curl request to master zone : Works nicely *


 [root@us-east-1 ceph]# curl http://us-east-1.crosslogic.com:7480

 ?xml version=1.0 encoding=UTF-8?ListAllMyBucketsResult xmlns=
 http://s3.amazonaws.com/doc/2006-03-01/
 OwnerIDanonymous/IDDisplayName/DisplayName/OwnerBuckets/Buckets/ListAllMyBucketsResult

 [root@us-east-1 ceph]#


 *Curl request to secondary zone : Works nicely *


 [root@us-east-1 ceph]# curl http://us-west-1.crosslogic.com:7480

 ?xml version=1.0 encoding=UTF-8?ListAllMyBucketsResult xmlns=
 http://s3.amazonaws.com/doc/2006-03-01/
 OwnerIDanonymous/IDDisplayName/DisplayName/OwnerBuckets/Buckets/ListAllMyBucketsResult

 [root@us-east-1 ceph]#


 *Rados Gateway agent configuration file : Seems correct, no TYPO errors*


 [root@us-east-1 ceph]# cat cluster-data-sync.conf

 src_access_key: M7QAKDH8CYGTK86CG93U

 src_secret_key: 0xQR6PINk23W\/GYrWJ14aF+1stG56M6xMkqkdloO

 destination: http://us-west-1.crosslogic.com:7480

 dest_access_key: ZQ32ES1WAWPG05YMZ7T7

 dest_secret_key: INvk8AkrZRsejLEL34yRpMLmOqydt8ncOXy4RHCM

 log_file: /var/log/radosgw/radosgw-sync-us-east-west.log

 [root@us-east-1 ceph]#


 *Rados Gateway agent SYNC : Fails , however it can fetch region map so i
 think src and dest KEYS are correct. But don't know why it fails on
 AttributeError *



 *[root@us-east-1 ceph]# radosgw-agent -c cluster-data-sync.conf*

 *region map is: {u'us': [u'us-west', u'us-east']}*

 *Traceback (most recent call last):*

 *  File /usr/bin/radosgw-agent, line 21, in module*

 *sys.exit(main())*

 *  File /usr/lib/python2.7/site-packages/radosgw_agent/cli.py, line 275,
 in main*

 *except client.ClientException as e:*

 *AttributeError: 'module' object has no attribute 'ClientException'*

 *[root@us-east-1 ceph]#*


 *Can query to Ceph cluster using us-east-1 ID*


 [root@us-east-1 ceph]# ceph -s --name client.radosgw.us-east-1

 cluster 9609b429-eee2-4e23-af31-28a24fcf5cbc

  health HEALTH_OK

  monmap e3: 3 mons at {ceph-node1=
 192.168.1.101:6789/0,ceph-node2=192.168.1.102:6789/0,ceph-node3=192.168.1.103:6789/0},
 election epoch 448, quorum 0,1,2 ceph-node1,ceph-node2,ceph-node3

  osdmap e1063: 9 osds: 9 up, 9 in

   pgmap v8473: 1500 pgs, 43 pools, 374 MB data, 2852 objects

 1193 MB used, 133 GB / 134 GB avail

 1500 active+clean

 [root@us-east-1 ceph]#


 *Can query to Ceph cluster using us-west-1 ID*


 [root@us-east-1 ceph]# ceph -s --name client.radosgw.us-west-1

 cluster 9609b429-eee2-4e23-af31-28a24fcf5cbc

  health HEALTH_OK

  monmap e3: 3 mons at {ceph-node1=
 192.168.1.101:6789/0,ceph-node2=192.168.1.102:6789/0,ceph-node3=192.168.1.103:6789/0},
 election epoch 448, quorum 0,1,2 ceph-node1,ceph-node2,ceph-node3

  osdmap e1063: 9 osds: 9 up, 9 in

   pgmap v8473: 1500 pgs, 43 pools, 374 MB data, 2852 objects

 1193 MB used, 133 GB / 134 GB avail

 1500 active+clean

 [root@us-east-1 ceph]#


 *Hope these packages are correct*


 [root@us-east-1 ceph]# rpm -qa | egrep -i ceph|radosgw

 libcephfs1-0.87.1-0.el7.centos.x86_64

 ceph-common-0.87.1-0.el7.centos.x86_64

 python-ceph-0.87.1-0.el7.centos.x86_64

 ceph-radosgw-0.87.1-0.el7.centos.x86_64

 ceph-release-1-0.el7.noarch

 ceph-0.87.1-0.el7.centos.x86_64

 radosgw-agent-1.2.1-0.el7.centos.noarch

 [root@us-east-1 ceph]#



 Regards

 VS

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Radosgw multi zone data replication failure

[Vickey Singh]

}}[root@us-east-1 ceph]#

[root@us-east-1 ceph]#

- VS -



On Mon, Apr 27, 2015 at 11:59 PM, Alfredo Deza ad...@redhat.com wrote:

 Hi Vickey (and all)

 It looks like this issue was introduced as part of the 1.2.1 release.

 I just finished getting 1.2.2 out (try upgrading please). You should no
 longer see that
 error.

 Hope that helps!

 -Alfredo

 - Original Message -
 From: Craig Lewis cle...@centraldesktop.com
 To: Vickey Singh vickey.singh22...@gmail.com
 Cc: ceph-users@lists.ceph.com
 Sent: Monday, April 27, 2015 4:23:52 PM
 Subject: Re: [ceph-users] Ceph Radosgw multi zone data replication failure

  [root@us-east-1 ceph] # ceph -s --name client.radosgw.us-east-1


  [root@us-east-1 ceph]# ceph -s --name client.radosgw.us-west-1

 Are you trying to setup two zones on one cluster? That's possible, but
 you'll also want to spend some time on your CRUSH map making sure that the
 two zones are as independent as possible (no shared disks, etc).

 Are you using Civetweb or Apache + FastCGI?

 Can you include the output (from both clusters):
 radosgw-admin --name=client.radosgw.us-east-1 region get
 radosgw-admin --name=client.radosgw.us-east-1 zone get

 Double check that both system users exist in both clusters, with the same
 secret.




 On Sun, Apr 26, 2015 at 8:01 AM, Vickey Singh 
 vickey.singh22...@gmail.com  wrote:





 Hello Geeks




 I am trying to setup Ceph Radosgw multi site data replication using
 official documentation
 http://ceph.com/docs/master/radosgw/federated-config/#multi-site-data-replication




 Everything seems to work except radosgw-agent sync , Request you to please
 check the below outputs and help me in any possible way.




 Environment :




 CentOS 7.0.1406

 Ceph Versino 0.87.1

 Rados Gateway configured using Civetweb







 Radosgw zone list : Works nicely





 [root@us-east-1 ceph]# radosgw-admin zone list --name
 client.radosgw.us-east-1

 { zones: [

 us-west,

 us-east]}

 [root@us-east-1 ceph]#




 Curl request to master zone : Works nicely





 [root@us-east-1 ceph]# curl http://us-east-1.crosslogic.com:7480

 ?xml version=1.0 encoding=UTF-8?ListAllMyBucketsResult xmlns=
 http://s3.amazonaws.com/doc/2006-03-01/
 OwnerIDanonymous/IDDisplayName/DisplayName/OwnerBuckets/Buckets/ListAllMyBucketsResult

 [root@us-east-1 ceph]#




 Curl request to secondary zone : Works nicely




 [root@us-east-1 ceph]# curl http://us-west-1.crosslogic.com:7480

 ?xml version=1.0 encoding=UTF-8?ListAllMyBucketsResult xmlns=
 http://s3.amazonaws.com/doc/2006-03-01/
 OwnerIDanonymous/IDDisplayName/DisplayName/OwnerBuckets/Buckets/ListAllMyBucketsResult

 [root@us-east-1 ceph]#




 Rados Gateway agent configuration file : Seems correct, no TYPO errors





 [root@us-east-1 ceph] # cat cluster-data-sync.conf

 src_access_key: M7QAKDH8CYGTK86CG93U

 src_secret_key: 0xQR6PINk23W\/GYrWJ14aF+1stG56M6xMkqkdloO

 destination: http://us-west-1.crosslogic.com:7480

 dest_access_key: ZQ32ES1WAWPG05YMZ7T7

 dest_secret_key: INvk8AkrZRsejLEL34yRpMLmOqydt8ncOXy4RHCM

 log_file: /var/log/radosgw/radosgw-sync-us-east-west.log

 [root@us-east-1 ceph]#




 Rados Gateway agent SYNC : Fails , however it can fetch region map so i
 think src and dest KEYS are correct. But don't know why it fails on
 AttributeError





 [root@us-east-1 ceph]# radosgw-agent -c cluster-data-sync.conf


 region map is: {u'us': [u'us-west', u'us-east']}

 Traceback (most recent call last):

 File /usr/bin/radosgw-agent, line 21, in module

 sys.exit(main())

 File /usr/lib/python2.7/site-packages/radosgw_agent/cli.py, line 275, in
 main

 except client.ClientException as e:

 AttributeError: 'module' object has no attribute 'ClientException'

 [root@us-east-1 ceph]#




 Can query to Ceph cluster using us-east-1 ID




 [root@us-east-1 ceph] # ceph -s --name client.radosgw.us-east-1

 cluster 9609b429-eee2-4e23-af31-28a24fcf5cbc

 health HEALTH_OK

 monmap e3: 3 mons at {ceph-node1=
 192.168.1.101:6789/0,ceph-node2=192.168.1.102:6789/0,ceph-node3=192.168.1.103:6789/0
 }, election epoch 448, quorum 0,1,2 ceph-node1,ceph-node2,ceph-node3

 osdmap e1063: 9 osds: 9 up, 9 in

 pgmap v8473: 1500 pgs, 43 pools, 374 MB data, 2852 objects

 1193 MB used, 133 GB / 134 GB avail

 1500 active+clean

 [root@us-east-1 ceph]#




 Can query to Ceph cluster using us-west-1 ID





 [root@us-east-1 ceph]# ceph -s --name client.radosgw.us-west-1

 cluster 9609b429-eee2-4e23-af31-28a24fcf5cbc

 health HEALTH_OK

 monmap e3: 3 mons at {ceph-node1=
 192.168.1.101:6789/0,ceph-node2=192.168.1.102:6789/0,ceph-node3=192.168.1.103:6789/0
 }, election epoch 448, quorum 0,1,2 ceph-node1,ceph-node2,ceph-node3

 osdmap e1063: 9 osds: 9 up, 9 in

 pgmap v8473: 1500 pgs, 43 pools, 374 MB data, 2852 objects

 1193 MB used, 133 GB / 134 GB avail

 1500 active+clean

 [root@us-east-1 ceph]#




 Hope these packages are correct





 [root@us-east-1 ceph]# rpm -qa | egrep -i ceph|radosgw

 libcephfs1-0.87.1-0.el7.centos.x86_64

 ceph-common-0.87.1-0

[ceph-users] Ceph Radosgw multi site data replication failure :

[Vickey Singh]

Hello Geeks


I am trying to setup Ceph Radosgw multi site data replication using
official documentation
http://ceph.com/docs/master/radosgw/federated-config/#multi-site-data-replication


Everything seems to work except radosgw-agent sync , Request you to please
check the below outputs and help me in any possible way.


*Environment : *


CentOS 7.0.1406

Ceph Versino 0.87.1

Rados Gateway configured using Civetweb



*Radosgw zone list : Works nicely *


[root@us-east-1 ceph]# radosgw-admin zone list --name
client.radosgw.us-east-1

{ zones: [

us-west,

us-east]}

[root@us-east-1 ceph]#


*Curl request to master zone : Works nicely *


[root@us-east-1 ceph]# curl http://us-east-1.crosslogic.com:7480

?xml version=1.0 encoding=UTF-8?ListAllMyBucketsResult xmlns=
http://s3.amazonaws.com/doc/2006-03-01/
OwnerIDanonymous/IDDisplayName/DisplayName/OwnerBuckets/Buckets/ListAllMyBucketsResult

[root@us-east-1 ceph]#


*Curl request to secondary zone : Works nicely *


[root@us-east-1 ceph]# curl http://us-west-1.crosslogic.com:7480

?xml version=1.0 encoding=UTF-8?ListAllMyBucketsResult xmlns=
http://s3.amazonaws.com/doc/2006-03-01/
OwnerIDanonymous/IDDisplayName/DisplayName/OwnerBuckets/Buckets/ListAllMyBucketsResult

[root@us-east-1 ceph]#


*Rados Gateway agent configuration file : Seems correct, no TYPO errors*


[root@us-east-1 ceph]# cat cluster-data-sync.conf

src_access_key: M7QAKDH8CYGTK86CG93U

src_secret_key: 0xQR6PINk23W\/GYrWJ14aF+1stG56M6xMkqkdloO

destination: http://us-west-1.crosslogic.com:7480

dest_access_key: ZQ32ES1WAWPG05YMZ7T7

dest_secret_key: INvk8AkrZRsejLEL34yRpMLmOqydt8ncOXy4RHCM

log_file: /var/log/radosgw/radosgw-sync-us-east-west.log

[root@us-east-1 ceph]#


*Rados Gateway agent SYNC : Fails , however it can fetch region map so i
think src and dest KEYS are correct. But don't know why it fails on
AttributeError *



*[root@us-east-1 ceph]# radosgw-agent -c cluster-data-sync.conf*

*region map is: {u'us': [u'us-west', u'us-east']}*

*Traceback (most recent call last):*

*  File /usr/bin/radosgw-agent, line 21, in module*

*sys.exit(main())*

*  File /usr/lib/python2.7/site-packages/radosgw_agent/cli.py, line 275,
in main*

*except client.ClientException as e:*

*AttributeError: 'module' object has no attribute 'ClientException'*

*[root@us-east-1 ceph]#*


*Can query to Ceph cluster using us-east-1 ID*


[root@us-east-1 ceph]# ceph -s --name client.radosgw.us-east-1

cluster 9609b429-eee2-4e23-af31-28a24fcf5cbc

 health HEALTH_OK

 monmap e3: 3 mons at {ceph-node1=
192.168.1.101:6789/0,ceph-node2=192.168.1.102:6789/0,ceph-node3=192.168.1.103:6789/0},
election epoch 448, quorum 0,1,2 ceph-node1,ceph-node2,ceph-node3

 osdmap e1063: 9 osds: 9 up, 9 in

  pgmap v8473: 1500 pgs, 43 pools, 374 MB data, 2852 objects

1193 MB used, 133 GB / 134 GB avail

1500 active+clean

[root@us-east-1 ceph]#


*Can query to Ceph cluster using us-west-1 ID*


[root@us-east-1 ceph]# ceph -s --name client.radosgw.us-west-1

cluster 9609b429-eee2-4e23-af31-28a24fcf5cbc

 health HEALTH_OK

 monmap e3: 3 mons at {ceph-node1=
192.168.1.101:6789/0,ceph-node2=192.168.1.102:6789/0,ceph-node3=192.168.1.103:6789/0},
election epoch 448, quorum 0,1,2 ceph-node1,ceph-node2,ceph-node3

 osdmap e1063: 9 osds: 9 up, 9 in

  pgmap v8473: 1500 pgs, 43 pools, 374 MB data, 2852 objects

1193 MB used, 133 GB / 134 GB avail

1500 active+clean

[root@us-east-1 ceph]#


*Hope these packages are correct*


[root@us-east-1 ceph]# rpm -qa | egrep -i ceph|radosgw

libcephfs1-0.87.1-0.el7.centos.x86_64

ceph-common-0.87.1-0.el7.centos.x86_64

python-ceph-0.87.1-0.el7.centos.x86_64

ceph-radosgw-0.87.1-0.el7.centos.x86_64

ceph-release-1-0.el7.noarch

ceph-0.87.1-0.el7.centos.x86_64

radosgw-agent-1.2.1-0.el7.centos.noarch

[root@us-east-1 ceph]#



Regards

Vickram
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] test pls ignore

[Vickey Singh]

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph Radosgw multi zone data replication failure

[Vickey Singh]

Hello Geeks


I am trying to setup Ceph Radosgw multi site data replication using
official documentation
http://ceph.com/docs/master/radosgw/federated-config/#multi-site-data-replication


Everything seems to work except radosgw-agent sync , Request you to please
check the below outputs and help me in any possible way.


*Environment : *


CentOS 7.0.1406

Ceph Versino 0.87.1

Rados Gateway configured using Civetweb



*Radosgw zone list : Works nicely *


[root@us-east-1 ceph]# radosgw-admin zone list --name
client.radosgw.us-east-1

{ zones: [

us-west,

us-east]}

[root@us-east-1 ceph]#


*Curl request to master zone : Works nicely *


[root@us-east-1 ceph]# curl http://us-east-1.crosslogic.com:7480

?xml version=1.0 encoding=UTF-8?ListAllMyBucketsResult xmlns=
http://s3.amazonaws.com/doc/2006-03-01/
OwnerIDanonymous/IDDisplayName/DisplayName/OwnerBuckets/Buckets/ListAllMyBucketsResult

[root@us-east-1 ceph]#


*Curl request to secondary zone : Works nicely *


[root@us-east-1 ceph]# curl http://us-west-1.crosslogic.com:7480

?xml version=1.0 encoding=UTF-8?ListAllMyBucketsResult xmlns=
http://s3.amazonaws.com/doc/2006-03-01/
OwnerIDanonymous/IDDisplayName/DisplayName/OwnerBuckets/Buckets/ListAllMyBucketsResult

[root@us-east-1 ceph]#


*Rados Gateway agent configuration file : Seems correct, no TYPO errors*


[root@us-east-1 ceph]# cat cluster-data-sync.conf

src_access_key: M7QAKDH8CYGTK86CG93U

src_secret_key: 0xQR6PINk23W\/GYrWJ14aF+1stG56M6xMkqkdloO

destination: http://us-west-1.crosslogic.com:7480

dest_access_key: ZQ32ES1WAWPG05YMZ7T7

dest_secret_key: INvk8AkrZRsejLEL34yRpMLmOqydt8ncOXy4RHCM

log_file: /var/log/radosgw/radosgw-sync-us-east-west.log

[root@us-east-1 ceph]#


*Rados Gateway agent SYNC : Fails , however it can fetch region map so i
think src and dest KEYS are correct. But don't know why it fails on
AttributeError *



*[root@us-east-1 ceph]# radosgw-agent -c cluster-data-sync.conf*

*region map is: {u'us': [u'us-west', u'us-east']}*

*Traceback (most recent call last):*

*  File /usr/bin/radosgw-agent, line 21, in module*

*sys.exit(main())*

*  File /usr/lib/python2.7/site-packages/radosgw_agent/cli.py, line 275,
in main*

*except client.ClientException as e:*

*AttributeError: 'module' object has no attribute 'ClientException'*

*[root@us-east-1 ceph]#*


*Can query to Ceph cluster using us-east-1 ID*


[root@us-east-1 ceph]# ceph -s --name client.radosgw.us-east-1

cluster 9609b429-eee2-4e23-af31-28a24fcf5cbc

 health HEALTH_OK

 monmap e3: 3 mons at {ceph-node1=
192.168.1.101:6789/0,ceph-node2=192.168.1.102:6789/0,ceph-node3=192.168.1.103:6789/0},
election epoch 448, quorum 0,1,2 ceph-node1,ceph-node2,ceph-node3

 osdmap e1063: 9 osds: 9 up, 9 in

  pgmap v8473: 1500 pgs, 43 pools, 374 MB data, 2852 objects

1193 MB used, 133 GB / 134 GB avail

1500 active+clean

[root@us-east-1 ceph]#


*Can query to Ceph cluster using us-west-1 ID*


[root@us-east-1 ceph]# ceph -s --name client.radosgw.us-west-1

cluster 9609b429-eee2-4e23-af31-28a24fcf5cbc

 health HEALTH_OK

 monmap e3: 3 mons at {ceph-node1=
192.168.1.101:6789/0,ceph-node2=192.168.1.102:6789/0,ceph-node3=192.168.1.103:6789/0},
election epoch 448, quorum 0,1,2 ceph-node1,ceph-node2,ceph-node3

 osdmap e1063: 9 osds: 9 up, 9 in

  pgmap v8473: 1500 pgs, 43 pools, 374 MB data, 2852 objects

1193 MB used, 133 GB / 134 GB avail

1500 active+clean

[root@us-east-1 ceph]#


*Hope these packages are correct*


[root@us-east-1 ceph]# rpm -qa | egrep -i ceph|radosgw

libcephfs1-0.87.1-0.el7.centos.x86_64

ceph-common-0.87.1-0.el7.centos.x86_64

python-ceph-0.87.1-0.el7.centos.x86_64

ceph-radosgw-0.87.1-0.el7.centos.x86_64

ceph-release-1-0.el7.noarch

ceph-0.87.1-0.el7.centos.x86_64

radosgw-agent-1.2.1-0.el7.centos.noarch

[root@us-east-1 ceph]#



Regards

VS
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph Hammer : Ceph-deploy 1.5.23-0 : RGW civetweb :: Not getting installed

[Vickey Singh]

Hello Cephers

I am trying to setup RGW using Ceph-deploy which is described here

http://docs.ceph.com/docs/master/start/quick-ceph-deploy/#add-an-rgw-instance


But unfortunately it doesn't seems to be working

Is there something i am missing  or you know some fix for this.




[root@ceph-node1 yum.repos.d]# ceph -v

*ceph version 0.94* (e61c4f093f88e44961d157f65091733580cea79a)

[root@ceph-node1 yum.repos.d]#



# yum update ceph-deploy


 SKIPPED 



  Verifying  : ceph-deploy-1.5.22-0.noarch
2/2


Updated:

 * ceph-deploy.noarch 0:1.5.23-0*


Complete!

[root@ceph-node1 ceph]#





[root@ceph-node1 ceph]# ceph-deploy rgw create rgw-node1

[ceph_deploy.conf][DEBUG ] found configuration file at:
/root/.cephdeploy.conf

[ceph_deploy.cli][INFO  ] Invoked (1.5.23): /usr/bin/ceph-deploy rgw create
rgw-node1

[ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts
rgw-node1:rgw.rgw-node1

*[ceph_deploy][ERROR ] RuntimeError: bootstrap-rgw keyring not found; run
'gatherkeys'*




[root@ceph-node1 ceph]# ceph-deploy --overwrite-conf mon create-initial

[ceph_deploy.conf][DEBUG ] found configuration file at:
/root/.cephdeploy.conf

[ceph_deploy.cli][INFO  ] Invoked (1.5.23): /usr/bin/ceph-deploy
--overwrite-conf mon create-initial

 SKIPPED 

[ceph_deploy.mon][INFO  ] mon.ceph-node1 monitor has reached quorum!

[ceph_deploy.mon][INFO  ] all initial monitors are running and have formed
quorum

[ceph_deploy.mon][INFO  ] Running gatherkeys...

[ceph_deploy.gatherkeys][DEBUG ] Have ceph.client.admin.keyring

[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring

[ceph_deploy.gatherkeys][DEBUG ] Have ceph.bootstrap-osd.keyring

[ceph_deploy.gatherkeys][DEBUG ] Have ceph.bootstrap-mds.keyring

[ceph_deploy.gatherkeys][DEBUG ] Checking ceph-node1 for
/var/lib/ceph/bootstrap-rgw/ceph.keyring

[ceph-node1][DEBUG ] connected to host: ceph-node1

[ceph-node1][DEBUG ] detect platform information from remote host

[ceph-node1][DEBUG ] detect machine type

[ceph-node1][DEBUG ] fetch remote file

*[ceph_deploy.gatherkeys][WARNIN] Unable to find
/var/lib/ceph/bootstrap-rgw/ceph.keyring on ceph-node1*

*[ceph_deploy.gatherkeys][WARNIN] No RGW bootstrap key found. Will not be
able to deploy RGW daemons*

[root@ceph-node1 ceph]#



[root@ceph-node1 ceph]# ceph-deploy gatherkeys ceph-node1

[ceph_deploy.conf][DEBUG ] found configuration file at:
/root/.cephdeploy.conf

[ceph_deploy.cli][INFO  ] Invoked (1.5.23): /usr/bin/ceph-deploy gatherkeys
ceph-node1

[ceph_deploy.gatherkeys][DEBUG ] Have ceph.client.admin.keyring

[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring

[ceph_deploy.gatherkeys][DEBUG ] Have ceph.bootstrap-osd.keyring

[ceph_deploy.gatherkeys][DEBUG ] Have ceph.bootstrap-mds.keyring

[ceph_deploy.gatherkeys][DEBUG ] Checking ceph-node1 for
/var/lib/ceph/bootstrap-rgw/ceph.keyring

[ceph-node1][DEBUG ] connected to host: ceph-node1

[ceph-node1][DEBUG ] detect platform information from remote host

[ceph-node1][DEBUG ] detect machine type

[ceph-node1][DEBUG ] fetch remote file

*[ceph_deploy.gatherkeys][WARNIN] Unable to find
/var/lib/ceph/bootstrap-rgw/ceph.keyring on ceph-node1*

*[ceph_deploy.gatherkeys][WARNIN] No RGW bootstrap key found. Will not be
able to deploy RGW daemons*

[root@ceph-node1 ceph]#



Regards

VS
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Vickey Singh]

Thanks for the help guys , here is my feedback with the tests


@Michael Kidd :yum install ceph ceph-common --disablerepo=base
--disablerepo=epel

Did not worked  here are the logs http://fpaste.org/208828/56448714/


@Travis Rhoden : Yep *exclude=python-rados python-rbd*  under epel.repo did
the trick and i can install Firefly / Giant without errors. Thanks


Any idea when this be fixed once for all ( so i no longer to patch
epel.repo to exclude python-r*)


- VS -

On Thu, Apr 9, 2015 at 4:26 AM, Michael Kidd linuxk...@redhat.com wrote:

 I don't think this came through the first time.. resending.. If it's a
 dupe, my apologies..

 For Firefly / Giant installs, I've had success with the following:

 yum install ceph ceph-common --disablerepo=base --disablerepo=epel

 Let us know if this works for you as well.

 Thanks,

 Michael J. Kidd
 Sr. Storage Consultant
 Inktank Professional Services
  - by Red Hat

 On Wed, Apr 8, 2015 at 9:07 PM, Michael Kidd linuxk...@redhat.com wrote:

 For Firefly / Giant installs, I've had success with the following:

 yum install ceph ceph-common --disablerepo=base --disablerepo=epel

 Let us know if this works for you as well.

 Thanks,

 Michael J. Kidd
 Sr. Storage Consultant
 Inktank Professional Services
  - by Red Hat

 On Wed, Apr 8, 2015 at 8:55 PM, Travis Rhoden trho...@gmail.com wrote:

 I did also confirm that, as Ken mentioned, this is not a problem on
 Hammer since Hammer includes the package split (python-ceph became
 python-rados and python-rbd).

  - Travis

 On Wed, Apr 8, 2015 at 5:00 PM, Travis Rhoden trho...@gmail.com wrote:

 Hi Vickey,

 The easiest way I know of to get around this right now is to add the
 following line in section for epel in /etc/yum.repos.d/epel.repo

 exclude=python-rados python-rbd

 So this is what my epel.repo file looks like: http://fpaste.org/208681/

 It is those two packages in EPEL that are causing problems.  I also
 tried enabling epel-testing, but that didn't work either.

 Unfortunately you would need to add this line on each node where Ceph
 Giant is being installed.

  - Travis

 On Wed, Apr 8, 2015 at 4:11 PM, Vickey Singh 
 vickey.singh22...@gmail.com wrote:

 Community , need help.


 -VS-

 On Wed, Apr 8, 2015 at 4:36 PM, Vickey Singh 
 vickey.singh22...@gmail.com wrote:

 Any suggestion  geeks


 VS

 On Wed, Apr 8, 2015 at 2:15 PM, Vickey Singh 
 vickey.singh22...@gmail.com wrote:


 Hi


 The below suggestion also didn’t worked


 Full logs here : http://paste.ubuntu.com/10771939/




 [root@rgw-node1 yum.repos.d]# yum --showduplicates list ceph

 Loaded plugins: fastestmirror, priorities

 Loading mirror speeds from cached hostfile

  * base: mirror.zetup.net

  * epel: ftp.fi.muni.cz

  * extras: mirror.zetup.net

  * updates: mirror.zetup.net

 25 packages excluded due to repository priority protections

 Available Packages

 ceph.x86_64
 0.80.6-0.el7.centos
 Ceph

 ceph.x86_64
 0.80.7-0.el7.centos
 Ceph

 ceph.x86_64
 0.80.8-0.el7.centos
 Ceph

 ceph.x86_64
 0.80.9-0.el7.centos
 Ceph

 [root@rgw-node1 yum.repos.d]#





 Its not able to install latest available package , yum is getting
 confused with other DOT releases.


 Any other suggestion to fix this ???



 -- Processing Dependency: libboost_system-mt.so.1.53.0()(64bit) for
 package: librbd1-0.80.9-0.el7.centos.x86_64

 -- Processing Dependency: libboost_thread-mt.so.1.53.0()(64bit) for
 package: librbd1-0.80.9-0.el7.centos.x86_64

 -- Finished Dependency Resolution

 Error: Package: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libaio.so.1(LIBAIO_0.4)(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_thread-mt.so.1.53.0()(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: librados2 = 0.80.7-0.el7.centos

Available: librados2-0.80.6-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.6-0.el7.centos

Available: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.7-0.el7.centos

Available: librados2-0.80.8-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.8-0.el7.centos

Installing: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.9-0.el7.centos

 Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_thread-mt.so.1.53.0()(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: python-requests

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: librbd1 = 0.80.7-0.el7.centos

Available: librbd1-0.80.6-0.el7.centos.x86_64 (Ceph)

librbd1 = 0.80.6-0

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Vickey Singh]

: fedora.uib.no

 * extras: ftp.funet.fi

 * updates: ftp.funet.fi

38 packages excluded due to repository priority protections

No package ceph-0.80.7-0.5.el7 available.

Error: Nothing to do

[root@rgw-node1 yum.repos.d]#






[root@rgw-node1 yum.repos.d]# cat epel.repo

[epel]

name=Extra Packages for Enterprise Linux 7 - $basearch

baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch

mirrorlist=
https://mirrors.fedoraproject.org/metalink?repo=epel-7arch=$basearch

failovermethod=priority

enabled=1

gpgcheck=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7


[epel-debuginfo]

name=Extra Packages for Enterprise Linux 7 - $basearch - Debug

baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch/debug

mirrorlist=
https://mirrors.fedoraproject.org/metalink?repo=epel-debug-7arch=$basearch

failovermethod=priority

enabled=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

gpgcheck=1


[epel-source]

name=Extra Packages for Enterprise Linux 7 - $basearch - Source

baseurl=http://download.fedoraproject.org/pub/epel/7/SRPMS

mirrorlist=
https://mirrors.fedoraproject.org/metalink?repo=epel-source-7arch=$basearch

failovermethod=priority

enabled=1

gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

gpgcheck=1

[root@rgw-node1 yum.repos.d]#


Regards

VS



On Wed, Apr 8, 2015 at 4:16 AM, Ken Dreyer kdre...@redhat.com wrote:

 Hi Vickey,

 Sorry about the issues you've been seeing. This looks very similar to
 http://tracker.ceph.com/issues/11104 .

 Here are two options you can try in order to work around this:

 - If you must run Firefly (0.80.x) or Giant (0.87.x), please try
 enabling the epel-testing repository on your system prior to
 installing Ceph. There is an update in epel-testing that should help
 with this issue.

 https://admin.fedoraproject.org/updates/FEDORA-EPEL-2015-1607/ceph-0.80.7-0.5.el7

 - If you can run Hammer (0.94), please try testing that out. The Hammer
 release's packages have been split up to match the split that happened
 in EPEL.

 - Ken

 On 04/07/2015 04:09 PM, Vickey Singh wrote:
  Hello There
 
  I am trying to install Giant on CentOS7 using ceph-deploy and
  encountered below problem.
 
  [rgw-node1][DEBUG ] Package python-ceph is obsoleted by python-rados,
  but obsoleting package does not provide for requirements
  [rgw-node1][DEBUG ] --- Package cups-libs.x86_64 1:1.6.3-17.el7 will be
  installed
  [rgw-node1][DEBUG ] -- Finished Dependency Resolution
  [rgw-node1][DEBUG ]  You could try using --skip-broken to work around
  the problem
  [rgw-node1][WARNIN] Error: Package:
  1:ceph-common-0.87.1-0.el7.centos.x86_64 (Ceph)
  [rgw-node1][WARNIN]Requires: python-ceph =
 1:0.87.1-0.el7.centos
  [rgw-node1][WARNIN]Available:
  1:python-ceph-0.86-0.el7.centos.x86_64 (Ceph)
  [rgw-node1][WARNIN]python-ceph = 1:0.86-0.el7.centos
  [rgw-node1][WARNIN]Available:
  1:python-ceph-0.87-0.el7.centos.x86_64 (Ceph)
  [rgw-node1][WARNIN]python-ceph = 1:0.87-0.el7.centos
  [rgw-node1][WARNIN]Available:
  1:python-ceph-0.87.1-0.el7.centos.x86_64 (Ceph)
  [rgw-node1][WARNIN]python-ceph = 1:0.87.1-0.el7.centos
  [rgw-node1][WARNIN] Error: Package: 1:ceph-0.87.1-0.el7.centos.x86_64
 (Ceph)
  [rgw-node1][WARNIN]Requires: python-ceph =
 1:0.87.1-0.el7.centos
  [rgw-node1][WARNIN]Available:
  1:python-ceph-0.86-0.el7.centos.x86_64 (Ceph)
  [rgw-node1][WARNIN]python-ceph = 1:0.86-0.el7.centos
  [rgw-node1][WARNIN]Available:
  1:python-ceph-0.87-0.el7.centos.x86_64 (Ceph)
  [rgw-node1][WARNIN]python-ceph = 1:0.87-0.el7.centos
  [rgw-node1][WARNIN]Available:
  1:python-ceph-0.87.1-0.el7.centos.x86_64 (Ceph)
  [rgw-node1][WARNIN]python-ceph = 1:0.87.1-0.el7.centos
  [rgw-node1][DEBUG ]  You could try running: rpm -Va --nofiles --nodigest
  [rgw-node1][ERROR ] RuntimeError: command returned non-zero exit status:
 1
  [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: yum -y
  install ceph
 
  [root@ceph-node1 ceph]#
  [root@ceph-node1 ceph]#
  [root@ceph-node1 ceph]#
  [root@ceph-node1 ceph]# ceph-deploy --version
  1.5.22
  [root@ceph-node1 ceph]#
  [root@ceph-node1 ceph]# ceph -v
  ceph version 0.87.1 (283c2e7cfa2457799f534744d7d549f83ea1335e)
  [root@ceph-node1 ceph]#
 
 
  On rgw-node1 macine
 
  /etc/yum.repos.d/ceph.repo seems to be correct
 
  [root@rgw-node1 yum.repos.d]# cat ceph.repo
  [Ceph]
  name=Ceph packages for $basearch
  baseurl=http://ceph.com/rpm-giant/el7/$basearch
  enabled=1
  gpgcheck=1
  type=rpm-md
  gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
  priority=1
 
  [Ceph-noarch]
  name=Ceph noarch packages
  baseurl=http://ceph.com/rpm-giant/el7/noarch
  enabled=1
  gpgcheck=1
  type=rpm-md
  gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
  priority=1
 
  [ceph-source]
  name=Ceph source packages
  baseurl

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Vickey Singh]

)

   Requires: libboost_thread-mt.so.1.53.0()(64bit)

Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: libboost_thread-mt.so.1.53.0()(64bit)

Error: Package: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: libboost_system-mt.so.1.53.0()(64bit)

Error: Package: librbd1-0.80.9-0.el7.centos.x86_64 (Ceph)

   Requires: libboost_thread-mt.so.1.53.0()(64bit)

Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

   Requires: librbd1 = 0.80.7-0.el7.centos

   Available: librbd1-0.80.6-0.el7.centos.x86_64 (Ceph)

   librbd1 = 0.80.6-0.el7.centos

   Available: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)

   librbd1 = 0.80.7-0.el7.centos

   Available: librbd1-0.80.8-0.el7.centos.x86_64 (Ceph)

   librbd1 = 0.80.8-0.el7.centos

   Installing: librbd1-0.80.9-0.el7.centos.x86_64 (Ceph)

   librbd1 = 0.80.9-0.el7.centos

Error: Package: librbd1-0.80.9-0.el7.centos.x86_64 (Ceph)

   Requires: libboost_system-mt.so.1.53.0()(64bit)

 You could try using --skip-broken to work around the problem

 You could try running: rpm -Va --nofiles --nodigest

[root@rgw-node1 yum.repos.d]#




Regards

VS


On Wed, Apr 8, 2015 at 12:58 PM, Irek Fasikhov malm...@gmail.com wrote:

 I use Centos 7.1. The problem is that in the basic package repository has
 ceph-common.

 [root@ceph01p24 cluster]# yum --showduplicates list ceph-common
 Loaded plugins: dellsysid, etckeeper, fastestmirror, priorities
 Loading mirror speeds from cached hostfile
  * base: centos-mirror.rbc.ru
  * epel: be.mirror.eurid.eu
  * extras: ftp.funet.fi
  * updates: centos-mirror.rbc.ru
 Installed Packages
 ceph-common.x86_64

  0.80.7-0.el7.centos
 @Ceph
 Available Packages
 ceph-common.x86_64

  0.80.6-0.el7.centos
 Ceph
 ceph-common.x86_64

  0.80.7-0.el7.centos
 Ceph
 ceph-common.x86_64

  0.80.8-0.el7.centos
 Ceph
 ceph-common.x86_64

  0.80.9-0.el7.centos
 Ceph
 ceph-common.x86_64
  1:0.80.7-0.4.el7

epel
 ceph-common.x86_64
  1:0.80.7-2.el7

base

 I make the installation as follows:

 rpm -ivh
 http://ceph.com/rpm-firefly/el7/noarch/ceph-release-1-0.el7.noarch.rpm
 yum install redhat-lsb-core-4.1-27.el7.centos.1.x86_64
 gperftools-libs.x86_64 yum-plugin-priorities.noarch ntp -y
 yum install librbd1-0.80.7-0.el7.centos
 librados2-0.80.7-0.el7.centos.x86_64.rpm -y
 yum install gdisk cryptsetup leveldb python-jinja2 hdparm -y

 yum install --disablerepo=base --disablerepo=epel
 ceph-common-0.80.7-0.el7.centos.x86_64 -y
 yum install --disablerepo=base --disablerepo=epel ceph-0.80.7-0.el7.centos
 -y

 2015-04-08 12:40 GMT+03:00 Vickey Singh vickey.singh22...@gmail.com:

 Hello Everyone


 I also tried setting higher priority as suggested by SAM but no luck


 Please see the Full logs here http://paste.ubuntu.com/10771358/


 While installing yum searches for correct Ceph repository but it founds 3
 versions of python-ceph under http://ceph.com/rpm-giant/el7/x86_64/


 How can i instruct yum to install latest version of ceph from giant
 repository ?? FYI i have this setting already


 [root@rgw-node1 yum.repos.d]# cat /etc/yum/pluginconf.d/priorities.conf

 [main]

 enabled = 1

 check_obsoletes = 1

 [root@rgw-node1 yum.repos.d]#




 This issue can be easily reproduced, just now i tried on a fresh server
 centos 7.0.1406 but it still fails.

 Please help.

 Please help.

 Please help.


 # cat /etc/redhat-release

 CentOS Linux release 7.0.1406 (Core)

 #

 # uname -r

 3.10.0-123.20.1.el7.x86_64

 #


 Regards

 VS


 On Wed, Apr 8, 2015 at 11:10 AM, Sam Wouters s...@ericom.be wrote:

  Hi Vickey,

 we had a similar issue and we resolved it by giving the centos base and
 update repo a higher priority (ex 10) then the epel repo.
 The ceph-deploy tool only sets a prio of 1 for the ceph repo's, but the
 centos and epel repo's stay on the default of 99.

 regards,
 Sam

 On 08-04-15 09:32, Vickey Singh wrote:

  Hi Ken


  As per your suggestion , i tried enabling epel-testing repository but
 still no luck.


  Please check the below output. I would really appreciate  any help
 here.



  # yum install ceph --enablerepo=epel-testing


  --- Package python-rbd.x86_64 1:0.80.7-0.5.el7 will be installed

 -- Processing Dependency: librbd1 = 1:0.80.7 for package:
 1:python-rbd-0.80.7-0.5.el7.x86_64

 -- Finished Dependency Resolution

 Error: Package: 1:python-cephfs-0.80.7-0.4.el7.x86_64 (epel)

Requires: libcephfs1 = 1

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Vickey Singh]

Hello Everyone


I also tried setting higher priority as suggested by SAM but no luck


Please see the Full logs here http://paste.ubuntu.com/10771358/


While installing yum searches for correct Ceph repository but it founds 3
versions of python-ceph under http://ceph.com/rpm-giant/el7/x86_64/


How can i instruct yum to install latest version of ceph from giant
repository ?? FYI i have this setting already


[root@rgw-node1 yum.repos.d]# cat /etc/yum/pluginconf.d/priorities.conf

[main]

enabled = 1

check_obsoletes = 1

[root@rgw-node1 yum.repos.d]#




This issue can be easily reproduced, just now i tried on a fresh server
centos 7.0.1406 but it still fails.

Please help.

Please help.

Please help.


# cat /etc/redhat-release

CentOS Linux release 7.0.1406 (Core)

#

# uname -r

3.10.0-123.20.1.el7.x86_64

#


Regards

VS


On Wed, Apr 8, 2015 at 11:10 AM, Sam Wouters s...@ericom.be wrote:

  Hi Vickey,

 we had a similar issue and we resolved it by giving the centos base and
 update repo a higher priority (ex 10) then the epel repo.
 The ceph-deploy tool only sets a prio of 1 for the ceph repo's, but the
 centos and epel repo's stay on the default of 99.

 regards,
 Sam

 On 08-04-15 09:32, Vickey Singh wrote:

  Hi Ken


  As per your suggestion , i tried enabling epel-testing repository but
 still no luck.


  Please check the below output. I would really appreciate  any help here.




  # yum install ceph --enablerepo=epel-testing


  --- Package python-rbd.x86_64 1:0.80.7-0.5.el7 will be installed

 -- Processing Dependency: librbd1 = 1:0.80.7 for package:
 1:python-rbd-0.80.7-0.5.el7.x86_64

 -- Finished Dependency Resolution

 Error: Package: 1:python-cephfs-0.80.7-0.4.el7.x86_64 (epel)

Requires: libcephfs1 = 1:0.80.7

Available: 1:libcephfs1-0.86-0.el7.centos.x86_64 (Ceph)

libcephfs1 = 1:0.86-0.el7.centos

Available: 1:libcephfs1-0.87-0.el7.centos.x86_64 (Ceph)

libcephfs1 = 1:0.87-0.el7.centos

Installing: 1:libcephfs1-0.87.1-0.el7.centos.x86_64 (Ceph)

libcephfs1 = 1:0.87.1-0.el7.centos

 *Error: Package: 1:python-rbd-0.80.7-0.5.el7.x86_64 (epel-testing)*

Requires: librbd1 = 1:0.80.7

Removing: librbd1-0.80.9-0.el7.centos.x86_64 (@Ceph)

librbd1 = 0.80.9-0.el7.centos

Updated By: 1:librbd1-0.87.1-0.el7.centos.x86_64 (Ceph)

librbd1 = 1:0.87.1-0.el7.centos

Available: 1:librbd1-0.86-0.el7.centos.x86_64 (Ceph)

librbd1 = 1:0.86-0.el7.centos

Available: 1:librbd1-0.87-0.el7.centos.x86_64 (Ceph)

librbd1 = 1:0.87-0.el7.centos

 *Error: Package: 1:python-rados-0.80.7-0.5.el7.x86_64 (epel-testing)*

Requires: librados2 = 1:0.80.7

Removing: librados2-0.80.9-0.el7.centos.x86_64 (@Ceph)

librados2 = 0.80.9-0.el7.centos

Updated By: 1:librados2-0.87.1-0.el7.centos.x86_64 (Ceph)

librados2 = 1:0.87.1-0.el7.centos

Available: 1:librados2-0.86-0.el7.centos.x86_64 (Ceph)

librados2 = 1:0.86-0.el7.centos

Available: 1:librados2-0.87-0.el7.centos.x86_64 (Ceph)

librados2 = 1:0.87-0.el7.centos

  You could try using --skip-broken to work around the problem

  You could try running: rpm -Va --nofiles --nodigest





  # yum install ceph --enablerepo=epel-testing --disablerepo=ceph*



  --- Package spax.x86_64 0:1.5.2-11.el7 will be installed

 --- Package time.x86_64 0:1.7-45.el7 will be installed

 -- Running transaction check

 --- Package cups-libs.x86_64 1:1.6.3-17.el7 will be installed

 --- Package python-cephfs.x86_64 1:0.80.7-0.4.el7 will be installed

 -- Processing Dependency: libcephfs1 = 1:0.80.7 for package:
 1:python-cephfs-0.80.7-0.4.el7.x86_64

 --- Package python-rados.x86_64 1:0.80.7-0.5.el7 will be installed

 -- Processing Dependency: librados2 = 1:0.80.7 for package:
 1:python-rados-0.80.7-0.5.el7.x86_64

 --- Package python-rbd.x86_64 1:0.80.7-0.5.el7 will be installed

 -- Processing Dependency: librbd1 = 1:0.80.7 for package:
 1:python-rbd-0.80.7-0.5.el7.x86_64

 -- Finished Dependency Resolution

 Error: Package: 1:python-cephfs-0.80.7-0.4.el7.x86_64 (epel)

Requires: libcephfs1 = 1:0.80.7

Available: 1:libcephfs1-0.86-0.el7.centos.x86_64 (Ceph)

libcephfs1 = 1:0.86-0.el7.centos

Available: 1:libcephfs1-0.87-0.el7.centos.x86_64 (Ceph)

libcephfs1 = 1:0.87-0.el7.centos

Installing: 1:libcephfs1-0.87.1-0.el7.centos.x86_64 (Ceph)

libcephfs1 = 1:0.87.1-0.el7.centos

 *Error: Package: 1:python-rbd-0.80.7-0.5.el7.x86_64 (epel-testing)*

Requires: librbd1 = 1:0.80.7

Removing: librbd1-0.80.9-0.el7.centos.x86_64 (@Ceph)

librbd1 = 0.80.9-0.el7.centos

Updated By: 1:librbd1-0.87.1-0.el7.centos.x86_64

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Vickey Singh]

Any suggestion  geeks


VS

On Wed, Apr 8, 2015 at 2:15 PM, Vickey Singh vickey.singh22...@gmail.com
wrote:


 Hi


 The below suggestion also didn’t worked


 Full logs here : http://paste.ubuntu.com/10771939/




 [root@rgw-node1 yum.repos.d]# yum --showduplicates list ceph

 Loaded plugins: fastestmirror, priorities

 Loading mirror speeds from cached hostfile

  * base: mirror.zetup.net

  * epel: ftp.fi.muni.cz

  * extras: mirror.zetup.net

  * updates: mirror.zetup.net

 25 packages excluded due to repository priority protections

 Available Packages

 ceph.x86_64
 0.80.6-0.el7.centos
 Ceph

 ceph.x86_64
 0.80.7-0.el7.centos
 Ceph

 ceph.x86_64
 0.80.8-0.el7.centos
 Ceph

 ceph.x86_64
 0.80.9-0.el7.centos
 Ceph

 [root@rgw-node1 yum.repos.d]#





 Its not able to install latest available package , yum is getting confused
 with other DOT releases.


 Any other suggestion to fix this ???



 -- Processing Dependency: libboost_system-mt.so.1.53.0()(64bit) for
 package: librbd1-0.80.9-0.el7.centos.x86_64

 -- Processing Dependency: libboost_thread-mt.so.1.53.0()(64bit) for
 package: librbd1-0.80.9-0.el7.centos.x86_64

 -- Finished Dependency Resolution

 Error: Package: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libaio.so.1(LIBAIO_0.4)(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_thread-mt.so.1.53.0()(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: librados2 = 0.80.7-0.el7.centos

Available: librados2-0.80.6-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.6-0.el7.centos

Available: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.7-0.el7.centos

Available: librados2-0.80.8-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.8-0.el7.centos

Installing: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.9-0.el7.centos

 Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_thread-mt.so.1.53.0()(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: python-requests

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: librbd1 = 0.80.7-0.el7.centos

Available: librbd1-0.80.6-0.el7.centos.x86_64 (Ceph)

librbd1 = 0.80.6-0.el7.centos

Available: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)

librbd1 = 0.80.7-0.el7.centos

Available: librbd1-0.80.8-0.el7.centos.x86_64 (Ceph)

librbd1 = 0.80.8-0.el7.centos

Installing: librbd1-0.80.9-0.el7.centos.x86_64 (Ceph)

librbd1 = 0.80.9-0.el7.centos

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: python-ceph = 0.80.7-0.el7.centos

Available: python-ceph-0.80.6-0.el7.centos.x86_64 (Ceph)

python-ceph = 0.80.6-0.el7.centos

Available: python-ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

python-ceph = 0.80.7-0.el7.centos

Available: python-ceph-0.80.8-0.el7.centos.x86_64 (Ceph)

python-ceph = 0.80.8-0.el7.centos

Installing: python-ceph-0.80.9-0.el7.centos.x86_64 (Ceph)

python-ceph = 0.80.9-0.el7.centos

 Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: python-requests

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: librados2 = 0.80.7-0.el7.centos

Available: librados2-0.80.6-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.6-0.el7.centos

Available: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.7-0.el7.centos

Available: librados2-0.80.8-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.8-0.el7.centos

Installing: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.9-0.el7.centos

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libaio.so.1(LIBAIO_0.1)(64bit)

 Error: Package: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)

Requires: libboost_thread-mt.so.1.53.0()(64bit)

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph

Re: [ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Vickey Singh]

Community , need help.


-VS-

On Wed, Apr 8, 2015 at 4:36 PM, Vickey Singh vickey.singh22...@gmail.com
wrote:

 Any suggestion  geeks


 VS

 On Wed, Apr 8, 2015 at 2:15 PM, Vickey Singh vickey.singh22...@gmail.com
 wrote:


 Hi


 The below suggestion also didn’t worked


 Full logs here : http://paste.ubuntu.com/10771939/




 [root@rgw-node1 yum.repos.d]# yum --showduplicates list ceph

 Loaded plugins: fastestmirror, priorities

 Loading mirror speeds from cached hostfile

  * base: mirror.zetup.net

  * epel: ftp.fi.muni.cz

  * extras: mirror.zetup.net

  * updates: mirror.zetup.net

 25 packages excluded due to repository priority protections

 Available Packages

 ceph.x86_64
 0.80.6-0.el7.centos
 Ceph

 ceph.x86_64
 0.80.7-0.el7.centos
 Ceph

 ceph.x86_64
 0.80.8-0.el7.centos
 Ceph

 ceph.x86_64
 0.80.9-0.el7.centos
 Ceph

 [root@rgw-node1 yum.repos.d]#





 Its not able to install latest available package , yum is getting
 confused with other DOT releases.


 Any other suggestion to fix this ???



 -- Processing Dependency: libboost_system-mt.so.1.53.0()(64bit) for
 package: librbd1-0.80.9-0.el7.centos.x86_64

 -- Processing Dependency: libboost_thread-mt.so.1.53.0()(64bit) for
 package: librbd1-0.80.9-0.el7.centos.x86_64

 -- Finished Dependency Resolution

 Error: Package: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libaio.so.1(LIBAIO_0.4)(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_thread-mt.so.1.53.0()(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: librados2 = 0.80.7-0.el7.centos

Available: librados2-0.80.6-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.6-0.el7.centos

Available: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.7-0.el7.centos

Available: librados2-0.80.8-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.8-0.el7.centos

Installing: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.9-0.el7.centos

 Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_thread-mt.so.1.53.0()(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: python-requests

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: librbd1 = 0.80.7-0.el7.centos

Available: librbd1-0.80.6-0.el7.centos.x86_64 (Ceph)

librbd1 = 0.80.6-0.el7.centos

Available: librbd1-0.80.7-0.el7.centos.x86_64 (Ceph)

librbd1 = 0.80.7-0.el7.centos

Available: librbd1-0.80.8-0.el7.centos.x86_64 (Ceph)

librbd1 = 0.80.8-0.el7.centos

Installing: librbd1-0.80.9-0.el7.centos.x86_64 (Ceph)

librbd1 = 0.80.9-0.el7.centos

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: ceph-common-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: python-ceph = 0.80.7-0.el7.centos

Available: python-ceph-0.80.6-0.el7.centos.x86_64 (Ceph)

python-ceph = 0.80.6-0.el7.centos

Available: python-ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

python-ceph = 0.80.7-0.el7.centos

Available: python-ceph-0.80.8-0.el7.centos.x86_64 (Ceph)

python-ceph = 0.80.8-0.el7.centos

Installing: python-ceph-0.80.9-0.el7.centos.x86_64 (Ceph)

python-ceph = 0.80.9-0.el7.centos

 Error: Package: libcephfs1-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: python-requests

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: librados2 = 0.80.7-0.el7.centos

Available: librados2-0.80.6-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.6-0.el7.centos

Available: librados2-0.80.7-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.7-0.el7.centos

Available: librados2-0.80.8-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.8-0.el7.centos

Installing: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)

librados2 = 0.80.9-0.el7.centos

 Error: Package: ceph-0.80.7-0.el7.centos.x86_64 (Ceph)

Requires: libaio.so.1(LIBAIO_0.1)(64bit)

 Error: Package: librados2-0.80.9-0.el7.centos.x86_64 (Ceph)

Requires: libboost_system-mt.so.1.53.0()(64bit)

 Error: Package: librados2-0.80.9-0.el7.centos.x86_64 (Ceph

[ceph-users] Firefly - Giant : CentOS 7 : install failed ceph-deploy

[Vickey Singh]

Hello There

I am trying to install Giant on CentOS7 using ceph-deploy and encountered
below problem.

[rgw-node1][DEBUG ] Package python-ceph is obsoleted by python-rados, but
obsoleting package does not provide for requirements
[rgw-node1][DEBUG ] --- Package cups-libs.x86_64 1:1.6.3-17.el7 will be
installed
[rgw-node1][DEBUG ] -- Finished Dependency Resolution
[rgw-node1][DEBUG ]  You could try using --skip-broken to work around the
problem
[rgw-node1][WARNIN] Error: Package:
1:ceph-common-0.87.1-0.el7.centos.x86_64 (Ceph)
[rgw-node1][WARNIN]Requires: python-ceph = 1:0.87.1-0.el7.centos
[rgw-node1][WARNIN]Available:
1:python-ceph-0.86-0.el7.centos.x86_64 (Ceph)
[rgw-node1][WARNIN]python-ceph = 1:0.86-0.el7.centos
[rgw-node1][WARNIN]Available:
1:python-ceph-0.87-0.el7.centos.x86_64 (Ceph)
[rgw-node1][WARNIN]python-ceph = 1:0.87-0.el7.centos
[rgw-node1][WARNIN]Available:
1:python-ceph-0.87.1-0.el7.centos.x86_64 (Ceph)
[rgw-node1][WARNIN]python-ceph = 1:0.87.1-0.el7.centos
[rgw-node1][WARNIN] Error: Package: 1:ceph-0.87.1-0.el7.centos.x86_64 (Ceph)
[rgw-node1][WARNIN]Requires: python-ceph = 1:0.87.1-0.el7.centos
[rgw-node1][WARNIN]Available:
1:python-ceph-0.86-0.el7.centos.x86_64 (Ceph)
[rgw-node1][WARNIN]python-ceph = 1:0.86-0.el7.centos
[rgw-node1][WARNIN]Available:
1:python-ceph-0.87-0.el7.centos.x86_64 (Ceph)
[rgw-node1][WARNIN]python-ceph = 1:0.87-0.el7.centos
[rgw-node1][WARNIN]Available:
1:python-ceph-0.87.1-0.el7.centos.x86_64 (Ceph)
[rgw-node1][WARNIN]python-ceph = 1:0.87.1-0.el7.centos
[rgw-node1][DEBUG ]  You could try running: rpm -Va --nofiles --nodigest
[rgw-node1][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: yum -y
install ceph

[root@ceph-node1 ceph]#
[root@ceph-node1 ceph]#
[root@ceph-node1 ceph]#
[root@ceph-node1 ceph]# ceph-deploy --version
1.5.22
[root@ceph-node1 ceph]#
[root@ceph-node1 ceph]# ceph -v
ceph version 0.87.1 (283c2e7cfa2457799f534744d7d549f83ea1335e)
[root@ceph-node1 ceph]#


On rgw-node1 macine

/etc/yum.repos.d/ceph.repo seems to be correct

[root@rgw-node1 yum.repos.d]# cat ceph.repo
[Ceph]
name=Ceph packages for $basearch
baseurl=http://ceph.com/rpm-giant/el7/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
priority=1

[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://ceph.com/rpm-giant/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
priority=1

[ceph-source]
name=Ceph source packages
baseurl=http://ceph.com/rpm-giant/el7/SRPMS
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc
priority=1


When i visit this directory http://ceph.com/rpm-giant/el7 , i can see
multiple versions of python-ceph i.e.
python-ceph-0.86-0.el7.centos.x86_64
python-ceph-0.87-0.el7.centos.x86_64
python-ceph-0.87-1.el7.centos.x86_64

*This is the reason , yum is getting confused to install the latest
available version python-ceph-0.87-1.el7.centos.x86_64. This issue looks
like yum priority plugin and RPM obsolete.*

http://tracker.ceph.com/issues/10476

[root@rgw-node1 yum.repos.d]# cat /etc/yum/pluginconf.d/priorities.conf
[main]
enabled = 1
check_obsoletes = 1

[root@rgw-node1 yum.repos.d]#

[root@rgw-node1 yum.repos.d]#
[root@rgw-node1 yum.repos.d]# uname -r
3.10.0-229.1.2.el7.x86_64
[root@rgw-node1 yum.repos.d]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
[root@rgw-node1 yum.repos.d]#


However it worked *fine 1 week back* on CentOS 7.0

[root@ceph-node1 ceph]# uname -r
3.10.0-123.20.1.el7.x86_64
[root@ceph-node1 ceph]# cat /etc/redhat-release
CentOS Linux release 7.0.1406 (Core)
[root@ceph-node1 ceph]#


Any fix to this is highly appreciated.

Regards
VS
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Introducing Learning Ceph : The First ever Book on Ceph

[Vickey Singh]

Amazing piece of work Karan , this was something which is missing since
long , thanks for filling the gap.

I got my book today and just finished reading couple of pages , excellent
introduction to Ceph.


Thanks again , its worth purchasing this book.


Best Regards
Vicky



On Fri, Feb 6, 2015 at 12:01 PM, pixelfairy pixelfa...@gmail.com wrote:

 congrats!

 page 17, xen is spelled with an X, not Z.

 On Fri, Feb 6, 2015 at 1:17 AM, Karan Singh karan.si...@csc.fi wrote:
  Hello Community Members
 
  I am happy to introduce the first book on Ceph with the title “Learning
  Ceph”.
 
  Me and many folks from the publishing house together with technical
  reviewers spent several months to get this book compiled and published.
 
  Finally the book is up for sale on , i hope you would like it and surely
  will learn a lot from it.
 
  Amazon :
 
 http://www.amazon.com/Learning-Ceph-Karan-Singh/dp/1783985623/ref=sr_1_1?s=booksie=UTF8qid=1423174441sr=1-1keywords=ceph
  Packtpub :
 https://www.packtpub.com/application-development/learning-ceph
 
  You can grab the sample copy from here :
  https://www.dropbox.com/s/ek76r01r9prs6pb/Learning_Ceph_Packt.pdf?dl=0
 
  Finally , I would like to express my sincere thanks to
 
  Sage Weil - For developing Ceph and everything around it as well as
 writing
  foreword for “Learning Ceph”.
  Patrick McGarry - For his usual off the track support that too always.
 
  Last but not the least , to our great community members , who are also
  reviewers of the book Don Talton , Julien Recurt , Sebastien Han and
 Zihong
  Chen , Thank you guys for your efforts.
 
 
  
  Karan Singh
  Systems Specialist , Storage Platforms
  CSC - IT Center for Science,
  Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
  mobile: +358 503 812758
  tel. +358 9 4572001
  fax +358 9 4572302
  http://www.csc.fi/
  
 
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD getting unmapped every time when server reboot

[Vickey Singh]

Hi Chris

yes i have checked this message and i am sure that secret file is present
in correct location. Any other suggestions are weelcome.

Hi Sebastien

Can you suggest something here.

Thanks
vicky


On Sun, Oct 26, 2014 at 1:23 AM, Christopher Armstrong ch...@opdemand.com
wrote:

 unable to read secretfile: No such file or directory

 Looks like it's trying to mount, but your secretfile is gone.


 *Chris Armstrong*Head of Services
 OpDemand / Deis.io

 GitHub: https://github.com/deis/deis -- Docs: http://docs.deis.io/


 On Sat, Oct 25, 2014 at 2:07 PM, Vickey Singh vickey.singh22...@gmail.com
  wrote:

 Hello Cephers , need your advice and tips here.

 *Problem statement : Ceph RBD getting unmapped each time i reboot my
 server . After reboot every time i need to manually map it and mount it.*

 *Setup : *

 Ceph Firefly 0.80.1
 CentOS 6.5  , Kernel : 3.15.0-1


 I have tried doing as mentioned in the blog , but looks like this does
 not works with CentOS

 http://ceph.com/planet/mapunmap-rbd-device-on-bootshutdown/



 # /etc/init.d/rbdmap start
 /etc/init.d/rbdmap: line 26: log_daemon_msg: command not found
 /etc/init.d/rbdmap: line 42: log_progress_msg: command not found
 /etc/init.d/rbdmap: line 47: echo: write error: Invalid argument
 /etc/init.d/rbdmap: line 52: log_end_msg: command not found
 /etc/init.d/rbdmap: line 56: log_action_begin_msg: command not found
 unable to read secretfile: No such file or directory
 error reading secret file
 failed to parse ceph_options
 Thread::try_create(): pthread_create failed with error
 13common/Thread.cc: In function 'void Thread::create(size_t)' thread
 7fb8ec4ed760 time 2014-10-26 00:01:10.180440
 common/Thread.cc: 110: FAILED assert(ret == 0)
  ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
  1: (Thread::create(unsigned long)+0x8a) [0x6ba82a]
  2: (CephContext::CephContext(unsigned int)+0xba) [0x60ef7a]
  3: (common_preinit(CephInitParameters const, code_environment_t,
 int)+0x45) [0x6e8305]
  4: (global_pre_init(std::vectorchar const*, std::allocatorchar const*
 *, std::vectorchar const*, std::allocatorchar const* , unsigned int,
 code_environment_t, int)+0xaf) [0x5ee21f]
  5: (global_init(std::vectorchar const*, std::allocatorchar const* *,
 std::vectorchar const*, std::allocatorchar const* , unsigned int,
 code_environment_t, int)+0x2f) [0x5eed6f]
  6: (main()+0x7f) [0x5289af]
  7: (__libc_start_main()+0xfd) [0x3efa41ed1d]
  8: ceph-fuse() [0x5287c9]
  NOTE: a copy of the executable, or `objdump -rdS executable` is needed
 to interpret this.
 terminate called after throwing an instance of 'ceph::FailedAssertion'
 /etc/init.d/rbdmap: line 58: log_action_end_msg: command not found
 #


 # cat /etc/ceph/rbdmap
 rbd/rbd-disk1 id=admin,secret=AQAinItT8Ip9AhAAS93FrXLrrnVp8/sQhjvTIg==
 #


 Many Thanks in Advance
 Vicky

 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RBD getting unmapped every time when server reboot

[Vickey Singh]

Hello Cephers , need your advice and tips here.

*Problem statement : Ceph RBD getting unmapped each time i reboot my server
. After reboot every time i need to manually map it and mount it.*

*Setup : *

Ceph Firefly 0.80.1
CentOS 6.5  , Kernel : 3.15.0-1


I have tried doing as mentioned in the blog , but looks like this does not
works with CentOS

http://ceph.com/planet/mapunmap-rbd-device-on-bootshutdown/



# /etc/init.d/rbdmap start
/etc/init.d/rbdmap: line 26: log_daemon_msg: command not found
/etc/init.d/rbdmap: line 42: log_progress_msg: command not found
/etc/init.d/rbdmap: line 47: echo: write error: Invalid argument
/etc/init.d/rbdmap: line 52: log_end_msg: command not found
/etc/init.d/rbdmap: line 56: log_action_begin_msg: command not found
unable to read secretfile: No such file or directory
error reading secret file
failed to parse ceph_options
Thread::try_create(): pthread_create failed with error 13common/Thread.cc:
In function 'void Thread::create(size_t)' thread 7fb8ec4ed760 time
2014-10-26 00:01:10.180440
common/Thread.cc: 110: FAILED assert(ret == 0)
 ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)
 1: (Thread::create(unsigned long)+0x8a) [0x6ba82a]
 2: (CephContext::CephContext(unsigned int)+0xba) [0x60ef7a]
 3: (common_preinit(CephInitParameters const, code_environment_t,
int)+0x45) [0x6e8305]
 4: (global_pre_init(std::vectorchar const*, std::allocatorchar const*
*, std::vectorchar const*, std::allocatorchar const* , unsigned int,
code_environment_t, int)+0xaf) [0x5ee21f]
 5: (global_init(std::vectorchar const*, std::allocatorchar const* *,
std::vectorchar const*, std::allocatorchar const* , unsigned int,
code_environment_t, int)+0x2f) [0x5eed6f]
 6: (main()+0x7f) [0x5289af]
 7: (__libc_start_main()+0xfd) [0x3efa41ed1d]
 8: ceph-fuse() [0x5287c9]
 NOTE: a copy of the executable, or `objdump -rdS executable` is needed
to interpret this.
terminate called after throwing an instance of 'ceph::FailedAssertion'
/etc/init.d/rbdmap: line 58: log_action_end_msg: command not found
#


# cat /etc/ceph/rbdmap
rbd/rbd-disk1 id=admin,secret=AQAinItT8Ip9AhAAS93FrXLrrnVp8/sQhjvTIg==
#


Many Thanks in Advance
Vicky
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph RGW + S3 Client (s3cmd)

[Vickey Singh]

*Hello Cephers*


*I have followed Ceph documentation and my radios gateway setup is working
fine.*


 # swift -V 1.0 -A  http://bmi-pocfe2.scc.fi/auth -U scc:swift -K secretkey
list

Hello-World

bmi-pocfe2

scc

packstack

test

 #



 # radosgw-admin bucket stats --bucket=scc

{ bucket: scc,

  pool: .rgw.buckets,

  index_pool: .rgw.buckets.index,

  id: default.11276.3,

  marker: default.11276.3,

  owner: scc,

  ver: 1,

  master_ver: 0,

  mtime: 1403091138,

  max_marker: ,

  usage: {}}

 #



*Next , i am trying to access my object storage using s3 client (s3cmd).
I have configured s3cmd with required keys and host information. ( do you
think any thing else needs to be modified in .s3cfg file)*



 # cat .s3cfg | egrep key|host_

access_key = SN04RLC4GVQBFIYNEJQG

host_base = bmi-pocfe2.scc.fi

host_bucket = %(bucket)s.bmi-pocfe2.scc.fi

secret_key = tCWCXrlGWAx\/kVjO\/Vqsqn1BflaLyAbzS7t7uM4N

 #


*But while accessing it , i am facing problems*


 # s3cmd ls

WARNING: Retrying failed request: / ([Errno -2] Name or service not known)

WARNING: Waiting 3 sec...

WARNING: Retrying failed request: / ([Errno -2] Name or service not known)

WARNING: Waiting 6 sec...

WARNING: Retrying failed request: / ([Errno -2] Name or service not known)

WARNING: Waiting 9 sec...




*Can you please suggest ,  what can be the problem here  which prevents me
using object storage via s3cmd.  (any pointers , wild guess  are welcome)*


*Also is DNS server mandatory for s3 ??  anyway i have DNS and its working
good.*



 # nslookup bmi-pocfe2.scc.fi

Server: 127.0.0.1

Address: 127.0.0.1#53


bmi-pocfe2.scc.fi canonical name = scc.fi.

Name: scc.fi

Address: 86.50.1.50


 # nslookup myobject.bmi-pocfe2.scc.fi

Server: 127.0.0.1

Address: 127.0.0.1#53


myobject.bmi-pocfe2.scc.fi canonical name = scc.fi.

Name: scc.fi

Address: 86.50.1.50


 # nslookup s3.bmi-pocfe2.scc.fi

Server: 127.0.0.1

Address: 127.0.0.1#53


s3.bmi-pocfe2.scc.fi canonical name = scc.fi.

Name: scc.fi

Address: 86.50.1.50




*Expecting support from Great Ceph Community.*




Regards

Vickram Singh
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RADOSGW + OpenStack basic question

[Vickey Singh]

Hello Cephers


I have been following ceph documentation to install and configure RGW and
fortunately everything went fine and RGW is correctly setup.

Next i would like to use RGW with OpenStack , and for this i have followed
http://ceph.com/docs/master/radosgw/keystone/   , as per the document i
have done all the steps.

But how should i test RGW and OpenStack integration , the document does not
show steps to verify the integration or how to use it further with
openstack.

Can you please point me to the right direction , for testing Ceph RGW and
OpenStack integration . Is there any work / blog on internet by
someone which can demonstrate the things “How to use Ceph RGW with
Openstack

Please help.


Regards
Vickram Singh
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com