Re: [ceph-users] ceph benchmark

[Karan Singh]

Agree with David

Its being cached , you can try
- oflag options for dd
- monitor system cache during dd


- Karan -

On Fri, Jun 17, 2016 at 1:58 AM, David  wrote:

> I'm probably misunderstanding the question but if you're getting 3GB/s
> from your dd, you're already caching. Can you provide some more detail on
> what you're trying to achieve.
> On 16 Jun 2016 21:53, "Patrick McGarry"  wrote:
>
>> Moving this over to ceph-user where it’ll get the eyeballs you need.
>>
>> On Mon, Jun 13, 2016 at 2:58 AM, Marcus Strasser
>>  wrote:
>> > Hello!
>> >
>> >
>> >
>> > I have a little test cluster with 2 server. Each Server have an osd
>> with 800
>> > GB, there is a 10 Gbps Link between the servers.
>> >
>> > On a ceph-client i have configured a cephfs, mount kernelspace. The
>> client
>> > is also connected with a 10 Gbps Link.
>> >
>> > All 3 use debian
>> >
>> > 4.5.5 kernel
>> >
>> > 64 GB mem
>> >
>> > There is no special configuration.
>> >
>> >
>> >
>> > Now the question:
>> >
>> > When i use the dd (~11GB) command in the cephfs mount, i get a result
>> of 3
>> > GB/s
>> >
>> >
>> >
>> > dd if=/dev/zero of=/cephtest/test bs=1M count=10240
>> >
>> >
>> >
>> > Is it possble to transfer the data faster (use full capacity oft he
>> network)
>> > and cache it with the memory?
>> >
>> >
>> >
>> > Thanks,
>> >
>> > Marcus Strasser
>> >
>> >
>> >
>> >
>> >
>> > Marcus Strasser
>> >
>> > Linux Systeme
>> >
>> > Russmedia IT GmbH
>> >
>> > A-6850 Schwarzach, Gutenbergstr. 1
>> >
>> >
>> >
>> > T +43 5572 501-872
>> >
>> > F +43 5572 501-97872
>> >
>> > marcus.stras...@highspeed.vol.at
>> >
>> > highspeed.vol.at
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>>
>> Best Regards,
>>
>> Patrick McGarry
>> Director Ceph Community || Red Hat
>> http://ceph.com  ||  http://community.redhat.com
>> @scuttlemonkey || @ceph
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd not removed from crush map after ceph osd crush remove

[Karan Singh]

Dimitar

Is it fixed ?

- is your cluster pool size is 2
- you can consider running ceph pg repair {pgid}  or ceph osd lost 4 ( this is 
a bit dangerous command )


Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


> On 22 Feb 2016, at 10:10, Dimitar Boichev <dimitar.boic...@axsmarine.com> 
> wrote:
> 
> Anyone ?
>  
> Regards.
>  
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
> Dimitar Boichev
> Sent: Thursday, February 18, 2016 5:06 PM
> To: ceph-users@lists.ceph.com
> Subject: [ceph-users] osd not removed from crush map after ceph osd crush 
> remove
>  
> Hello,
> I am running a tiny cluster of 2 nodes.
> ceph -v
> ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)
>  
> One osd died and I added a new osd (not replacing the old one).
> After that I wanted to remove the failed osd completely from the cluster.
> Here is what I did:
> ceph osd reweight osd.4 0.0
> ceph osd crush reweight osd.4 0.0
> ceph osd out osd.4
> ceph osd crush remove osd.4
> ceph auth del osd.4
> ceph osd rm osd.4
>  
>  
> But after the rebalancing I ended up with 155 PGs in stale+active+clean  
> state.
>  
> @storage1:/tmp# ceph -s
> cluster 7a9120b9-df42-4308-b7b1-e1f3d0f1e7b3
>  health HEALTH_WARN 155 pgs stale; 155 pgs stuck stale; 1 requests are 
> blocked > 32 sec; nodeep-scrub flag(s) set
>  monmap e1: 1 mons at {storage1=192.168.10.3:6789/0}, election epoch 1, 
> quorum 0 storage1
>  osdmap e1064: 6 osds: 6 up, 6 in
> flags nodeep-scrub
>   pgmap v26760322: 712 pgs, 8 pools, 532 GB data, 155 kobjects
> 1209 GB used, 14210 GB / 15419 GB avail
>  155 stale+active+clean
>  557 active+clean
>   client io 91925 B/s wr, 5 op/s
>  
> I know about the 1 monitor problem I just want to fix the cluster to healthy 
> state then I will add the third storage node and go up to 3 monitors.
>  
> The problem is as follows:
> @storage1:/tmp# ceph pg map 2.3a
> osdmap e1064 pg 2.3a (2.3a) -> up [6] acting [6]
> @storage1:/tmp# ceph pg 2.3a query
> Error ENOENT: i don't have pgid 2.3a
>  
>  
> @storage1:/tmp# ceph health detail
> HEALTH_WARN 155 pgs stale; 155 pgs stuck stale; 1 requests are blocked > 32 
> sec; 1 osds have slow requests; nodeep-scrub flag(s) set
> pg 7.2a is stuck stale for 8887559.656879, current state stale+active+clean, 
> last acting [4]
> pg 5.28 is stuck stale for 8887559.656886, current state stale+active+clean, 
> last acting [4]
> pg 7.2b is stuck stale for 8887559.656889, current state stale+active+clean, 
> last acting [4]
> pg 7.2c is stuck stale for 8887559.656892, current state stale+active+clean, 
> last acting [4]
> pg 0.2b is stuck stale for 8887559.656893, current state stale+active+clean, 
> last acting [4]
> pg 6.2c is stuck stale for 8887559.656894, current state stale+active+clean, 
> last acting [4]
> pg 6.2f is stuck stale for 8887559.656893, current state stale+active+clean, 
> last acting [4]
> pg 2.2b is stuck stale for 8887559.656896, current state stale+active+clean, 
> last acting [4]
> pg 2.25 is stuck stale for 8887559.656896, current state stale+active+clean, 
> last acting [4]
> pg 6.20 is stuck stale for 8887559.656898, current state stale+active+clean, 
> last acting [4]
> pg 5.21 is stuck stale for 8887559.656898, current state stale+active+clean, 
> last acting [4]
> pg 0.24 is stuck stale for 8887559.656904, current state stale+active+clean, 
> last acting [4]
> pg 2.21 is stuck stale for 8887559.656904, current state stale+active+clean, 
> last acting [4]
> pg 5.27 is stuck stale for 8887559.656906, current state stale+active+clean, 
> last acting [4]
> pg 2.23 is stuck stale for 8887559.656908, current state stale+active+clean, 
> last acting [4]
> pg 6.26 is stuck stale for 8887559.656909, current state stale+active+clean, 
> last acting [4]
> pg 7.27 is stuck stale for 8887559.656913, current state stale+active+clean, 
> last acting [4]
> pg 7.18 is stuck stale for 8887559.656914, current state stale+active+clean, 
> last acting [4]
> pg 0.1e is stuck stale for 8887559.656914, current state stale+active+clean, 
> last acting [4]
> pg 6.18 is stuck stale for 8887559.656919, current state stale+active+clean, 
> last acting [4]
> pg 2.1f is stuck stale for 8887559.656919, current state stale+active+clean, 
> last acting [4]
> pg 7.1b is stuck stale for

Re: [ceph-users] Help: pool not responding

[Karan Singh]

Hey Mario

Agreed to Ferhat.

Recheck your network ( bonds , interfaces , network switches , even cables ) I 
have seen this several times before and in most of the cases its because of 
network.
BTW are you using Mellanox ?

- Karan -

> On 15 Feb 2016, at 10:12, Mario Giammarco  wrote:
> 
> koukou73gr  writes:
> 
>> 
>> Have you tried restarting  osd.0 ?
>> 
> Yes I have restarted all osds many times.
> Also launched repair and scrub.
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Using straw2 crush also with Hammer

[Karan Singh]

> On 11 Nov 2015, at 22:49, David Clarke  wrote:
> 
> On 12/11/15 09:37, Gregory Farnum wrote:
>> On Wednesday, November 11, 2015, Wido den Hollander > > wrote:
>> 
>>On 11/10/2015 09:49 PM, Vickey Singh wrote:
>>> On Mon, Nov 9, 2015 at 8:16 PM, Wido den Hollander >> wrote:
>>> 
 On 11/09/2015 05:27 PM, Vickey Singh wrote:
> Hello Ceph Geeks
> 
> Need your comments with my understanding on straw2.
> 
>   - Is Straw2 better than straw ?
 
 It is not persé better then straw(1).
 
 straw2 distributes data better when not all OSDs are equally
 sized/weighted.
 
>   - Is it straw2 recommended  for production usage ?
> 
 
 I'm using it in multiple clusters and it works fine.
 
>>> 
>>> Yes i followed your update on twitter  :)
>>> 
>>> 
 
> I have a production Ceph Firefly cluster , that i am going to
>>upgrade to
> Ceph hammer pretty soon. Should i use straw2 for all my ceph pools ?
> 
 
 I would upgrade to Hammer first and make sure that ALL clients
>>are updated.
 
 In case you are using KVM/Qemu, you will have to stop those processes
 first and start them again before they are using the new code.
 
>>> 
>>> Thanks a lot for this pointer, i didn't knew this. So restarting
>>KVM / QEMU
>>> process effect running VMs ? ( some downtime )
>>> 
>> 
>>Yes. You can also (live) migrate to another host since that will spawn
>>Qemu with fresh code on the other host.
>> 
>>But you have to make sure all running/connected clients support straw2
>>before you enable straw2.
>> 
>> 
>> I believe straw2 only requires monitor support -- unlike the tuna led
>> involved in executing CRUSH, straw2 is just about how the OSD/bucket
>> weights get converted into a sort of "internal" straw weight. That's
>> done on the monitors and encoded into the maps.
>> 
>> Right?
>> -Greg
> 
> I don't believe that's the case.  If you convert a CRUSH map to use
> straw2 then any connected QEMU/librbd clients without straw2 support
> will die with something like:
> 
> terminate called after throwing an instance of
> 'ceph::buffer::malformed_input'
>  what():  buffer::malformed_input: unsupported bucket algorithm: 5
> 
> Where:
> 
> ceph/src/crush/crush.h:CRUSH_BUCKET_STRAW2 = 5,
> 

Thanks David / Wido for pointing this out.

So does it means , before changing crush with Straw2 , one should make sure 
their clients ( openstack or anything else ) supports straw2 ??
Also does straw2 support comes comes within Kernel OR by installing Ceph Hammer 
/ Later binaries ?
Does Centos 7.1 ( 3.10 ) supports straw2 ?



> 
> -- 
> David Clarke
> Systems Architect
> Catalyst IT
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Did maximum performance reached?

[Karan Singh]

Hi

What type of clients do you have.

- Are they Linux physical OR VM mounting Ceph RBD or CephFS ??
- Or they are simply openstack / cloud instances using Ceph as cinder volumes 
or something like that ??


- Karan -

 On 28 Jul 2015, at 11:53, Shneur Zalman Mattern shz...@eimsys.co.il wrote:
 
 We've built Ceph cluster:
 3 mon nodes (one of them is combined with mds)
 3 osd nodes (each one have 10 osd + 2 ssd for journaling)
 switch 24 ports x 10G
 10 gigabit - for public network
 20 gigabit bonding - between osds 
 Ubuntu 12.04.05
 Ceph 0.87.2
 -
 Clients has:
 10 gigabit for ceph-connection
 CentOS 6.6 with kernel 3.19.8 equipped by cephfs-kmodule 
 
 
 
 == fio-2.0.13 seqwrite, bs=1M, filesize=10G, parallel-jobs=16 ===
 Single client:
 
 
 Starting 16 processes
 
 .below is just 1 job info
 trivial-readwrite-grid01: (groupid=0, jobs=1): err= 0: pid=10484: Tue Jul 28 
 13:26:24 2015
   write: io=10240MB, bw=78656KB/s, iops=76 , runt=133312msec
 slat (msec): min=1 , max=117 , avg=13.01, stdev=12.57
 clat (usec): min=1 , max=68 , avg= 3.61, stdev= 1.99
  lat (msec): min=1 , max=117 , avg=13.01, stdev=12.57
 clat percentiles (usec):
  |  1.00th=[1],  5.00th=[2], 10.00th=[2], 20.00th=[2],
  | 30.00th=[3], 40.00th=[3], 50.00th=[3], 60.00th=[4],
  | 70.00th=[4], 80.00th=[5], 90.00th=[5], 95.00th=[6],
  | 99.00th=[9], 99.50th=[   10], 99.90th=[   23], 99.95th=[   28],
  | 99.99th=[   62]
 bw (KB/s)  : min=35790, max=318215, per=6.31%, avg=78816.91, 
 stdev=26397.76
 lat (usec) : 2=1.33%, 4=54.43%, 10=43.54%, 20=0.56%, 50=0.11%
 lat (usec) : 100=0.03%
   cpu  : usr=0.89%, sys=12.85%, ctx=58248, majf=0, minf=9
   IO depths: 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0%
  submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 =64=0.0%
  complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 =64=0.0%
  issued: total=r=0/w=10240/d=0, short=r=0/w=0/d=0
 
 ...what's above repeats 16 times... 
 
 Run status group 0 (all jobs):
   WRITE: io=163840MB, aggrb=1219.8MB/s, minb=78060KB/s, maxb=78655KB/s, 
 mint=133312msec, maxt=134329msec
 
 +
 Two clients:
 +
 below is just 1 job info
 trivial-readwrite-gridsrv: (groupid=0, jobs=1): err= 0: pid=10605: Tue Jul 28 
 14:05:59 2015
   write: io=10240MB, bw=43154KB/s, iops=42 , runt=242984msec
 slat (usec): min=991 , max=285653 , avg=23716.12, stdev=23960.60
 clat (usec): min=1 , max=65 , avg= 3.67, stdev= 2.02
  lat (usec): min=994 , max=285664 , avg=23723.39, stdev=23962.22
 clat percentiles (usec):
  |  1.00th=[2],  5.00th=[2], 10.00th=[2], 20.00th=[2],
  | 30.00th=[3], 40.00th=[3], 50.00th=[3], 60.00th=[4],
  | 70.00th=[4], 80.00th=[5], 90.00th=[5], 95.00th=[6],
  | 99.00th=[8], 99.50th=[   10], 99.90th=[   28], 99.95th=[   37],
  | 99.99th=[   56]
 bw (KB/s)  : min=20630, max=276480, per=6.30%, avg=43328.34, 
 stdev=21905.92
 lat (usec) : 2=0.84%, 4=49.45%, 10=49.13%, 20=0.37%, 50=0.18%
 lat (usec) : 100=0.03%
   cpu  : usr=0.49%, sys=5.68%, ctx=31428, majf=0, minf=9
   IO depths: 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0%
  submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 =64=0.0%
  complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 =64=0.0%
  issued: total=r=0/w=10240/d=0, short=r=0/w=0/d=0
 
 ...what's above repeats 16 times... 
 
 Run status group 0 (all jobs):
   WRITE: io=163840MB, aggrb=687960KB/s, minb=42997KB/s, maxb=43270KB/s, 
 mint=242331msec, maxt=243869msec
 
 - And almost the same(?!) aggregated result from the second client: 
 -
 
 Run status group 0 (all jobs):
   WRITE: io=163840MB, aggrb=679401KB/s, minb=42462KB/s, maxb=42852KB/s, 
 mint=244697msec, maxt=246941msec
 
 - If I'll summarize: -
 aggrb1 + aggrb2 = 687960KB/s + 679401KB/s = 1367MB/s 
 
 it looks like the same bandwidth from just one client aggrb=1219.8MB/s and it 
 was divided? why?
 Question: If I'll connect 12 clients nodes - each one can write just on 
 100MB/s?
 Perhaps, I need to scale out our ceph up to 15(how many?) OSD nodes - and 
 it'll serve 2 clients on the 1.3GB/s (bw of 10gig nic), or not? 
 
 
 
 health HEALTH_OK
  monmap e1: 3 mons at 
 {mon1=192.168.56.251:6789/0,mon2=192.168.56.252:6789/0,mon3=192.168.56.253:6789/0},
  election epoch 140, quorum 0,1,2 mon1,mon2,mon3
  mdsmap e12: 1/1/1 up {0=mon3=up:active}
  osdmap e832: 31 osds: 30 up, 30 in
   pgmap v106186: 6144 pgs, 3 pools, 2306 GB 

Re: [ceph-users] what's the difference between pg and pgp?

[Karan Singh]

PG = Placement Group
PGP = Placement Group for Placement purpose 

pg_num = number of placement groups mapped to an OSD

When pg_num is increased for any pool, every PG of this pool splits into half, 
but they all remain mapped to their parent OSD. 

Until this time, Ceph does not start rebalancing. Now, when you increase the 
pgp_num value for the same pool, PGs start
to migrate from the parent to some other OSD, and cluster rebalancing starts. 
This is how PGP plays an important role.


Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


 On 22 May 2015, at 02:06, Florent MONTHEL fmont...@flox-arts.net wrote:
 
 To be sure to understand, if I create 2 times replicated pool toto with 1024 
 pgs and 1 pgp, pg and data of pool toto will be mapped on only 2 OSDs and on 
 2 servers right ?
 
 Sent from my iPhone
 
 On 21 mai 2015, at 18:58, Florent MONTHEL fmont...@flox-arts.net wrote:
 
 Thanks Ilya for this clear explanation!
 I'm searching that for a long time
 
 Best practices is to have pg = pgp in order to avoid using of the same set 
 of osd right ? (On a small cluster you will have)
 
 Sent from my iPhone
 
 On 21 mai 2015, at 07:49, Ilya Dryomov idryo...@gmail.com wrote:
 
 On Thu, May 21, 2015 at 12:12 PM, baijia...@126.com baijia...@126.com 
 wrote:
 Re: what's the difference between pg and pgp?
 
 pg-num is the number of PGs, pgp-num is the number of PGs that will be
 considered for placement, i.e. it's the pgp-num value that is used by
 CRUSH, not pg-num.  For example, consider pg-num = 1024 and pgp-num
 = 1.  In that case you will see 1024 PGs but all of those PGs will map
 to the same set of OSDs.
 
 When you increase pg-num you are splitting PGs, when you increase
 pgp-num you are moving them, i.e. changing sets of OSDs they map to.
 
 Thanks,
 
  Ilya
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can not access the Ceph's main page ceph.com intermittently

[Karan Singh]

Hi Wenjun

Are you sure , is everything all right at your end. 

ceph.com http://ceph.com/ seems to be working fine for me from Finland


Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


 On 30 Apr 2015, at 09:56, 黄文俊 huangwenjun...@gmail.com wrote:
 
 Hi, Sage
 
 This is Wenjun Huang from Beijing China, I found that I can not access ceph's 
 main site ceph.com http://ceph.com/ intermittently. The issue looks so 
 strange, always I can access the site normally at a time, but I can not 
 access it some seconds later(the site does not response for a very long time 
 and then timeout). The issue has existed some days, so I beg for your help. 
 Could you please help to check the problem. I do not think it is the GFW's 
 problem of China, because I can access it normally sometimes,  but seems like 
 that there is some problem with the site server.
 
 Thanks
 
 Wenjun
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] about rgw region and zone

[Karan Singh]

You should try to create a new user without —system option so basically create 
a normal user , then create some bucket and object and finally try to resync 
cluster.



Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


 On 28 Apr 2015, at 21:22, Karan Singh karan.si...@csc.fi wrote:
 
 You should try to create a new user without —system option so basically 
 create a normal user , then create some bucket and object and finally try to 
 resync cluster.
 
 -Karan-
 
 On 28 Apr 2015, at 10:32, TERRY 316828...@qq.com mailto:316828...@qq.com 
 wrote:
 
  Hi：
 
 karan， singh. First of all thank you so  much for  replying and giving 
 your precious time to this problem!
 
 I tried repeat steps 2, 4 and 5 in by executing them with 
 --nameclient.radosgw-us-west-1. and the case has progressed a lot。(below are 
 some the  logs)
 
  
 I am now getting
 
 sudo radosgw-agent -c ceph-data-sync2.conf 
 region map is: {u'us': [u'us-west', u'us-east']}
 INFO:radosgw_agent.sync:Starting sync
 INFO:radosgw_agent.worker:20585 is processing shard number 0
 INFO:radosgw_agent.worker:finished processing shard 0
 INFO:radosgw_agent.worker:20585 is processing shard number 1
 INFO:radosgw_agent.sync:1/64 items processed
 INFO:radosgw_agent.sync:2/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 1
 INFO:radosgw_agent.worker:20585 is processing shard number 2
 INFO:radosgw_agent.sync:3/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 2
 INFO:radosgw_agent.worker:20585 is processing shard number 3
 INFO:radosgw_agent.sync:4/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 3
 INFO:radosgw_agent.worker:20585 is processing shard number 4
 INFO:radosgw_agent.sync:5/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 4
 INFO:radosgw_agent.worker:20585 is processing shard number 5
 INFO:radosgw_agent.sync:6/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 5
 INFO:radosgw_agent.worker:20585 is processing shard number 6
 INFO:radosgw_agent.sync:7/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 6
 INFO:radosgw_agent.worker:20585 is processing shard number 7
 INFO:radosgw_agent.sync:8/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 7
 INFO:radosgw_agent.worker:20585 is processing shard number 8
 INFO:radosgw_agent.sync:9/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 8
 INFO:radosgw_agent.worker:20585 is processing shard number 9
 INFO:radosgw_agent.sync:10/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 9
 INFO:radosgw_agent.worker:20585 is processing shard number 10
 INFO:radosgw_agent.sync:11/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 10
 INFO:radosgw_agent.worker:20585 is processing shard number 11
 INFO:radosgw_agent.sync:12/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 11
 INFO:radosgw_agent.worker:20585 is processing shard number 12
 INFO:radosgw_agent.sync:13/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 12
 INFO:radosgw_agent.worker:20585 is processing shard number 13
 INFO:radosgw_agent.sync:14/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 13
 INFO:radosgw_agent.worker:20585 is processing shard number 14
 INFO:radosgw_agent.sync:15/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 14
 INFO:radosgw_agent.worker:20585 is processing shard number 15
 INFO:radosgw_agent.sync:16/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 15
 INFO:radosgw_agent.worker:20585 is processing shard number 16
 INFO:radosgw_agent.sync:17/64 items processed
 INFO:radosgw_agent.worker:finished processing shard 16
 INFO:radosgw_agent.worker:20585 is processing shard number 17
 
 ..
 
 ..
 
 ..
 
 INFO:radosgw_agent.worker:syncing bucket my_container
 ERROR:radosgw_agent.worker:failed to sync object my_container/us.json: state 
 is error
 
  
 fyi: my_container  is created by me in master zone, us.json is a object in  
 my_container.
 
 i want to sync the object us.json to zone us-west zone.
 
  
  
 
 
 -- 原始邮件 --
 发件人: Karan Singh;karan.si...@csc.fi mailto:karan.si...@csc.fi;
 发送时间: 2015年4月28日(星期二) 下午3:02
 收件人: TERRY316828...@qq.com mailto:316828...@qq.com;
 抄送: ceph-usersceph-users@lists.ceph.com 
 mailto:ceph-users@lists.ceph.com;
 主题: Re: [ceph-users] about rgw region and zone
 
  Hi
 
 
 
 On 28 Apr 2015, at 07:12, TERRY 316828...@qq.com 
 mailto:316828...@qq.com wrote:
 
 Hi: all
  
 when I Configuring Federated Gateways， I got  the error as below:
  
 sudo

Re: [ceph-users] about rgw region and zone

[Karan Singh]

 Hi 



 On 28 Apr 2015, at 07:12, TERRY 316828...@qq.com wrote:
 
 Hi: all
  
 when I Configuring Federated Gateways， I got  the error as below:
  
 sudo radosgw-agent  -c  /etc/ceph/ceph-data-sync.conf  
 ERROR:root:Could not retrieve region map from destination
 You should check that the region map is correct , especially the endpoints. 
Make sure firewall is not blocking the way between RGW instances.

 Traceback (most recent call last):
   File /usr/lib/python2.6/site-packages/radosgw_agent/cli.py, line 269, in 
 main
 region_map = client.get_region_map(dest_conn)
   File /usr/lib/python2.6/site-packages/radosgw_agent/client.py, line 391, 
 in get_region_map
 region_map = request(connection, 'get', 'admin/config')
   File /usr/lib/python2.6/site-packages/radosgw_agent/client.py, line 155, 
 in request
 check_result_status(result)
   File /usr/lib/python2.6/site-packages/radosgw_agent/client.py, line 116, 
 in check_result_status
 HttpError)(result.status_code, result.content)
 NotFound: Http error code 404 content {Code:NoSuchKey”}

Cross verify the key

 I  have some quesions when I execute the command
 1、radosgw-admin zone set --rgw-zone=us-west --infile us-west.json --name 
 client.radosgw.us-west-1
 i  have no idea about the option --name ,   what's the difference  if  i  do 
 it without --name config;

—name us used to supply ceph user name that should be used to run radosgw-admin 
command against. Default is client.admin , if client.admin keyring are not 
present you need to supply another user with —name option to run Ceph commands. 

  
 2、Create a Region
 there is a conversation near the end of doc :
If you use different Ceph Storage Cluster instances for regions, you 
 should repeat steps 2, 4 and 5 in by executing them with 
 --nameclient.radosgw-us-west-1. You may also export the region map from the 
 initial gateway instance and import it followed by updating the region map.
  
 I has one cluster named ceph, one region named us, and two zones: us-east、 
 us-west， us-east is the  master  zone. I has two gateway 
 instances：client.radosgw.us-east-1、client.radosgw.us-west-1.  Do i need 
 repeat steps 2,4,and 5? do  i need export the region map from the  initial 
 gateway instance and import it

If you have 2 Different Ceph Cluster then only you should do this.  I assume 
you have only 1 Ceph cluster with 1 region and 2 zones and 2 gateway instances 
, then you need not to repeat those steps.

  
  
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Radosgw: upgrade Firefly to Hammer, impossible to create bucket

[Karan Singh]

Things you can check 

* Is RGW node able to resolve bucket-2.ostore.athome.priv  , try ping 
bucket-2.ostore.athome.priv
* Is # s3cmd ls working or throwing errors ?

l
Are you sure the below entries are correct ? Generally host_base and 
host_bucket should point to RGW FQDN in your case ceph-radosgw1 FQDN . 
ostore.athome.priv looks like a different host to me.

host_base-ostore.athome.priv
host_bucket-%(bucket)s.ostore.athome.priv



Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


 On 13 Apr 2015, at 06:47, Francois Lafont flafdiv...@free.fr wrote:
 
 Hi,
 
 On a testing cluster, I have a radosgw on Firefly and the other
 nodes, OSDs and monitors, are on Hammer. The nodes are installed
 with puppet in personal VM, so I can reproduce the problem.
 Generally, I use s3cmd to check the radosgw. While radosgw is on
 Firefly, I can create bucket, no problem. Then, I upgrade the
 radosgw (it's a Ubuntu Trusty):
 
sed -i 's/firefly/hammer/g' /etc/apt/sources.list.d/ceph.list
apt-get update  apt-get dist-upgrade -y
service stop apache2
stop radosgw-all
start radosgw-all 
service apache2 start
 
 After that, impossible to create a bucket with s3cmd:
 
 --
 ~# s3cmd -d mb s3://bucket-2
 DEBUG: ConfigParser: Reading file '/root/.s3cfg'
 DEBUG: ConfigParser: bucket_location-US
 DEBUG: ConfigParser: cloudfront_host-cloudfront.amazonaws.com
 DEBUG: ConfigParser: default_mime_type-binary/octet-stream
 DEBUG: ConfigParser: delete_removed-False
 DEBUG: ConfigParser: dry_run-False
 DEBUG: ConfigParser: enable_multipart-True
 DEBUG: ConfigParser: encoding-UTF-8
 DEBUG: ConfigParser: encrypt-False
 DEBUG: ConfigParser: follow_symlinks-False
 DEBUG: ConfigParser: force-False
 DEBUG: ConfigParser: get_continue-False
 DEBUG: ConfigParser: gpg_command-/usr/bin/gpg
 DEBUG: ConfigParser: gpg_decrypt-%(gpg_command)s -d --verbose --no-use-agent 
 --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s 
 %(input_file)s
 DEBUG: ConfigParser: gpg_encrypt-%(gpg_command)s -c --verbose --no-use-agent 
 --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s 
 %(input_file)s
 DEBUG: ConfigParser: gpg_passphrase-...-3_chars...
 DEBUG: ConfigParser: guess_mime_type-True
 DEBUG: ConfigParser: host_base-ostore.athome.priv
 DEBUG: ConfigParser: access_key-5R...17_chars...Y
 DEBUG: ConfigParser: secret_key-Ij...37_chars...I
 DEBUG: ConfigParser: host_bucket-%(bucket)s.ostore.athome.priv
 DEBUG: ConfigParser: human_readable_sizes-False
 DEBUG: ConfigParser: invalidate_on_cf-False
 DEBUG: ConfigParser: list_md5-False
 DEBUG: ConfigParser: log_target_prefix-
 DEBUG: ConfigParser: mime_type-
 DEBUG: ConfigParser: multipart_chunk_size_mb-15
 DEBUG: ConfigParser: preserve_attrs-True
 DEBUG: ConfigParser: progress_meter-True
 DEBUG: ConfigParser: proxy_host-
 DEBUG: ConfigParser: proxy_port-0
 DEBUG: ConfigParser: recursive-False
 DEBUG: ConfigParser: recv_chunk-4096
 DEBUG: ConfigParser: reduced_redundancy-False
 DEBUG: ConfigParser: send_chunk-4096
 DEBUG: ConfigParser: simpledb_host-sdb.amazonaws.com
 DEBUG: ConfigParser: skip_existing-False
 DEBUG: ConfigParser: socket_timeout-300
 DEBUG: ConfigParser: urlencoding_mode-normal
 DEBUG: ConfigParser: use_https-False
 DEBUG: ConfigParser: verbosity-WARNING
 DEBUG: ConfigParser: 
 website_endpoint-http://%(bucket)s.s3-website-%(location)s.amazonaws.com/
 DEBUG: ConfigParser: website_error-
 DEBUG: ConfigParser: website_index-index.html
 DEBUG: Updating Config.Config encoding - UTF-8
 DEBUG: Updating Config.Config follow_symlinks - False
 DEBUG: Updating Config.Config verbosity - 10
 DEBUG: Unicodising 'mb' using UTF-8
 DEBUG: Unicodising 's3://bucket-2' using UTF-8
 DEBUG: Command: mb
 DEBUG: SignHeaders: 'PUT\n\n\n\nx-amz-date:Mon, 13 Apr 2015 03:32:23 
 +\n/bucket-2/'
 DEBUG: CreateRequest: resource[uri]=/
 DEBUG: SignHeaders: 'PUT\n\n\n\nx-amz-date:Mon, 13 Apr 2015 03:32:23 
 +\n/bucket-2/'
 DEBUG: Processing request, please wait...
 DEBUG: get_hostname(bucket-2): bucket-2.ostore.athome.priv
 DEBUG: format_uri(): /
 DEBUG: Sending request method_string='PUT', uri='/', 
 headers={'content-length': '0', 'Authorization': 'AWS 
 5RUS0Z3SBG6IK263PLFY:3V1MdXoCGFrJKrO2LSJaBpNMcK4=', 'x-amz-date': 'Mon, 13 
 Apr 2015 03:32:23 +'}, body=(0 bytes)
 DEBUG: Response: {'status': 405, 'headers': {'date': 'Mon, 13 Apr 2015 
 03:32:23 GMT', 'accept-ranges': 'bytes', 'content-type': 'application/xml', 
 'content-length': '82', 'server': 'Apache/2.4.7 (Ubuntu)'}, 'reason': 'Method 
 Not Allowed', 'data': '?xml version=1.0 
 encoding=UTF-8?ErrorCodeMethodNotAllowed/Code/Error'}
 DEBUG: S3Error: 405 (Method Not Allowed

Re: [ceph-users] Radosgw: upgrade Firefly to Hammer, impossible to create bucket

[Karan Singh]

You can give a try with swift API as well.


Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


 On 13 Apr 2015, at 15:19, Francois Lafont flafdiv...@free.fr wrote:
 
 Karan Singh wrote:
 
 Things you can check 
 
 * Is RGW node able to resolve bucket-2.ostore.athome.priv  , try ping 
 bucket-2.ostore.athome.priv
 
 Yes, my DNS configuration is ok. In fact, I test s3cmd directly
 on my radosgw (its hostname is ceph-radosgw1 but its fqdn is
 ostore.athome.priv):
 
 ---
 ~# hostname
 ceph-radosgw1
 
 ~# ip addr show dev eth0 | grep 'inet '
inet 172.31.10.6/16 brd 172.31.255.255 scope global eth0
 
 ~# dig +short  ostore.athome.priv \
  bucket-2.ostore.athome.priv \
   foo.ostore.athome.priv \
   bar.ostore.athome.priv \
   hjffhkj.ostore.athome.priv 
 172.31.10.6
 172.31.10.6
 172.31.10.6
 172.31.10.6
 172.31.10.6
 
 ~# getent hosts ostore.athome.priv
 172.31.10.6 ostore.athome.priv
 
 ~# getent hosts jfkjfl.ostore.athome.priv
 172.31.10.6 jfkjfl.ostore.athome.priv
 ---
 
 * Is # s3cmd ls working or throwing errors ?
 
 It doesn't work after upgrading with Hammer too. More
 precisely, in Firefly radosgw, It works:
 
 ---
 ~# s3cmd ls s3://bucket
 2015-04-12 23:35 735985664   s3://bucket/les_evades.avi
 
 ~# s3cmd ls
 2015-04-12 23:28  s3://bucket
 ---
 
 But after the upgrade to Hammer, it doesn't work:
 
 ---
 ~# s3cmd ls s3://bucket
 ERROR: S3 error: 403 (SignatureDoesNotMatch): 
 
 ~# s3cmd ls
 2015-04-12 23:28  s3://bucket
 ---
 
 As you can see, the second command works but not the first.
 [1] At the end of this message, I put the output of the first
 command with the debug option, just in case.
 
 Are you sure the below entries are correct ? Generally host_base and 
 host_bucket should point to RGW FQDN in your case ceph-radosgw1 FQDN . 
 ostore.athome.priv looks like a different host to me.
 
 host_base-ostore.athome.priv
 host_bucket-%(bucket)s.ostore.athome.priv
 
 For me it's ok:
 
 ---
 ~# grep 'host_' ~/.s3cfg 
 host_base = ostore.athome.priv
 host_bucket = %(bucket)s.ostore.athome.priv
 ---
 
 And ostore.athome.priv is really my radosgw (see the dig
 commands above). And when I try a s3cmd command, I can
 see new lines in the apache logs of my radosgw.
 
 Thanks for your help Karan.
 
 [1]
 
 ---
 ~# s3cmd -d ls s3://bucket
 DEBUG: ConfigParser: Reading file '/root/.s3cfg'
 DEBUG: ConfigParser: bucket_location-US
 DEBUG: ConfigParser: cloudfront_host-ostore.athome.priv
 DEBUG: ConfigParser: default_mime_type-binary/octet-stream
 DEBUG: ConfigParser: delete_removed-False
 DEBUG: ConfigParser: dry_run-False
 DEBUG: ConfigParser: enable_multipart-True
 DEBUG: ConfigParser: encoding-UTF-8
 DEBUG: ConfigParser: encrypt-False
 DEBUG: ConfigParser: follow_symlinks-False
 DEBUG: ConfigParser: force-False
 DEBUG: ConfigParser: get_continue-False
 DEBUG: ConfigParser: gpg_command-/usr/bin/gpg
 DEBUG: ConfigParser: gpg_decrypt-%(gpg_command)s -d --verbose --no-use-agent 
 --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s 
 %(input_file)s
 DEBUG: ConfigParser: gpg_encrypt-%(gpg_command)s -c --verbose --no-use-agent 
 --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s 
 %(input_file)s
 DEBUG: ConfigParser: gpg_passphrase-...-3_chars...
 DEBUG: ConfigParser: guess_mime_type-True
 DEBUG: ConfigParser: host_base-ostore.athome.priv
 DEBUG: ConfigParser: access_key-1Q...17_chars...Y
 DEBUG: ConfigParser: secret_key-92...37_chars...W
 DEBUG: ConfigParser: host_bucket-%(bucket)s.ostore.athome.priv
 DEBUG: ConfigParser: human_readable_sizes-False
 DEBUG: ConfigParser: invalidate_on_cf-False
 DEBUG: ConfigParser: list_md5-False
 DEBUG: ConfigParser: log_target_prefix-
 DEBUG: ConfigParser: mime_type-
 DEBUG: ConfigParser: multipart_chunk_size_mb-15
 DEBUG: ConfigParser: preserve_attrs-True
 DEBUG: ConfigParser: progress_meter-True
 DEBUG: ConfigParser: proxy_host-
 DEBUG: ConfigParser: proxy_port-0
 DEBUG: ConfigParser: recursive-False
 DEBUG: ConfigParser: recv_chunk-4096
 DEBUG: ConfigParser: reduced_redundancy-False
 DEBUG: ConfigParser: send_chunk-4096
 DEBUG: ConfigParser: simpledb_host-ostore.athome.priv
 DEBUG: ConfigParser: skip_existing-False
 DEBUG: ConfigParser: socket_timeout-300
 DEBUG: ConfigParser: urlencoding_mode-normal
 DEBUG: ConfigParser: use_https-False
 DEBUG: ConfigParser: verbosity-WARNING
 DEBUG: ConfigParser

Re: [ceph-users] Radosgw: upgrade Firefly to Hammer, impossible to create bucket

[Karan Singh]

Also what version of s3cmd you are using ??

To me the error “S3 error: 403 (SignatureDoesNotMatch)” seems to be from s3cmd 
side rather RGW. 

But lets diagnose.


Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


 On 13 Apr 2015, at 15:43, Karan Singh karan.si...@csc.fi wrote:
 
 You can give a try with swift API as well.
 
 
 Karan Singh 
 Systems Specialist , Storage Platforms
 CSC - IT Center for Science,
 Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
 mobile: +358 503 812758
 tel. +358 9 4572001
 fax +358 9 4572302
 http://www.csc.fi/ http://www.csc.fi/
 
 
 On 13 Apr 2015, at 15:19, Francois Lafont flafdiv...@free.fr 
 mailto:flafdiv...@free.fr wrote:
 
 Karan Singh wrote:
 
 Things you can check 
 
 * Is RGW node able to resolve bucket-2.ostore.athome.priv  , try ping 
 bucket-2.ostore.athome.priv
 
 Yes, my DNS configuration is ok. In fact, I test s3cmd directly
 on my radosgw (its hostname is ceph-radosgw1 but its fqdn is
 ostore.athome.priv):
 
 ---
 ~# hostname
 ceph-radosgw1
 
 ~# ip addr show dev eth0 | grep 'inet '
inet 172.31.10.6/16 brd 172.31.255.255 scope global eth0
 
 ~# dig +short  ostore.athome.priv \
  bucket-2.ostore.athome.priv \
   foo.ostore.athome.priv \
   bar.ostore.athome.priv \
   hjffhkj.ostore.athome.priv 
 172.31.10.6
 172.31.10.6
 172.31.10.6
 172.31.10.6
 172.31.10.6
 
 ~# getent hosts ostore.athome.priv
 172.31.10.6 ostore.athome.priv
 
 ~# getent hosts jfkjfl.ostore.athome.priv
 172.31.10.6 jfkjfl.ostore.athome.priv
 ---
 
 * Is # s3cmd ls working or throwing errors ?
 
 It doesn't work after upgrading with Hammer too. More
 precisely, in Firefly radosgw, It works:
 
 ---
 ~# s3cmd ls s3://bucket s3://bucket
 2015-04-12 23:35 735985664   s3://bucket/les_evades.avi 
 s3://bucket/les_evades.avi
 
 ~# s3cmd ls
 2015-04-12 23:28  s3://bucket s3://bucket
 ---
 
 But after the upgrade to Hammer, it doesn't work:
 
 ---
 ~# s3cmd ls s3://bucket s3://bucket
 ERROR: S3 error: 403 (SignatureDoesNotMatch): 
 
 ~# s3cmd ls
 2015-04-12 23:28  s3://bucket s3://bucket
 ---
 
 As you can see, the second command works but not the first.
 [1] At the end of this message, I put the output of the first
 command with the debug option, just in case.
 
 Are you sure the below entries are correct ? Generally host_base and 
 host_bucket should point to RGW FQDN in your case ceph-radosgw1 FQDN . 
 ostore.athome.priv looks like a different host to me.
 
 host_base-ostore.athome.priv
 host_bucket-%(bucket)s.ostore.athome.priv
 
 For me it's ok:
 
 ---
 ~# grep 'host_' ~/.s3cfg 
 host_base = ostore.athome.priv
 host_bucket = %(bucket)s.ostore.athome.priv
 ---
 
 And ostore.athome.priv is really my radosgw (see the dig
 commands above). And when I try a s3cmd command, I can
 see new lines in the apache logs of my radosgw.
 
 Thanks for your help Karan.
 
 [1]
 
 ---
 ~# s3cmd -d ls s3://bucket s3://bucket
 DEBUG: ConfigParser: Reading file '/root/.s3cfg'
 DEBUG: ConfigParser: bucket_location-US
 DEBUG: ConfigParser: cloudfront_host-ostore.athome.priv
 DEBUG: ConfigParser: default_mime_type-binary/octet-stream
 DEBUG: ConfigParser: delete_removed-False
 DEBUG: ConfigParser: dry_run-False
 DEBUG: ConfigParser: enable_multipart-True
 DEBUG: ConfigParser: encoding-UTF-8
 DEBUG: ConfigParser: encrypt-False
 DEBUG: ConfigParser: follow_symlinks-False
 DEBUG: ConfigParser: force-False
 DEBUG: ConfigParser: get_continue-False
 DEBUG: ConfigParser: gpg_command-/usr/bin/gpg
 DEBUG: ConfigParser: gpg_decrypt-%(gpg_command)s -d --verbose 
 --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o 
 %(output_file)s %(input_file)s
 DEBUG: ConfigParser: gpg_encrypt-%(gpg_command)s -c --verbose 
 --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o 
 %(output_file)s %(input_file)s
 DEBUG: ConfigParser: gpg_passphrase-...-3_chars...
 DEBUG: ConfigParser: guess_mime_type-True
 DEBUG: ConfigParser: host_base-ostore.athome.priv
 DEBUG: ConfigParser: access_key-1Q...17_chars...Y
 DEBUG: ConfigParser: secret_key-92...37_chars...W
 DEBUG: ConfigParser: host_bucket-%(bucket)s.ostore.athome.priv
 DEBUG: ConfigParser: human_readable_sizes-False
 DEBUG: ConfigParser: invalidate_on_cf-False
 DEBUG: ConfigParser

Re: [ceph-users] Ceph and Openstack

[Karan Singh]

Geddes

If you are still struggling with this , ping me in IRC #CEPH ( ksingh )  


Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


 On 02 Apr 2015, at 19:48, Quentin Hartman qhart...@direwolfdigital.com 
 wrote:
 
 Well, 100% may be overstating things. When I try to create a volume from an 
 image it fails. I'm digging through the logs right now. glance alone works (I 
 can upload and delete images) and cinder alone works (I can create and delete 
 volumes) but when cinder tries to get the glance service it fails, it seems 
 to be trying to contact the completely wrong IP:
 
 2015-04-02 16:39:05.033 24986 TRACE cinder.api.middleware.fault 
 CommunicationError: Error finding address for 
 http://192.168.1.18:9292/v2/schemas/image 
 http://192.168.1.18:9292/v2/schemas/image: 
 HTTPConnectionPool(host='192.168.1.18', port=9292): Max retries exceeded with 
 url: /v2/schemas/image (Caused by class 'socket.error': [Errno 111] 
 ECONNREFUSED)
 
 Which I would expect to fail, since my glance service is not on that machine. 
 I assume that cinder gets this information out of keystone's endpoint 
 registry, but that lists the correct IP for glance:
 
 | cf833cf63944490ba69a49a7af7fa2f5 |  office   |  
 http://glance-host:9292 http://glance-host:9292/ |  
 http://192.168.1.20:9292 http://192.168.1.20:9292/ |  
 http://glance-host:9292 http://glance-host:9292/ | 
 a2a74e440b134e08bd526d6dd36540d2 |
 
 But this is probably something to move to an Openstack list. Thanks for all 
 the ideas and talking through things.
 
 QH
 
 On Thu, Apr 2, 2015 at 10:41 AM, Erik McCormick emccorm...@cirrusseven.com 
 mailto:emccorm...@cirrusseven.com wrote:
 
 
 On Thu, Apr 2, 2015 at 12:18 PM, Quentin Hartman 
 qhart...@direwolfdigital.com mailto:qhart...@direwolfdigital.com wrote:
 Hm, even lacking the mentions of rbd in the glance docs, and the lack of 
 cephx auth information in the config, glance seems to be working after all. 
 S, hooray! It was probably working all along, I just hadn't gotten to 
 really testing it since I was getting blocked by my typo on the cinder config.
 
 
 
 Glance sets defaults for almost everything, so just enabling the default 
 store will work. I thought you needed to specify a username still, but maybe 
 that's defaulted now as well. Glad it's working. So Quentin is 100% working 
 now and  Iain has no Cinder and slow Glance. Right? 
 
 
 Erik - 
 
 Here's my output for the requested grep (though I am on Ubuntu, so the path 
 was slightly different:
 
 cfg.IntOpt('rbd_store_chunk_size', default=DEFAULT_CHUNKSIZE,
 def __init__(self, name, store, chunk_size=None):
 self.chunk_size = chunk_size or store.READ_CHUNKSIZE
 length = min(self.chunk_size, bytes_left)
 chunk = self.conf.glance_store.rbd_store_chunk_size
 self.chunk_size = chunk * (1024 ** 2)
 self.READ_CHUNKSIZE = self.chunk_size
 def get(self, location, offset=0, chunk_size=None, context=None):
 return (ImageIterator(loc.image, self, chunk_size=chunk_size),
 chunk_size or self.get_size(location))
 
 
 This all looks correct, so any slowness isn't the bug I was thinking of.  
 
 QH
 
 On Thu, Apr 2, 2015 at 10:06 AM, Erik McCormick emccorm...@cirrusseven.com 
 mailto:emccorm...@cirrusseven.com wrote:
 The RDO glance-store package had a bug in it that miscalculated the chunk 
 size. I should hope that it's been patched by Redhat now since the fix was 
 committed upstream before the first Juno rleease, but perhaps not. The 
 symptom of the bug was horribly slow uploads to glance.
 
 Run this and send back the output:
 
 grep chunk_size /usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py
 
 -Erik
 
 On Thu, Apr 2, 2015 at 7:34 AM, Iain Geddes iain.ged...@cyaninc.com 
 mailto:iain.ged...@cyaninc.com wrote:
 Oh, apologies, I missed the versions ...
 
 # glance --version   :   0.14.2
 # cinder --version   :   1.1.1
 # ceph -v:   ceph version 0.87.1 
 (283c2e7cfa2457799f534744d7d549f83ea1335e)
 
 From rpm I can confirm that Cinder and Glance are both of the February 2014 
 vintage:
 
 # rpm -qa |grep -e ceph -e glance -e cinder
 ceph-0.87.1-0.el7.x86_64
 libcephfs1-0.87.1-0.el7.x86_64
 ceph-common-0.87.1-0.el7.x86_64
 python-ceph-0.87.1-0.el7.x86_64
 openstack-cinder-2014.2.2-1.el7ost.noarch
 python-cinder-2014.2.2-1.el7ost.noarch
 python-cinderclient-1.1.1-1.el7ost.noarch
 python-glanceclient-0.14.2-2.el7ost.noarch
 python-glance-2014.2.2-1.el7ost.noarch
 python-glance-store-0.1.10-2.el7ost.noarch
 openstack-glance-2014.2.2-1.el7ost.noarch
 
 On Thu, Apr 2, 2015 at 4:24 AM, Iain

Re: [ceph-users] Ceph and Openstack

[Karan Singh]

Fortunately Ceph Giant + OpenStack Juno works flawlessly for me.

If you have configured cinder / glance correctly , then after restarting  
cinder and glance services , you should see something like this in cinder and 
glance logs.


Cinder logs : 

volume.log:2015-04-02 13:20:43.943 2085 INFO cinder.volume.manager 
[req-526cb14e-42ef-4c49-b033-e9bf2096be8f - - - - -] Starting volume driver 
RBDDriver (1.1.0)


Glance Logs:

api.log:2015-04-02 13:20:50.448 1266 DEBUG glance.common.config [-] 
glance_store.default_store = rbd log_opt_values 
/usr/lib/python2.7/site-packages/oslo/config/cfg.py:2004
api.log:2015-04-02 13:20:50.449 1266 DEBUG glance.common.config [-] 
glance_store.rbd_store_ceph_conf = /etc/ceph/ceph.conf log_opt_values 
/usr/lib/python2.7/site-packages/oslo/config/cfg.py:2004
api.log:2015-04-02 13:20:50.449 1266 DEBUG glance.common.config [-] 
glance_store.rbd_store_chunk_size = 8 log_opt_values 
/usr/lib/python2.7/site-packages/oslo/config/cfg.py:2004
api.log:2015-04-02 13:20:50.449 1266 DEBUG glance.common.config [-] 
glance_store.rbd_store_pool= images log_opt_values 
/usr/lib/python2.7/site-packages/oslo/config/cfg.py:2004
api.log:2015-04-02 13:20:50.449 1266 DEBUG glance.common.config [-] 
glance_store.rbd_store_user= glance log_opt_values 
/usr/lib/python2.7/site-packages/oslo/config/cfg.py:2004
api.log:2015-04-02 13:20:50.451 1266 DEBUG glance.common.config [-] 
glance_store.stores= ['rbd'] log_opt_values 
/usr/lib/python2.7/site-packages/oslo/config/cfg.py:2004


If Cinder and Glance are able to initialize RBD driver , then everything should 
work like charm.



Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


 On 02 Apr 2015, at 03:10, Erik McCormick emccorm...@cirrusseven.com wrote:
 
 Can you both set Cinder and / or Glance logging to debug and provide some 
 logs? There was an issue with the first Juno release of Glance in some vendor 
 packages, so make sure you're fully updated to 2014.2.2
 
 On Apr 1, 2015 7:12 PM, Quentin Hartman qhart...@direwolfdigital.com 
 mailto:qhart...@direwolfdigital.com wrote:
 I am conincidentally going through the same process right now. The best 
 reference I've found is this: http://ceph.com/docs/master/rbd/rbd-openstack/ 
 http://ceph.com/docs/master/rbd/rbd-openstack/
 
 When I did Firefly / icehouse, this (seemingly) same guide Just Worked(tm), 
 but now with Giant / Juno I'm running into similar trouble  to that which you 
 describe. Everything _seems_ right, but creating volumes via openstack just 
 sits and spins forever, never creating anything and (as far as i've found so 
 far) not logging anything interesting. Normal Rados operations work fine.
 
 Feel free to hit me up off list if you want to confer and then we can return 
 here if we come up with anything to be shared with the group.
 
 QH
 
 On Wed, Apr 1, 2015 at 3:43 PM, Iain Geddes iain.ged...@cyaninc.com 
 mailto:iain.ged...@cyaninc.com wrote:
 All,
 
 Apologies for my ignorance but I don't seem to be able to search an archive. 
 
 I've spent a lot of time trying but am having difficulty in integrating Ceph 
 (Giant) into Openstack (Juno). I don't appear to be recording any errors 
 anywhere, but simply don't seem to be writing to the cluster if I try 
 creating a new volume or importing an image. The cluster is good and I can 
 create a static rbd mapping so I know the key components are in place. My 
 problem is almost certainly finger trouble on my part but am completely lost 
 and wondered if there was a well thumbed guide to integration?
 
 Thanks
 
 
 Iain
 
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Production Ceph :: PG data lost : Cluster PG incomplete, inactive, unclean

[Karan Singh]

   
[15,102,147]15  [15,102,147]15  33568'304   2015-03-16 
10:04:19.894789  2246'4  2015-03-09 11:43:44.176331
8.1d4   0   0   0   0   0   0   0   down+incomplete 
2015-04-01 21:21:16.218644  0'0 262813:21867[126,43,174]126 
[126,43,174]126 0'0 2015-03-12 14:34:35.258338  0'0 
2015-03-12 14:34:35.258338
4.2f4   0   0   0   0   0   0   0   down+incomplete 
2015-04-01 21:21:16.117515  0'0 262813:116150   [181,186,13]181 
[181,186,13]181 0'0 2015-03-12 14:59:03.529264  0'0 
2015-03-09 13:46:40.601301
3.e5a   76  70  0   0   623902741   325 325 
incomplete  2015-04-01 21:21:16.043300  33569'325   262813:73426
[97,22,62]  97  [97,22,62]  97  33569'325   2015-03-12 
13:58:05.813966  28433'442015-03-11 13:57:53.909795
8.3a0   0   0   0   0   0   0   0   incomplete  
2015-04-01 21:21:16.056437  0'0 262813:175168   [62,14,224] 62  
[62,14,224] 62  0'0 2015-03-12 13:52:44.546418  0'0 
2015-03-12 13:52:44.546418
3.24e   0   0   0   0   0   0   0   incomplete  
2015-04-01 21:21:16.130831  0'0 262813:165  [39,202,90] 39  
[39,202,90] 39  33556'272   2015-03-13 11:44:41.263725  2327'4  
2015-03-09 17:54:43.675552
5.f70   0   0   0   0   0   0   incomplete  
2015-04-01 21:21:16.145298  0'0 262813:153  [54,193,123]54  
[54,193,123]54  0'0 2015-03-12 17:58:30.257371  0'0 
2015-03-09 17:55:11.725629
[root@pouta-s01 ceph]#


##  Example 1 : PG 10.70 ###


10.70   0   0   0   0   0   0   0   incomplete  
2015-04-01 21:21:16.152179  0'0 262813:163  [213,88,80] 213 
[213,88,80] 213 0'0 2015-03-12 17:59:43.275049  0'0 
2015-03-09 17:55:58.745662


This is how i found location of each OSD

[root@pouta-s01 ceph]# ceph osd find 88

{ osd: 88,
  ip: 10.100.50.3:7079\/916853,
  crush_location: { host: pouta-s03,
  root: default”}}
[root@pouta-s01 ceph]#


When i manually check current/pg_head directory , data is not present here ( 
i.e. data is lost from all the copies )


[root@pouta-s04 current]# ls -l /var/lib/ceph/osd/ceph-80/current/10.70_head
total 0
[root@pouta-s04 current]#


On some of the OSD’s HEAD directory does not exists 

[root@pouta-s03 ~]# ls -l /var/lib/ceph/osd/ceph-88/current/10.70_head
ls: cannot access /var/lib/ceph/osd/ceph-88/current/10.70_head: No such file or 
directory
[root@pouta-s03 ~]#

[root@pouta-s02 ~]# ls -l /var/lib/ceph/osd/ceph-213/current/10.70_head
total 0
[root@pouta-s02 ~]#


# ceph pg 10.70 query  ---  http://paste.ubuntu.com/10719840/


##  Example 2 : PG 3.7d0 ###

3.7d0   78  0   0   0   609222686   376 376 
down+incomplete 2015-04-01 21:21:16.135599  33538'376   262813:185045   
[117,118,177]   117 [117,118,177]   117 33538'376   2015-03-12 
13:51:03.984454  28394'622015-03-11 13:50:58.196288


[root@pouta-s04 current]# ceph pg map 3.7d0
osdmap e262813 pg 3.7d0 (3.7d0) - up [117,118,177] acting [117,118,177]
[root@pouta-s04 current]#


Data is present here , so 1 copy is present out of 3 

[root@pouta-s04 current]# ls -l /var/lib/ceph/osd/ceph-117/current/3.7d0_head/ 
| wc -l
63
[root@pouta-s04 current]#



[root@pouta-s03 ~]#  ls -l /var/lib/ceph/osd/ceph-118/current/3.7d0_head/
total 0
[root@pouta-s03 ~]#


[root@pouta-s01 ceph]# ceph osd find 177
{ osd: 177,
  ip: 10.100.50.2:7062\/99,
  crush_location: { host: pouta-s02,
  root: default”}}
[root@pouta-s01 ceph]#

Even directory is not present here 

[root@pouta-s02 ~]#  ls -l /var/lib/ceph/osd/ceph-177/current/3.7d0_head/
ls: cannot access /var/lib/ceph/osd/ceph-177/current/3.7d0_head/: No such file 
or directory
[root@pouta-s02 ~]#


# ceph pg  3.7d0 query http://paste.ubuntu.com/10720107/ 
http://paste.ubuntu.com/10720107/


- Karan -

 On 20 Mar 2015, at 22:43, Craig Lewis cle...@centraldesktop.com wrote:
 
  osdmap e261536: 239 osds: 239 up, 238 in
 
 Why is that last OSD not IN?  The history you need is probably there.
 
 Run  ceph pg pgid query on some of the stuck PGs.  Look for the 
 recovery_state section.  That should tell you what Ceph needs to complete the 
 recovery.
 
 
 If you need more help, post the output of a couple pg queries.
 
 
 
 On Fri, Mar 20, 2015 at 4:22 AM, Karan Singh karan.si...@csc.fi 
 mailto:karan.si...@csc.fi wrote:
 Hello Guys
 
 My CEPH cluster lost data and not its not recovering. This problem occurred 
 when Ceph performed recovery when one of the node was down. 
 Now all the nodes are up but Ceph is showing PG as incomplete , unclean , 
 recovering.
 
 
 I have tried several things to recover them

Re: [ceph-users] Migrating objects from one pool to another?

[Karan Singh]

Hi Jean

You would probably need this

ceph osd pool create glance-images-bkp 128 128
rados cppool glance-images glance-images-bkp
ceph osd pool rename glance-images glance-images-old
ceph osd pool rename glance-images-bkp glance-images
ceph osd pool delete glance-images-old glance-images-old 
--yes-i-really-really-mean-it  ( once you are sure data is moved 100% )

I would suggest to stop openstack services that are using the original pool , 
then copy the data , rename pools , finally start openstack services and check 
everything is there.

I have done this once with success.



Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


 On 27 Mar 2015, at 00:01, Gregory Farnum g...@gregs42.com wrote:
 
 On Thu, Mar 26, 2015 at 2:53 PM, Steffen W Sørensen ste...@me.com wrote:
 
 On 26/03/2015, at 21.07, J-P Methot jpmet...@gtcomm.net wrote:
 
 That's a great idea. I know I can setup cinder (the openstack volume 
 manager) as a multi-backend manager and migrate from one backend to the 
 other, each backend linking to different pools of the same ceph cluster. 
 What bugs me though is that I'm pretty sure the image store, glance, 
 wouldn't let me do that. Additionally, since the compute component also has 
 its own ceph pool, I'm pretty sure it won't let me migrate the data through 
 openstack.
 Hm wouldn’t it be possible to do something similar ala:
 
 # list object from src pool
 rados ls objects loop | filter-obj-id | while read obj; do
 # export $obj to local disk
 rados -p pool-wth-too-many-pgs get $obj
 # import $obj from local disk to new pool
 rados -p better-sized-pool put $obj
 done
 
 You would also have issues with snapshots if you do this on an RBD
 pool. That's unfortunately not feasible.
 -Greg
 
 
 
 possible split/partition list of objects into multiple concurrent loops, 
 possible from multiple boxes as seems fit for resources at hand, cpu, 
 memory, network, ceph perf.
 
 /Steffen
 
 
 
 
 On 3/26/2015 3:54 PM, Steffen W Sørensen wrote:
 On 26/03/2015, at 20.38, J-P Methot jpmet...@gtcomm.net wrote:
 
 Lately I've been going back to work on one of my first ceph setup and now 
 I see that I have created way too many placement groups for the pools on 
 that setup (about 10 000 too many). I believe this may impact 
 performances negatively, as the performances on this ceph cluster are 
 abysmal. Since it is not possible to reduce the number of PGs in a pool, 
 I was thinking of creating new pools with a smaller number of PGs, moving 
 the data from the old pools to the new pools and then deleting the old 
 pools.
 
 I haven't seen any command to copy objects from one pool to another. 
 Would that be possible? I'm using ceph for block storage with openstack, 
 so surely there must be a way to move block devices from a pool to 
 another, right?
 What I did a one point was going one layer higher in my storage 
 abstraction, and created new Ceph pools and used those for new storage 
 resources/pool in my VM env. (ProxMox) on top of Ceph RBD and then did a 
 live migration of virtual disks there, assume you could do the same in 
 OpenStack.
 
 My 0.02$
 
 /Steffen
 
 
 --
 ==
 Jean-Philippe Méthot
 Administrateur système / System administrator
 GloboTech Communications
 Phone: 1-514-907-0050
 Toll Free: 1-(888)-GTCOMM1
 Fax: 1-(514)-907-0750
 jpmet...@gtcomm.net
 http://www.gtcomm.net
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Production Ceph :: PG data lost : Cluster PG incomplete, inactive, unclean

[Karan Singh]

Hello Guys

My CEPH cluster lost data and not its not recovering. This problem occurred 
when Ceph performed recovery when one of the node was down. 
Now all the nodes are up but Ceph is showing PG as incomplete , unclean , 
recovering.


I have tried several things to recover them like , scrub , deep-scrub , pg 
repair , try changing primary affinity and then scrubbing , 
osd_pool_default_size etc. BUT NO LUCK

Could yo please advice , how to recover PG and achieve HEALTH_OK

# ceph -s
cluster 2bd3283d-67ef-4316-8b7e-d8f4747eae33
 health HEALTH_WARN 19 pgs incomplete; 3 pgs recovering; 20 pgs stuck 
inactive; 23 pgs stuck unclean; 2 requests are blocked  32 sec; recovery 
531/980676 objects degraded (0.054%); 243/326892 unfound (0.074%)
 monmap e3: 3 mons at 
{xxx=:6789/0,xxx=:6789:6789/0,xxx=:6789:6789/0}, election epoch 
1474, quorum 0,1,2 xx,xx,xx
 osdmap e261536: 239 osds: 239 up, 238 in
  pgmap v415790: 18432 pgs, 13 pools, 2330 GB data, 319 kobjects
20316 GB used, 844 TB / 864 TB avail
531/980676 objects degraded (0.054%); 243/326892 unfound (0.074%)
   1 creating
   18409 active+clean
   3 active+recovering
  19 incomplete




# ceph pg dump_stuck unclean
ok
pg_stat objects mip degrunf bytes   log disklog state   
state_stamp v   reportedup  up_primary  acting  
acting_primary  last_scrub  scrub_stamp last_deep_scrub deep_scrub_stamp
10.70   0   0   0   0   0   0   0   incomplete  
2015-03-20 12:19:49.534911  0'0 261536:1015 [153,140,80]153 
[153,140,80]153 0'0 2015-03-12 17:59:43.275049  0'0 
2015-03-09 17:55:58.745662
3.dde   68  66  0   66  552861709   297 297 
incomplete  2015-03-20 12:19:49.584839  33547'297   261536:228352   
[174,5,179] 174 [174,5,179] 174 33547'297   2015-03-12 
14:19:15.261595  28522'432015-03-11 14:19:13.894538
5.a20   0   0   0   0   0   0   incomplete  
2015-03-20 12:19:49.560756  0'0 261536:897  [214,191,170]   214 
[214,191,170]   214 0'0 2015-03-12 17:58:29.257085  0'0 
2015-03-09 17:55:07.684377
13.1b6  0   0   0   0   0   0   0   incomplete  
2015-03-20 12:19:49.846253  0'0 261536:1050 [0,176,131] 0   
[0,176,131] 0   0'0 2015-03-12 18:00:13.286920  0'0 
2015-03-09 17:56:18.715208
7.25b   16  0   0   0   6710886416  16  
incomplete  2015-03-20 12:19:49.639102  27666'16261536:4777 
[194,145,45]194 [194,145,45]194 27666'162015-03-12 
17:59:06.357864  2330'3  2015-03-09 17:55:30.754522
5.190   0   0   0   0   0   0   incomplete  
2015-03-20 12:19:49.742698  0'0 261536:25410[212,43,131]212 
[212,43,131]212 0'0 2015-03-12 13:51:37.777026  0'0 
2015-03-11 13:51:35.406246
3.a2f   0   0   0   0   0   0   0   creating
2015-03-20 12:42:15.586372  0'0 0:0 []  -1  []  -1  
0'0 0.000'0 0.00
7.298   0   0   0   0   0   0   0   incomplete  
2015-03-20 12:19:49.566966  0'0 261536:900  [187,95,225]187 
[187,95,225]187 27666'132015-03-12 17:59:10.308423  2330'4  
2015-03-09 17:55:35.750109
3.a5a   77  87  261 87  623902741   325 325 
active+recovering   2015-03-20 10:54:57.443670  33569'325   
261536:182464   [150,149,181]   150 [150,149,181]   150 33569'325   
2015-03-12 13:58:05.813966  28433'442015-03-11 13:57:53.909795
1.1e7   0   0   0   0   0   0   0   incomplete  
2015-03-20 12:19:49.610547  0'0 261536:772  [175,182]   175 
[175,182]   175 0'0 2015-03-12 17:55:45.203232  0'0 
2015-03-09 17:53:49.694822
3.774   79  0   0   0   645136397   339 339 
incomplete  2015-03-20 12:19:49.821708  33570'339   261536:166857   
[162,39,161]162 [162,39,161]162 33570'339   2015-03-12 
14:49:03.869447  2226'2  2015-03-09 13:46:49.783950
3.7d0   78  0   0   0   609222686   376 376 
incomplete  2015-03-20 12:19:49.534004  33538'376   261536:182810   
[117,118,177]   117 [117,118,177]   117 33538'376   2015-03-12 
13:51:03.984454  28394'622015-03-11 13:50:58.196288
3.d60   0   0   0   0   0   0   0   incomplete  
2015-03-20 12:19:49.647196  0'0 261536:833  [154,172,1] 154 
[154,172,1] 154 33552'321   2015-03-12 

[ceph-users] Production Ceph :: PG data lost : Cluster PG incomplete, inactive, unclean

[Karan Singh]

Hello Guys

My CEPH cluster lost data and not its not recovering. This problem occurred 
when Ceph performed recovery when one of the node was down. 
Now all the nodes are up but Ceph is showing PG as incomplete , unclean , 
recovering.


I have tried several things to recover them like , scrub , deep-scrub , pg 
repair , try changing primary affinity and then scrubbing , 
osd_pool_default_size etc. BUT NO LUCK

Could yo please advice , how to recover PG and achieve HEALTH_OK

# ceph -s
cluster 2bd3283d-67ef-4316-8b7e-d8f4747eae33
 health HEALTH_WARN 19 pgs incomplete; 3 pgs recovering; 20 pgs stuck 
inactive; 23 pgs stuck unclean; 2 requests are blocked  32 sec; recovery 
531/980676 objects degraded (0.054%); 243/326892 unfound (0.074%)
 monmap e3: 3 mons at 
{xxx=:6789/0,xxx=:6789:6789/0,xxx=:6789:6789/0}, election epoch 
1474, quorum 0,1,2 xx,xx,xx
 osdmap e261536: 239 osds: 239 up, 238 in
  pgmap v415790: 18432 pgs, 13 pools, 2330 GB data, 319 kobjects
20316 GB used, 844 TB / 864 TB avail
531/980676 objects degraded (0.054%); 243/326892 unfound (0.074%)
   1 creating
   18409 active+clean
   3 active+recovering
  19 incomplete




# ceph pg dump_stuck unclean
ok
pg_stat objects mip degrunf bytes   log disklog state   
state_stamp v   reportedup  up_primary  acting  
acting_primary  last_scrub  scrub_stamp last_deep_scrub deep_scrub_stamp
10.70   0   0   0   0   0   0   0   incomplete  
2015-03-20 12:19:49.534911  0'0 261536:1015 [153,140,80]153 
[153,140,80]153 0'0 2015-03-12 17:59:43.275049  0'0 
2015-03-09 17:55:58.745662
3.dde   68  66  0   66  552861709   297 297 
incomplete  2015-03-20 12:19:49.584839  33547'297   261536:228352   
[174,5,179] 174 [174,5,179] 174 33547'297   2015-03-12 
14:19:15.261595  28522'432015-03-11 14:19:13.894538
5.a20   0   0   0   0   0   0   incomplete  
2015-03-20 12:19:49.560756  0'0 261536:897  [214,191,170]   214 
[214,191,170]   214 0'0 2015-03-12 17:58:29.257085  0'0 
2015-03-09 17:55:07.684377
13.1b6  0   0   0   0   0   0   0   incomplete  
2015-03-20 12:19:49.846253  0'0 261536:1050 [0,176,131] 0   
[0,176,131] 0   0'0 2015-03-12 18:00:13.286920  0'0 
2015-03-09 17:56:18.715208
7.25b   16  0   0   0   6710886416  16  
incomplete  2015-03-20 12:19:49.639102  27666'16261536:4777 
[194,145,45]194 [194,145,45]194 27666'162015-03-12 
17:59:06.357864  2330'3  2015-03-09 17:55:30.754522
5.190   0   0   0   0   0   0   incomplete  
2015-03-20 12:19:49.742698  0'0 261536:25410[212,43,131]212 
[212,43,131]212 0'0 2015-03-12 13:51:37.777026  0'0 
2015-03-11 13:51:35.406246
3.a2f   0   0   0   0   0   0   0   creating
2015-03-20 12:42:15.586372  0'0 0:0 []  -1  []  -1  
0'0 0.000'0 0.00
7.298   0   0   0   0   0   0   0   incomplete  
2015-03-20 12:19:49.566966  0'0 261536:900  [187,95,225]187 
[187,95,225]187 27666'132015-03-12 17:59:10.308423  2330'4  
2015-03-09 17:55:35.750109
3.a5a   77  87  261 87  623902741   325 325 
active+recovering   2015-03-20 10:54:57.443670  33569'325   
261536:182464   [150,149,181]   150 [150,149,181]   150 33569'325   
2015-03-12 13:58:05.813966  28433'442015-03-11 13:57:53.909795
1.1e7   0   0   0   0   0   0   0   incomplete  
2015-03-20 12:19:49.610547  0'0 261536:772  [175,182]   175 
[175,182]   175 0'0 2015-03-12 17:55:45.203232  0'0 
2015-03-09 17:53:49.694822
3.774   79  0   0   0   645136397   339 339 
incomplete  2015-03-20 12:19:49.821708  33570'339   261536:166857   
[162,39,161]162 [162,39,161]162 33570'339   2015-03-12 
14:49:03.869447  2226'2  2015-03-09 13:46:49.783950
3.7d0   78  0   0   0   609222686   376 376 
incomplete  2015-03-20 12:19:49.534004  33538'376   261536:182810   
[117,118,177]   117 [117,118,177]   117 33538'376   2015-03-12 
13:51:03.984454  28394'622015-03-11 13:50:58.196288
3.d60   0   0   0   0   0   0   0   incomplete  
2015-03-20 12:19:49.647196  0'0 261536:833  [154,172,1] 154 
[154,172,1] 154 33552'321   2015-03-12 

Re: [ceph-users] ceph days

[Karan Singh]

Check out ceph youtube page.

- Karan -

 On 11 Mar 2015, at 00:45, Tom Deneau tom.den...@amd.com wrote:
 
 Are the slides or videos from ceph days presentations made available
 somewhere?  I noticed some links in the Frankfurt Ceph day, but not for the
 other Ceph Days.
 
 -- Tom
 
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph BIG outage : 200+ OSD are down , OSD cannot create thread

[Karan Singh]

Thanks Sage

I will create a “new feature” request on tracker.ceph.com 
http://tracker.ceph.com/ so that this discussion should not get buried under 
mailing list. 

Developers can implement this as per their convenience.



Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


 On 10 Mar 2015, at 14:26, Sage Weil s...@newdream.net wrote:
 
 On Tue, 10 Mar 2015, Christian Eichelmann wrote:
 Hi Sage,
 
 we hit this problem a few monthes ago as well and it took us quite a while to
 figure out what's wrong.
 
 As a Systemadministrator I don't like the idea that daemons or even init
 scripts are changing system wide configuration parameters, so I wouldn't like
 to see the OSDs do it themself.
 
 This is my general feeling as well.  As we move to systemd, I'd like to 
 have the ceph unit file get away from this entirely and have the admin set 
 these values in /etc/security/limits.conf or /etc/sysctl.d.  The main 
 thing making this problematic right now is that the daemons run as root 
 instead of a 'ceph' user.
 
 The idea with the warning is on one hand a good hint, on the other hand it
 also may confuse people, since changing this setting is not required for
 common hardware.
 
 If we make it warn only if it reaches  50% of the threshold that is 
 probably safe...
 
 sage
 
 
 
 Regards,
 Christian
 
 On 03/09/2015 08:01 PM, Sage Weil wrote:
 On Mon, 9 Mar 2015, Karan Singh wrote:
 Thanks Guys kernel.pid_max=4194303 did the trick.
 Great to hear!  Sorry we missed that you only had it at 65536.
 
 This is a really common problem that people hit when their clusters start
 to grow.  Is there somewhere in the docs we can put this to catch more
 users?  Or maybe a warning issued by the osds themselves or something if
 they see limits that are low?
 
 sage
 
 - Karan -
 
   On 09 Mar 2015, at 14:48, Christian Eichelmann
   christian.eichelm...@1und1.de wrote:
 
 Hi Karan,
 
 as you are actually writing in your own book, the problem is the
 sysctl
 setting kernel.pid_max. I've seen in your bug report that you were
 setting it to 65536, which is still to low for high density hardware.
 
 In our cluster, one OSD server has in an idle situation about 66.000
 Threads (60 OSDs per Server). The number of threads increases when you
 increase the number of placement groups in the cluster, which I think
 has triggered your problem.
 
 Set the kernel.pid_max setting to 4194303 (the maximum) like Azad
 Aliyar suggested, and the problem should be gone.
 
 Regards,
 Christian
 
 Am 09.03.2015 11:41, schrieb Karan Singh:
   Hello Community need help to fix a long going Ceph
   problem.
 
   Cluster is unhealthy , Multiple OSDs are DOWN. When i am
   trying to
   restart OSD?s i am getting this error
 
 
   /2015-03-09 12:22:16.312774 7f760dac9700 -1
   common/Thread.cc
   http://Thread.cc: In function 'void
   Thread::create(size_t)' thread
   7f760dac9700 time 2015-03-09 12:22:16.311970/
   /common/Thread.cc http://Thread.cc: 129: FAILED
   assert(ret == 0)/
 
 
   *Environment *:  4 Nodes , OSD+Monitor , Firefly latest ,
   CentOS6.5
   , 3.17.2-1.el6.elrepo.x86_64
 
   Tried upgrading from 0.80.7 to 0.80.8  but no Luck
 
   Tried centOS stock kernel 2.6.32  but no Luck
 
   Memory is not a problem more then 150+GB is free
 
 
   Did any one every faced this problem ??
 
   *Cluster status *
   *
   *
   / cluster 2bd3283d-67ef-4316-8b7e-d8f4747eae33/
   / health HEALTH_WARN 7334 pgs degraded; 1185 pgs down;
   1 pgs
   incomplete; 1735 pgs peering; 8938 pgs stale; 1/
   /736 pgs stuck inactive; 8938 pgs stuck stale; 10320 pgs
   stuck unclean;
   recovery 6061/31080 objects degraded (19/
   /.501%); 111/196 in osds are down; clock skew detected on
   mon.pouta-s02,
   mon.pouta-s03/
   / monmap e3: 3 mons at
 {pouta-s01=10.XXX.50.1:6789/0,pouta-s02=10.XXX.50.2:6789/0,pouta-s03=10.XXX
   .50.3:6789/
   //0}, election epoch 1312, quorum 0,1,2
   pouta-s01,pouta-s02,pouta-s03/
   /   * osdmap e26633: 239 osds: 85 up, 196 in*/
   /  pgmap v60389: 17408 pgs, 13 pools, 42345 MB data,
   10360 objects/
   /4699 GB used, 707 TB / 711 TB avail/
   /6061/31080 objects degraded (19.501%)/
   /  14 down+remapped+peering/
   /  39 active/
   /3289 active+clean/
   / 547 peering/
   / 663 stale+down+peering/
   / 705 stale+active+remapped/
   /   1 active+degraded+remapped/
   /   1 stale

Re: [ceph-users] Ceph BIG outage : 200+ OSD are down , OSD cannot create thread

[Karan Singh]

Thanks Guys kernel.pid_max=4194303 did the trick.

- Karan -

 On 09 Mar 2015, at 14:48, Christian Eichelmann 
 christian.eichelm...@1und1.de wrote:
 
 Hi Karan,
 
 as you are actually writing in your own book, the problem is the sysctl
 setting kernel.pid_max. I've seen in your bug report that you were
 setting it to 65536, which is still to low for high density hardware.
 
 In our cluster, one OSD server has in an idle situation about 66.000
 Threads (60 OSDs per Server). The number of threads increases when you
 increase the number of placement groups in the cluster, which I think
 has triggered your problem.
 
 Set the kernel.pid_max setting to 4194303 (the maximum) like Azad
 Aliyar suggested, and the problem should be gone.
 
 Regards,
 Christian
 
 Am 09.03.2015 11:41, schrieb Karan Singh:
 Hello Community need help to fix a long going Ceph problem.
 
 Cluster is unhealthy , Multiple OSDs are DOWN. When i am trying to
 restart OSD’s i am getting this error 
 
 
 /2015-03-09 12:22:16.312774 7f760dac9700 -1 common/Thread.cc
 http://Thread.cc: In function 'void Thread::create(size_t)' thread
 7f760dac9700 time 2015-03-09 12:22:16.311970/
 /common/Thread.cc http://Thread.cc: 129: FAILED assert(ret == 0)/
 
 
 *Environment *:  4 Nodes , OSD+Monitor , Firefly latest , CentOS6.5
 , 3.17.2-1.el6.elrepo.x86_64
 
 Tried upgrading from 0.80.7 to 0.80.8  but no Luck
 
 Tried centOS stock kernel 2.6.32  but no Luck
 
 Memory is not a problem more then 150+GB is free 
 
 
 Did any one every faced this problem ??
 
 *Cluster status *
 *
 *
 / cluster 2bd3283d-67ef-4316-8b7e-d8f4747eae33/
 / health HEALTH_WARN 7334 pgs degraded; 1185 pgs down; 1 pgs
 incomplete; 1735 pgs peering; 8938 pgs stale; 1/
 /736 pgs stuck inactive; 8938 pgs stuck stale; 10320 pgs stuck unclean;
 recovery 6061/31080 objects degraded (19/
 /.501%); 111/196 in osds are down; clock skew detected on mon.pouta-s02,
 mon.pouta-s03/
 / monmap e3: 3 mons at
 {pouta-s01=10.XXX.50.1:6789/0,pouta-s02=10.XXX.50.2:6789/0,pouta-s03=10.XXX.50.3:6789/
 //0}, election epoch 1312, quorum 0,1,2 pouta-s01,pouta-s02,pouta-s03/
 /   * osdmap e26633: 239 osds: 85 up, 196 in*/
 /  pgmap v60389: 17408 pgs, 13 pools, 42345 MB data, 10360 objects/
 /4699 GB used, 707 TB / 711 TB avail/
 /6061/31080 objects degraded (19.501%)/
 /  14 down+remapped+peering/
 /  39 active/
 /3289 active+clean/
 / 547 peering/
 / 663 stale+down+peering/
 / 705 stale+active+remapped/
 /   1 active+degraded+remapped/
 /   1 stale+down+incomplete/
 / 484 down+peering/
 / 455 active+remapped/
 /3696 stale+active+degraded/
 /   4 remapped+peering/
 /  23 stale+down+remapped+peering/
 /  51 stale+active/
 /3637 active+degraded/
 /3799 stale+active+clean/
 
 *OSD :  Logs *
 
 /2015-03-09 12:22:16.312774 7f760dac9700 -1 common/Thread.cc
 http://Thread.cc: In function 'void Thread::create(size_t)' thread
 7f760dac9700 time 2015-03-09 12:22:16.311970/
 /common/Thread.cc http://Thread.cc: 129: FAILED assert(ret == 0)/
 /
 /
 / ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)/
 / 1: (Thread::create(unsigned long)+0x8a) [0xaf41da]/
 / 2: (SimpleMessenger::add_accept_pipe(int)+0x6a) [0xae84fa]/
 / 3: (Accepter::entry()+0x265) [0xb5c635]/
 / 4: /lib64/libpthread.so.0() [0x3c8a6079d1]/
 / 5: (clone()+0x6d) [0x3c8a2e89dd]/
 / NOTE: a copy of the executable, or `objdump -rdS executable` is
 needed to interpret this./
 
 
 *More information at Ceph Tracker Issue :
 *http://tracker.ceph.com/issues/10988#change-49018
 
 
 
 Karan Singh 
 Systems Specialist , Storage Platforms
 CSC - IT Center for Science,
 Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
 mobile: +358 503 812758
 tel. +358 9 4572001
 fax +358 9 4572302
 http://www.csc.fi/
 
 
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 
 -- 
 Christian Eichelmann
 Systemadministrator
 
 11 Internet AG - IT Operations Mail  Media Advertising  Targeting
 Brauerstraße 48 · DE-76135 Karlsruhe
 Telefon: +49 721 91374-8026
 christian.eichelm...@1und1.de
 
 Amtsgericht Montabaur / HRB 6484
 Vorstände: Henning Ahlert, Ralph Dommermuth, Matthias Ehrlich, Robert
 Hoffmann, Markus Huhn, Hans-Henning Kettler, Dr. Oliver Mauss, Jan Oetjen
 Aufsichtsratsvorsitzender: Michael Scheeren



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph BIG outage : 200+ OSD are down , OSD cannot create thread

[Karan Singh]

Hello Community need help to fix a long going Ceph problem.

Cluster is unhealthy , Multiple OSDs are DOWN. When i am trying to restart 
OSD’s i am getting this error 


2015-03-09 12:22:16.312774 7f760dac9700 -1 common/Thread.cc: In function 'void 
Thread::create(size_t)' thread 7f760dac9700 time 2015-03-09 12:22:16.311970
common/Thread.cc: 129: FAILED assert(ret == 0)


Environment :  4 Nodes , OSD+Monitor , Firefly latest , CentOS6.5 , 
3.17.2-1.el6.elrepo.x86_64

Tried upgrading from 0.80.7 to 0.80.8  but no Luck

Tried centOS stock kernel 2.6.32  but no Luck

Memory is not a problem more then 150+GB is free 


Did any one every faced this problem ??

Cluster status 

   cluster 2bd3283d-67ef-4316-8b7e-d8f4747eae33
 health HEALTH_WARN 7334 pgs degraded; 1185 pgs down; 1 pgs incomplete; 
1735 pgs peering; 8938 pgs stale; 1
736 pgs stuck inactive; 8938 pgs stuck stale; 10320 pgs stuck unclean; recovery 
6061/31080 objects degraded (19
.501%); 111/196 in osds are down; clock skew detected on mon.pouta-s02, 
mon.pouta-s03
 monmap e3: 3 mons at 
{pouta-s01=10.XXX.50.1:6789/0,pouta-s02=10.XXX.50.2:6789/0,pouta-s03=10.XXX.50.3:6789
/0}, election epoch 1312, quorum 0,1,2 pouta-s01,pouta-s02,pouta-s03
 osdmap e26633: 239 osds: 85 up, 196 in
  pgmap v60389: 17408 pgs, 13 pools, 42345 MB data, 10360 objects
4699 GB used, 707 TB / 711 TB avail
6061/31080 objects degraded (19.501%)
  14 down+remapped+peering
  39 active
3289 active+clean
 547 peering
 663 stale+down+peering
 705 stale+active+remapped
   1 active+degraded+remapped
   1 stale+down+incomplete
 484 down+peering
 455 active+remapped
3696 stale+active+degraded
   4 remapped+peering
  23 stale+down+remapped+peering
  51 stale+active
3637 active+degraded
3799 stale+active+clean

OSD :  Logs 

2015-03-09 12:22:16.312774 7f760dac9700 -1 common/Thread.cc: In function 'void 
Thread::create(size_t)' thread 7f760dac9700 time 2015-03-09 12:22:16.311970
common/Thread.cc: 129: FAILED assert(ret == 0)

 ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)
 1: (Thread::create(unsigned long)+0x8a) [0xaf41da]
 2: (SimpleMessenger::add_accept_pipe(int)+0x6a) [0xae84fa]
 3: (Accepter::entry()+0x265) [0xb5c635]
 4: /lib64/libpthread.so.0() [0x3c8a6079d1]
 5: (clone()+0x6d) [0x3c8a2e89dd]
 NOTE: a copy of the executable, or `objdump -rdS executable` is needed to 
interpret this.


More information at Ceph Tracker Issue :  
http://tracker.ceph.com/issues/10988#change-49018 
http://tracker.ceph.com/issues/10988#change-49018



Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/




smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

[Karan Singh]

Thanks Mark , for a superb explanation . This is indeed very useful.


Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


 On 17 Feb 2015, at 22:16, Stephen Hindle shin...@llnw.com wrote:
 
 I was wondering what the 'CBT' tool is ?  Google is useless for that 
 acronym...
 
 Thanks!
 Steve
 
 On Tue, Feb 17, 2015 at 10:37 AM, Mark Nelson mnel...@redhat.com wrote:
 Hi All,
 
 I wrote up a short document describing some tests I ran recently to look at
 how SSD backed OSD performance has changed across our LTS releases. This is
 just looking at RADOS performance and not RBD or RGW.  It also doesn't offer
 any real explanations regarding the results.  It's just a first high level
 step toward understanding some of the behaviors folks on the mailing list
 have reported over the last couple of releases.  I hope you find it useful.
 
 Mark
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 -- 
 The information in this message may be confidential.  It is intended solely 
 for
 the addressee(s).  If you are not the intended recipient, any disclosure,
 copying or distribution of the message, or any action or omission taken by 
 you
 in reliance on it, is prohibited and may be unlawful.  Please immediately
 contact the sender if you have received this message in error.
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Introducing Learning Ceph : The First ever Book on Ceph

[Karan Singh]

Here is the new link for sample book : 
https://www.dropbox.com/s/2zcxawtv4q29fm9/Learning_Ceph_Sample.pdf?dl=0 
https://www.dropbox.com/s/2zcxawtv4q29fm9/Learning_Ceph_Sample.pdf?dl=0



Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


 On 13 Feb 2015, at 05:25, Frank Yu flyxia...@gmail.com wrote:
 
 Wow, Cong
 BTW, I found the link of sample copy is 404.
 
 
 
 2015-02-06 6:53 GMT+08:00 Karan Singh karan.si...@csc.fi 
 mailto:karan.si...@csc.fi:
 Hello Community Members
 
 I am happy to introduce the first book on Ceph with the title “Learning 
 Ceph”. 
 
 Me and many folks from the publishing house together with technical reviewers 
 spent several months to get this book compiled and published.
 
 Finally the book is up for sale on , i hope you would like it and surely will 
 learn a lot from it.
 
 Amazon :  
 http://www.amazon.com/Learning-Ceph-Karan-Singh/dp/1783985623/ref=sr_1_1?s=booksie=UTF8qid=1423174441sr=1-1keywords=ceph
  
 http://www.amazon.com/Learning-Ceph-Karan-Singh/dp/1783985623/ref=sr_1_1?s=booksie=UTF8qid=1423174441sr=1-1keywords=ceph
 Packtpub : https://www.packtpub.com/application-development/learning-ceph 
 https://www.packtpub.com/application-development/learning-ceph
 
 You can grab the sample copy from here :  
 https://www.dropbox.com/s/ek76r01r9prs6pb/Learning_Ceph_Packt.pdf?dl=0 
 https://www.dropbox.com/s/ek76r01r9prs6pb/Learning_Ceph_Packt.pdf?dl=0
 
 Finally , I would like to express my sincere thanks to 
 
 Sage Weil - For developing Ceph and everything around it as well as writing 
 foreword for “Learning Ceph”.
 Patrick McGarry - For his usual off the track support that too always.
 
 Last but not the least , to our great community members , who are also 
 reviewers of the book Don Talton , Julien Recurt , Sebastien Han and Zihong 
 Chen , Thank you guys for your efforts.
 
 
 
 Karan Singh 
 Systems Specialist , Storage Platforms
 CSC - IT Center for Science,
 Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
 mobile: +358 503 812758
 tel. +358 9 4572001 tel:%2B358%209%204572001
 fax +358 9 4572302 tel:%2B358%209%204572302
 http://www.csc.fi/ http://www.csc.fi/
 
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 
 
 -- 
 Regards
 Frank Yu



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Introducing Learning Ceph : The First ever Book on Ceph

[Karan Singh]

Hello Community Members

I am happy to introduce the first book on Ceph with the title “Learning Ceph”. 

Me and many folks from the publishing house together with technical reviewers 
spent several months to get this book compiled and published.

Finally the book is up for sale on , i hope you would like it and surely will 
learn a lot from it.

Amazon :  
http://www.amazon.com/Learning-Ceph-Karan-Singh/dp/1783985623/ref=sr_1_1?s=booksie=UTF8qid=1423174441sr=1-1keywords=ceph
 
http://www.amazon.com/Learning-Ceph-Karan-Singh/dp/1783985623/ref=sr_1_1?s=booksie=UTF8qid=1423174441sr=1-1keywords=ceph
Packtpub : https://www.packtpub.com/application-development/learning-ceph 
https://www.packtpub.com/application-development/learning-ceph

You can grab the sample copy from here :  
https://www.dropbox.com/s/ek76r01r9prs6pb/Learning_Ceph_Packt.pdf?dl=0 
https://www.dropbox.com/s/ek76r01r9prs6pb/Learning_Ceph_Packt.pdf?dl=0

Finally , I would like to express my sincere thanks to 

Sage Weil - For developing Ceph and everything around it as well as writing 
foreword for “Learning Ceph”.
Patrick McGarry - For his usual off the track support that too always.

Last but not the least , to our great community members , who are also 
reviewers of the book Don Talton , Julien Recurt , Sebastien Han and Zihong 
Chen , Thank you guys for your efforts.



Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/ http://www.csc.fi/




smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help with Integrating Ceph with various Cloud Storage

[Karan Singh]

Hello Manoj

My answers to your queries.

# For testing purpose you can install Ceph on virtual machines ( multiple 
instances of virtual box for multiple mon, osd ). Its good to practice Ceph 
with multiple MON and OSD.
# For real data storage , please use physical servers , virtual servers are 
good to go only for testing and learning purpose.

# S3,azure blob,swift are object storage which are different from each other , 
similarly Ceph also supports object storage and you can access Ceph object 
storage with S3 and Swift protocols. You cannot use Ceph storage with 
amazon,azure but you can use ceph with Swift and s3 API.

# I am not much familiar with cloud storage , but based on the experience of 
openstack , libvirt should be installed from cloudstack servers only , Ceph 
servers should only do storage for you.


Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


On 17 Dec 2014, at 09:57, Manoj Singh respond2ma...@gmail.com wrote:

 Hi All,
 
 I am new to Ceph. Due to physical machines shortage I have installed Ceph 
 cluster with single OSD and MON in a single Virtual Machine.  
 
 I have few queries as below:
 1.  Whether having the Ceph setup on a VM is fine or it require to be on 
 Physical server.
 
 2. Since Amazon S3, Azure Blob Storage, Swift are Object based Storage, what 
 is the feasibility of attaching these Cloud Storage to Ceph and to be able to 
 allocate disc space from the same while creating new VM from local CloudStack 
 or OpenStack
 
 3. When I am integrating CloudStack with Ceph whether libvert should be 
 installed on the CloudStack management server or on Ceph server. From diagram 
 given in Ceph documentation it's bit confusing.
 
 Thank you in advance. your help shall be really appreciated.
 
 Best Regards,
 Manoj Kumar
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] can not add osd

[Karan Singh]

Hi

You logs does not provides much information , if you are following any other 
documentation for Ceph , i would recommend you to follow official Ceph docs.

http://ceph.com/docs/master/start/quick-start-preflight/




Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


On 16 Dec 2014, at 09:55, yang.bi...@zte.com.cn wrote:

 hi 
 
 When i execute ceph-deploy osd prepare node3:/dev/sdb,always come out err 
 like this : 
 
 [node3][WARNIN] INFO:ceph-disk:Running command: /bin/umount -- 
 /var/lib/ceph/tmp/mnt.u2KXW3 
 [node3][WARNIN] umount: /var/lib/ceph/tmp/mnt.u2KXW3: target is busy. 
 
 Then i execute /bin/umount -- /var/lib/ceph/tmp/mnt.u2KXW3,result is ok. 
 
 
 ZTE Information Security Notice: The information contained in this mail (and 
 any attachment transmitted herewith) is privileged and confidential and is 
 intended for the exclusive use of the addressee(s).  If you are not an 
 intended recipient, any disclosure, reproduction, distribution or other 
 dissemination or use of the information contained is strictly prohibited.  If 
 you have received this mail in error, please delete it and notify us 
 immediately.
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cache tier unable to auto flush data to storage tier

[Karan Singh]

Thanks JC  , it worked , now cache tiering agent is migrating data between 
tiers.


But Now , i am seeing a new ISSUE :  Cache-pool has got some EXTRA objects , 
that is not visible with # rados -p cache-pool ls but under #ceph df i can see 
the count of those objects.

[root@ceph-node1 ~]# ceph df | egrep -i objects|pool
POOLS:
NAME   ID USED  %USED OBJECTS
EC-pool15 1000M 1.21  2
cache-pool 16 252   0 3
[root@ceph-node1 ~]#
[root@ceph-node1 ~]# rados -p cache-pool ls
[root@ceph-node1 ~]# rados -p cache-pool  cache-flush-evict-all
[root@ceph-node1 ~]# rados -p cache-pool ls
[root@ceph-node1 ~]# ceph df | egrep -i objects|pool
POOLS:
NAME   ID USED  %USED OBJECTS
EC-pool15 1000M 1.21  2
cache-pool 16 252   0 3
[root@ceph-node1 ~]#


# Also when i create ONE object manually , #ceph df says that 2 objects has 
been added. From where this extra object coming

[root@ceph-node1 ~]# ceph df | egrep -i objects|pool
POOLS:
NAME   ID USED  %USED OBJECTS
EC-pool15 1000M 1.21  2
cache-pool 16 252   0 3
[root@ceph-node1 ~]#
[root@ceph-node1 ~]#
[root@ceph-node1 ~]# rados -p cache-pool put test /etc/hosts( I have 
added one object in this step )
[root@ceph-node1 ~]# rados -p cache-pool ls 
( when i list i can see only 1 object that i have recently created)
test
[root@ceph-node1 ~]# ceph df | egrep -i objects|pool
POOLS:
NAME   ID USED  %USED OBJECTS
EC-pool15 1000M 1.21  2
cache-pool 16 651   0 5 
(Why it is showing 5 objects  , while earlier its showing 3 
Objects , why it has increased by 2  on adding only 1 object )
[root@ceph-node1 ~]#


- Karan -

On 14 Sep 2014, at 03:42, Jean-Charles LOPEZ jc.lo...@inktank.com wrote:

 Hi Karan,
 
 May be setting the dirty byte ratio (flush) and the full ratio (eviction). 
 Just try to see if it makes any difference
 - cache_target_dirty_ratio .1
 - cache_target_full_ratio .2
 
 Tune the percentage as desired relatively to target_max_bytes and 
 target_max_objects. The first threshold reached will trigger flush or 
 eviction (num objects or num bytes)
 
 JC
 
 
 
 On Sep 13, 2014, at 15:23, Karan Singh karan.si...@csc.fi wrote:
 
 Hello Cephers
 
 I have created a Cache pool and looks like cache tiering agent is not able 
 to flush/evict data as per defined policy. However when i manually evict / 
 flush data , it migrates data from cache-tier to storage-tier
 
 Kindly advice if there is something wrong with policy or anything else i am 
 missing.
 
 Ceph Version: 0.80.5
 OS : Cent OS 6.4
 
 Cache pool created using the following commands :
 
 ceph osd tier add data cache-pool 
 ceph osd tier cache-mode cache-pool writeback
 ceph osd tier set-overlay data cache-pool
 ceph osd pool set cache-pool hit_set_type bloom
 ceph osd pool set cache-pool hit_set_count 1
 ceph osd pool set cache-pool hit_set_period 300
 ceph osd pool set cache-pool target_max_bytes 1
 ceph osd pool set cache-pool target_max_objects 100
 ceph osd pool set cache-pool cache_min_flush_age 60
 ceph osd pool set cache-pool cache_min_evict_age 60
 
 
 [root@ceph-node1 ~]# date
 Sun Sep 14 00:49:59 EEST 2014
 [root@ceph-node1 ~]# rados -p data  put file1 /etc/hosts
 [root@ceph-node1 ~]# rados -p data ls
 [root@ceph-node1 ~]# rados -p cache-pool ls
 file1
 [root@ceph-node1 ~]#
 
 
 [root@ceph-node1 ~]# date
 Sun Sep 14 00:59:33 EEST 2014
 [root@ceph-node1 ~]# rados -p data ls
 [root@ceph-node1 ~]# 
 [root@ceph-node1 ~]# rados -p cache-pool ls
 file1
 [root@ceph-node1 ~]#
 
 
 [root@ceph-node1 ~]# date
 Sun Sep 14 01:08:02 EEST 2014
 [root@ceph-node1 ~]# rados -p data ls
 [root@ceph-node1 ~]# rados -p cache-pool ls
 file1
 [root@ceph-node1 ~]#
 
 
 
 [root@ceph-node1 ~]# rados -p cache-pool  cache-flush-evict-all
 file1
 [root@ceph-node1 ~]#
 [root@ceph-node1 ~]# rados -p data ls
 file1
 [root@ceph-node1 ~]# rados -p cache-pool ls
 [root@ceph-node1 ~]#
 
 
 Regards
 Karan Singh
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Cache tier unable to auto flush data to storage tier

[Karan Singh]

Hello Cephers

I have created a Cache pool and looks like cache tiering agent is not able to 
flush/evict data as per defined policy. However when i manually evict / flush 
data , it migrates data from cache-tier to storage-tier

Kindly advice if there is something wrong with policy or anything else i am 
missing.

Ceph Version: 0.80.5
OS : Cent OS 6.4

Cache pool created using the following commands :

ceph osd tier add data cache-pool 
ceph osd tier cache-mode cache-pool writeback
ceph osd tier set-overlay data cache-pool
ceph osd pool set cache-pool hit_set_type bloom
ceph osd pool set cache-pool hit_set_count 1
ceph osd pool set cache-pool hit_set_period 300
ceph osd pool set cache-pool target_max_bytes 1
ceph osd pool set cache-pool target_max_objects 100
ceph osd pool set cache-pool cache_min_flush_age 60
ceph osd pool set cache-pool cache_min_evict_age 60


[root@ceph-node1 ~]# date
Sun Sep 14 00:49:59 EEST 2014
[root@ceph-node1 ~]# rados -p data  put file1 /etc/hosts
[root@ceph-node1 ~]# rados -p data ls
[root@ceph-node1 ~]# rados -p cache-pool ls
file1
[root@ceph-node1 ~]#


[root@ceph-node1 ~]# date
Sun Sep 14 00:59:33 EEST 2014
[root@ceph-node1 ~]# rados -p data ls
[root@ceph-node1 ~]# 
[root@ceph-node1 ~]# rados -p cache-pool ls
file1
[root@ceph-node1 ~]#


[root@ceph-node1 ~]# date
Sun Sep 14 01:08:02 EEST 2014
[root@ceph-node1 ~]# rados -p data ls
[root@ceph-node1 ~]# rados -p cache-pool ls
file1
[root@ceph-node1 ~]#



[root@ceph-node1 ~]# rados -p cache-pool  cache-flush-evict-all
file1
[root@ceph-node1 ~]#
[root@ceph-node1 ~]# rados -p data ls
file1
[root@ceph-node1 ~]# rados -p cache-pool ls
[root@ceph-node1 ~]#


Regards
Karan Singh

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] HEALTH_WARN 4 pgs incomplete; 4 pgs stuck inactive; 4 pgs stuck unclean

[Karan Singh]

Can you provide your cluster’s ceph osd dump | grep -i pooland crush map 
output. 


- Karan -

On 12 Aug 2014, at 10:40, Riederer, Michael michael.riede...@br.de wrote:

 Hi all,
 
 How do I get my Ceph Cluster back to a healthy state?
 
 root@ceph-admin-storage:~# ceph -v
 ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6)
 root@ceph-admin-storage:~# ceph -s
 cluster 6b481875-8be5-4508-b075-e1f660fd7b33
  health HEALTH_WARN 4 pgs incomplete; 4 pgs stuck inactive; 4 pgs stuck 
 unclean
  monmap e2: 3 mons at 
 {ceph-1-storage=10.65.150.101:6789/0,ceph-2-storage=10.65.150.102:6789/0,ceph-3-storage=10.65.150.103:6789/0},
  election epoch 5010, quorum 0,1,2 
 ceph-1-storage,ceph-2-storage,ceph-3-storage
  osdmap e30748: 55 osds: 55 up, 55 in
   pgmap v10800465: 6144 pgs, 3 pools, 11002 GB data, 2762 kobjects
 22077 GB used, 79933 GB / 102010 GB avail
 6138 active+clean
4 incomplete
2 active+clean+replay
 root@ceph-admin-storage:~# ceph health detail
 HEALTH_WARN 4 pgs incomplete; 4 pgs stuck inactive; 4 pgs stuck unclean
 pg 2.92 is stuck inactive since forever, current state incomplete, last 
 acting [8,13]
 pg 2.c1 is stuck inactive since forever, current state incomplete, last 
 acting [13,7]
 pg 2.e3 is stuck inactive since forever, current state incomplete, last 
 acting [20,7]
 pg 2.587 is stuck inactive since forever, current state incomplete, last 
 acting [13,5]
 pg 2.92 is stuck unclean since forever, current state incomplete, last acting 
 [8,13]
 pg 2.c1 is stuck unclean since forever, current state incomplete, last acting 
 [13,7]
 pg 2.e3 is stuck unclean since forever, current state incomplete, last acting 
 [20,7]
 pg 2.587 is stuck unclean since forever, current state incomplete, last 
 acting [13,5]
 pg 2.587 is incomplete, acting [13,5]
 pg 2.e3 is incomplete, acting [20,7]
 pg 2.c1 is incomplete, acting [13,7]
 pg 2.92 is incomplete, acting [8,13]
 root@ceph-admin-storage:~# ceph pg dump_stuck inactive
 ok
 pg_statobjectsmipdegrunfbyteslogdisklogstate  
   state_stampvreportedupup_primaryacting
 acting_primarylast_scrubscrub_stamplast_deep_scrub
 deep_scrub_stamp
 2.920000000incomplete2014-08-08 
 12:39:20.2045920'030748:7729[8,13]8[8,13]8
 13503'13904192014-06-26 01:57:48.72762513503'13904192014-06-22 
 01:57:30.114186
 2.c10000000incomplete2014-08-08 
 12:39:18.8465420'030748:7117[13,7]13[13,7]13
 13503'16870172014-06-26 20:52:51.24986413503'16870172014-06-22 
 14:24:22.633554
 2.e30000000incomplete2014-08-08 
 12:39:29.3115520'030748:8027[20,7]20[20,7]20
 13503'13987272014-06-26 07:03:25.89925413503'13987272014-06-21 
 07:02:31.393053
 2.5870000000incomplete2014-08-08 
 12:39:19.7157240'030748:7060[13,5]13[13,5]13
 13646'15429342014-06-26 07:48:42.08993513646'15429342014-06-22 
 07:46:20.363695
 root@ceph-admin-storage:~# ceph osd tree
 # idweighttype nameup/downreweight
 -199.7root default
 -851.06room room0
 -219.33host ceph-1-storage
 00.91osd.0up1
 20.91osd.2up1
 30.91osd.3up1
 41.82osd.4up1
 91.36osd.9up1
 110.68osd.11up1
 63.64osd.6up1
 51.82osd.5up1
 73.64osd.7up1
 83.64osd.8up1
 -320host ceph-2-storage
 143.64osd.14up1
 181.36osd.18up1
 191.36osd.19up1
 153.64osd.15up1
 13.64osd.1up1
 123.64osd.12up1
 220.68osd.22up1
 230.68osd.23up1
 260.68osd.26up1
 360.68osd.36up1
 -411.73host ceph-5-storage
 320.27osd.32up1
 370.27osd.37up1
 420.27osd.42up1
 431.82osd.43up1
 441.82osd.44up1
 451.82osd.45up1
 461.82osd.46up1
 471.82osd.47up1
 481.82osd.48up1
 -948.64room room1
 -515.92host 

Re: [ceph-users] Fobidden 403 and fail to create subuser key when use radosgw

[Karan Singh]

For your item number 3 , can you try :

Removing the keys for sub user  ( testuser:swf0001 )

Once Key is removed for sub user ,  try recreating the key  [ # radosgw-admin 
key create --subuser=testuser:swf0001 --key-type=swift --gen-secret ]


- Karan -

On 12 Aug 2014, at 11:26, debian Only onlydeb...@gmail.com wrote:

 Dear all
 
 i have meet some issue when access radosgw.
 Fobidden 403 and fail to create subuser key when use radosgw
 
 ceph version 0.80.5(ceph osd, radosgw), OS Wheezy
 
 (1) Reference of installation 
 http://ceph.com/docs/master/radosgw/config/#configuring-print-continue
 
 (2) Config File
 root@ceph-radosgw:~# more /etc/ceph/ceph.conf
 [global]
 fsid = ae3da4d2-eef0-47cf-a872-24df8f2c8df4
 mon_initial_members = ceph01-vm
 mon_host = 192.168.123.251
 auth_cluster_required = cephx
 auth_service_required = cephx
 auth_client_required = cephx
 filestore_xattr_use_omap = true
 
 rgw print continue = false
 rgw dns name = ceph-radosgw
 debug rgw = 20
 
 
 [client.radosgw.gateway]
 host = ceph-radosgw
 keyring = /etc/ceph/ceph.client.radosgw.keyring
 rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
 log file = /var/log/ceph/client.radosgw.gateway.log
 
 root@ceph-admin:~# rados lspools
  data
  metadata
  rbd
  testpool
  iscsi
  pool-A
  pool-B
  iscsi_pool
  .rgw.root
  .rgw.control
  .rgw
  .rgw.gc
  .users.uid
  .users
  .users.swift
  .users.email
  .rgw.buckets
  .rgw.buckets.index
  .log
  .intent-log
  .usage
 
  when access radosgw http://192.168.123.191, seam ok
ListAllMyBucketsResult xmlns=http://s3.amazonaws.com/doc/2006-03-01/;
Owner
IDanonymous/ID
DisplayName/
/Owner
Buckets/
/ListAllMyBucketsResult
 
 
 (3) error meet when create radosgw user(swift) and gen-key
 
 root@ceph-radosgw:~# radosgw-admin user create --uid=testuser 
 --display-nameFirst User
 { user_id: testuser,
   display_name: First User,
   email: ,
   suspended: 0,
   max_buckets: 1000,
   auid: 0,
   subusers: [],
   keys: [
 { user: testuser,
   access_key: SU3L3KCDXQ31KJ6BZ04B,
   secret_key: nhA2XNsqwJN8bZlkOEd2UyexMADC9THOhc7UmW4l}],
   swift_keys: [],
   caps: [],
   op_mask: read, write, delete,
   default_placement: ,
   placement_tags: [],
   bucket_quota: { enabled: false,
   max_size_kb: -1,
   max_objects: -1},
   user_quota: { enabled: false,
   max_size_kb: -1,
   max_objects: -1},
   temp_url_keys: []}
 root@ceph-radosgw:~# radosgw-admin usage show --show-log-entries=false
 { summary: []}root@ceph-radosgw:~# radosgw-admin user create 
 --uid=testuser First User^Ce=F
 root@ceph-radosgw:~# radosgw-admin subuser create --uid=testuser 
 --subuser=testuser:swf0001 --access=full
 { user_id: testuser,
   display_name: First User,
   email: ,
   suspended: 0,
   max_buckets: 1000,
   auid: 0,
   subusers: [
 { id: testuser:swf0001,
   permissions: full-control}],
   keys: [
 { user: testuser:swf0001,
   access_key: 9IN7P6HA6K4JCDO61N67,
   secret_key: },
 { user: testuser,
   access_key: SU3L3KCDXQ31KJ6BZ04B,
   secret_key: nhA2XNsqwJN8bZlkOEd2UyexMADC9THOhc7UmW4l}],
   swift_keys: [],
   caps: [],
   op_mask: read, write, delete,
   default_placement: ,
   placement_tags: [],
   bucket_quota: { enabled: false,
   max_size_kb: -1,
   max_objects: -1},
   user_quota: { enabled: false,
   max_size_kb: -1,
   max_objects: -1},
   temp_url_keys: []}
 root@ceph-radosgw:~# radosgw-admin key create --subuser=testuser:swf0001 
 --key-type=swift --gen-secret
 could not create key: unable to add access key, unable to store user info
 2014-08-11 19:56:35.834507 7f4c4f1af780  0 WARNING: can't store user 
 info, swift id () already mapped to another user (testuser)
 
 (find it maybe bug ,  http://tracker.ceph.com/issues/9002)
 
 
 root@ceph-radosgw:~# radosgw-admin user create --subuser=testuser:swf0001 
 --display-name=Test User One --key-type=swift --access=full
 could not create user: unable to create user, user: testuser exists
 root@ceph-radosgw:~# radosgw-admin user create --subuser=testuser:swf0001 
 --display-name=Test User One --key-type=swift --access=full
 could not create user: unable to create user, user: testuser exists
 root@ceph-radosgw:~# radosgw-admin user rm --uid=testuser
 
 root@ceph-radosgw:~# radosgw-admin user create --subuser=testuser:swf0001 
 --display-name=Test User One --key-type=swift --access=full
 { user_id: testuser,
   display_name: Test User One,
   email: ,
 

Re: [ceph-users] HEALTH_WARN 4 pgs incomplete; 4 pgs stuck inactive; 4 pgs stuck unclean

[Karan Singh]

# rjenkins1
 item room0 weight 51.060
 item room1 weight 48.640
 }
 
 # rules
 rule data {
 ruleset 0
 type replicated
 min_size 1
 max_size 10
 step take default
 step chooseleaf firstn 0 type host
 step emit
 }
 rule metadata {
 ruleset 1
 type replicated
 min_size 1
 max_size 10
 step take default
 step chooseleaf firstn 0 type host
 step emit
 }
 rule rbd {
 ruleset 2
 type replicated
 min_size 1
 max_size 10
 step take default
 step chooseleaf firstn 0 type host
 step emit
 }
 
 # end crush map
 
 root@ceph-admin-storage:~# ceph osd dump | grep -i pool
 pool 0 'data' replicated size 2 min_size 1 crush_ruleset 0 object_hash 
 rjenkins pg_num 2048 pgp_num 2048 last_change 4623 crash_replay_interval 45 
 stripe_width 0
 pool 1 'metadata' replicated size 2 min_size 1 crush_ruleset 1 object_hash 
 rjenkins pg_num 2048 pgp_num 2048 last_change 4627 stripe_width 0
 pool 2 'rbd' replicated size 2 min_size 1 crush_ruleset 2 object_hash 
 rjenkins pg_num 2048 pgp_num 2048 last_change 4632 stripe_width 0
 
 
 Mike
 Von: Karan Singh [karan.si...@csc.fi]
 Gesendet: Dienstag, 12. August 2014 10:35
 An: Riederer, Michael
 Cc: ceph-users@lists.ceph.com
 Betreff: Re: [ceph-users] HEALTH_WARN 4 pgs incomplete; 4 pgs stuck inactive; 
 4 pgs stuck unclean
 
 Can you provide your cluster’s ceph osd dump | grep -i pooland crush map 
 output. 
 
 
 - Karan -
 
 On 12 Aug 2014, at 10:40, Riederer, Michael michael.riede...@br.de wrote:
 
 Hi all,
 
 How do I get my Ceph Cluster back to a healthy state?
 
 root@ceph-admin-storage:~# ceph -v
 ceph version 0.80.5 (38b73c67d375a2552d8ed67843c8a65c2c0feba6)
 root@ceph-admin-storage:~# ceph -s
 cluster 6b481875-8be5-4508-b075-e1f660fd7b33
  health HEALTH_WARN 4 pgs incomplete; 4 pgs stuck inactive; 4 pgs stuck 
 unclean
  monmap e2: 3 mons at 
 {ceph-1-storage=10.65.150.101:6789/0,ceph-2-storage=10.65.150.102:6789/0,ceph-3-storage=10.65.150.103:6789/0},
  election epoch 5010, quorum 0,1,2 
 ceph-1-storage,ceph-2-storage,ceph-3-storage
  osdmap e30748: 55 osds: 55 up, 55 in
   pgmap v10800465: 6144 pgs, 3 pools, 11002 GB data, 2762 kobjects
 22077 GB used, 79933 GB / 102010 GB avail
 6138 active+clean
4 incomplete
2 active+clean+replay
 root@ceph-admin-storage:~# ceph health detail
 HEALTH_WARN 4 pgs incomplete; 4 pgs stuck inactive; 4 pgs stuck unclean
 pg 2.92 is stuck inactive since forever, current state incomplete, last 
 acting [8,13]
 pg 2.c1 is stuck inactive since forever, current state incomplete, last 
 acting [13,7]
 pg 2.e3 is stuck inactive since forever, current state incomplete, last 
 acting [20,7]
 pg 2.587 is stuck inactive since forever, current state incomplete, last 
 acting [13,5]
 pg 2.92 is stuck unclean since forever, current state incomplete, last 
 acting [8,13]
 pg 2.c1 is stuck unclean since forever, current state incomplete, last 
 acting [13,7]
 pg 2.e3 is stuck unclean since forever, current state incomplete, last 
 acting [20,7]
 pg 2.587 is stuck unclean since forever, current state incomplete, last 
 acting [13,5]
 pg 2.587 is incomplete, acting [13,5]
 pg 2.e3 is incomplete, acting [20,7]
 pg 2.c1 is incomplete, acting [13,7]
 pg 2.92 is incomplete, acting [8,13]
 root@ceph-admin-storage:~# ceph pg dump_stuck inactive
 ok
 pg_statobjectsmipdegrunfbyteslogdisklogstate 
state_stampvreportedupup_primaryacting
 acting_primarylast_scrubscrub_stamplast_deep_scrub
 deep_scrub_stamp
 2.920000000incomplete2014-08-08 
 12:39:20.2045920'030748:7729[8,13]8[8,13]8
 13503'13904192014-06-26 01:57:48.72762513503'13904192014-06-22 
 01:57:30.114186
 2.c10000000incomplete2014-08-08 
 12:39:18.8465420'030748:7117[13,7]13[13,7]13
 13503'16870172014-06-26 20:52:51.24986413503'16870172014-06-22 
 14:24:22.633554
 2.e30000000incomplete2014-08-08 
 12:39:29.3115520'030748:8027[20,7]20[20,7]20
 13503'13987272014-06-26 07:03:25.89925413503'13987272014-06-21 
 07:02:31.393053
 2.5870000000incomplete2014-08-08 
 12:39:19.7157240'030748:7060[13,5]13[13,5]13
 13646'15429342014-06-26 07:48:42.08993513646'15429342014-06-22 
 07:46:20.363695
 root@ceph-admin-storage:~# ceph osd tree
 # idweighttype nameup/downreweight
 -199.7root default
 -851.06room room0
 -219.33host ceph-1-storage
 00.91osd.0up1
 20.91osd.2up1
 30.91osd.3up1
 41.82osd.4up1
 91.36

Re: [ceph-users] Issues with installing 2 node system

[Karan Singh]

Try to add proxy settings in wgetrc file (/etc/wgetrc) and rpm macros 
(/etc/rpm/macros)

# cat /etc/wgetrc  | grep -i proxy
#https_proxy = http://proxy.yoyodyne.com:18023/
http_proxy = your_proxy_url:your_proxy_port
#ftp_proxy = http://proxy.yoyodyne.com:18023/
# If you do not want to use proxy at all, set this to off.
use_proxy = yes
#


# cat /etc/rpm/macros
%_httpproxy your_proxy_url
%_httpport your_proxy_port
#


- Karan -

On 11 Aug 2014, at 21:48, Ojwang, Wilson O (Wilson) 
wilson.ojw...@alcatel-lucent.com wrote:

  
 I am currently new to Ceph and had the following error while trying to 
 install a 2 node system (admin and one other node) using quick installation 
 guide from http://ceph.com/docs/master/start/
  
 =\
 [root@nfv2 ~]# ceph-deploy install nfv2 nfv3
 [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
 [ceph_deploy.cli][INFO  ] Invoked (1.5.10): /usr/bin/ceph-deploy install nfv2 
 nfv3
 [ceph_deploy.install][DEBUG ] Installing stable version firefly on cluster 
 ceph hosts nfv2 nfv3
 [ceph_deploy.install][DEBUG ] Detecting platform for host nfv2 ...
 [nfv2][DEBUG ] connected to host: nfv2
 [nfv2][DEBUG ] detect platform information from remote host
 [nfv2][DEBUG ] detect machine type
 [ceph_deploy.install][INFO  ] Distro info: CentOS 6.5 Final
 [nfv2][INFO  ] installing ceph on nfv2
 [nfv2][INFO  ] Running command: yum clean all
 [nfv2][DEBUG ] Loaded plugins: fastestmirror, priorities, refresh-packagekit, 
 security
 [nfv2][DEBUG ] Cleaning repos: apache2-ceph-noarch base ceph ceph-extras 
 ceph-extras-noarch
 [nfv2][DEBUG ]   : ceph-extras-source ceph-noarch epel extras 
 fastcgi-ceph-basearch
 [nfv2][DEBUG ]   : fastcgi-ceph-noarch foreman foreman-plugins 
 openstack-icehouse
 [nfv2][DEBUG ]   : puppetlabs-deps puppetlabs-products updates
 [nfv2][DEBUG ] Cleaning up Everything
 [nfv2][DEBUG ] Cleaning up list of fastest mirrors
 [nfv2][INFO  ] Running command: yum -y install wget
 [nfv2][DEBUG ] Loaded plugins: fastestmirror, priorities, refresh-packagekit, 
 security
 [nfv2][DEBUG ] Determining fastest mirrors
 [nfv2][DEBUG ]  * base: bay.uchicago.edu
 [nfv2][DEBUG ]  * epel: archive.linux.duke.edu
 [nfv2][DEBUG ]  * extras: centos.mbni.med.umich.edu
 [nfv2][DEBUG ]  * updates: ftpmirror.your.org
 [nfv2][DEBUG ] 308 packages excluded due to repository priority protections
 [nfv2][DEBUG ] Setting up Install Process
 [nfv2][DEBUG ] Package wget-1.12-1.11.el6_5.x86_64 already installed and 
 latest version
 [nfv2][DEBUG ] Nothing to do
 [nfv2][INFO  ] adding EPEL repository
 [nfv2][INFO  ] Running command: wget 
 http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
 [nfv2][WARNIN] No data was received after 300 seconds, disconnecting...
 [nfv2][INFO  ] Running command: rpm -Uvh --replacepkgs epel-release-6*.rpm
 [nfv2][WARNIN] error: File not found by glob: epel-release-6*.rpm
 [nfv2][ERROR ] RuntimeError: command returned non-zero exit status: 1
 [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: rpm -Uvh 
 --replacepkgs epel-release-6*.rpm
  
 =\
  
 I have setup http, https, global proxy and was able to run “wget 
 http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm” on 
 the command line without any issue.
  
 Thanks in advance for your support.
  
 Regards
 Wilson
  
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] wired output of ceph df : Firefly 0.80.5

[Karan Singh]

Hello Developers 

I have encountered some wired output of ceph df command , suddenly


When i was writing some data on cache-pool , and checked its used % , i found 
some used as 8E ( don’t know what is this ) and the used % for cache-pool was 0


# ceph df
GLOBAL:
SIZE  AVAIL RAW USED %RAW USED
1301T 1284T 17518G   1.31
POOLS:
NAME   ID USED   %USED OBJECTS
data   0  801M   0 2
metadata   1  801M   0 22
rbd2  0  0 0
.rgw   3  3524   0 26
.rgw.root  4  7780 3
.rgw.control   5  0  0 8
.rgw.buckets   6  8201M  0 2298
.rgw.buckets.index 7  0  0 13
.users.swift   8  7  0 1
volumes9  1106G  0.08  283387
images 10 40960k 0 8
backups11 0  0 0
.rgw.gc12 0  0 32
.users.uid 13 8480 5
.users 14 16 0 2
.log   15 153k   0 37
   16 0  0 0
hpsl4540   21 110G   0 28152
hpdl38022 245G   0.02  62688
EC-2-2 23 6338G  0.48  4859
cache-pool 24 8E 0 5849 
## What is the meaning of E here , also please note used % for 
cache-pool is 0 here
ssd25 25196M 0 5464


After some time when cache-poo used value changed to 7E and used % as 644301.19 
 . While there were no objects in the cache-pool


# ceph df
GLOBAL:
SIZE  AVAIL RAW USED %RAW USED
1301T 1284T 17508G   1.31
POOLS:
NAME   ID USED   %USED OBJECTS
data   0  801M   0 2
metadata   1  801M   0 22
rbd2  0  0 0
.rgw   3  3524   0 26
.rgw.root  4  7780 3
.rgw.control   5  0  0 8
.rgw.buckets   6  8201M  0 2298
.rgw.buckets.index 7  0  0 13
.users.swift   8  7  0 1
volumes9  1106G  0.08  283387
images 10 40960k 0 8
backups11 0  0 0
.rgw.gc12 0  0 32
.users.uid 13 8480 5
.users 14 16 0 2
.log   15 153k   0 37
   16 0  0 0
hpsl4540   21 110G   0 28152
hpdl38022 245G   0.02  62688
EC-2-2 23 6338G  0.48  4843
cache-pool 24 7E 644301.19 1056 
  ## The used % for cache-pool has become 644301.19 
ssd25 25196M 0 5464
#


# rados -p cache-pool ls 
#




Is this a bug , if yes , then is is already known. Do you want me to raise a 
bug ticket in tracker.ceph.com ?




Karan Singh 
Cloud computing group
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] wired output of ceph df : Firefly 0.80.5

[Karan Singh]

Forgot to mention i am observing  EB  in ceph -s output  ,  does it mean 
Exabyte  ;-)


# ceph -s
cluster 009d3518-e60d-4f74-a26d-c08c1976263c
 health HEALTH_WARN 'cache-pool' at/near target max
 monmap e3: 3 mons at 
 mdsmap e14: 1/1/1 up {0=storage0101-ib=up:active}
 osdmap e194215: 402 osds: 402 up, 402 in
  pgmap v743051: 31168 pgs, 22 pools, 8 EB data, 378 kobjects
17508 GB used, 1284 TB / 1301 TB avail
   31168 active+clean


- Karan Singh -


On 12 Aug 2014, at 16:45, Karan Singh karan.si...@csc.fi wrote:

 Hello Developers 
 
 I have encountered some wired output of ceph df command , suddenly
 
 
 When i was writing some data on cache-pool , and checked its used % , i found 
 some used as 8E ( don’t know what is this ) and the used % for cache-pool was  0
 
 
 # ceph df
 GLOBAL:
 SIZE  AVAIL RAW USED %RAW USED
 1301T 1284T 17518G   1.31
 POOLS:
 NAME   ID USED   %USED OBJECTS
 data   0  801M   0 2
 metadata   1  801M   0 22
 rbd2  0  0 0
 .rgw   3  3524   0 26
 .rgw.root  4  7780 3
 .rgw.control   5  0  0 8
 .rgw.buckets   6  8201M  0 2298
 .rgw.buckets.index 7  0  0 13
 .users.swift   8  7  0 1
 volumes9  1106G  0.08  283387
 images 10 40960k 0 8
 backups11 0  0 0
 .rgw.gc12 0  0 32
 .users.uid 13 8480 5
 .users 14 16 0 2
 .log   15 153k   0 37
16 0  0 0
 hpsl4540   21 110G   0 28152
 hpdl38022 245G   0.02  62688
 EC-2-2 23 6338G  0.48  4859
 cache-pool 24 8E 0 5849   
 ## What is the meaning of E here , also please note used % for 
 cache-pool is 0 here
 ssd25 25196M 0 5464
 
 
 After some time when cache-poo used value changed to 7E and used % as 
 644301.19  . While there were no objects in the cache-pool
 
 
 # ceph df
 GLOBAL:
 SIZE  AVAIL RAW USED %RAW USED
 1301T 1284T 17508G   1.31
 POOLS:
 NAME   ID USED   %USED OBJECTS
 data   0  801M   0 2
 metadata   1  801M   0 22
 rbd2  0  0 0
 .rgw   3  3524   0 26
 .rgw.root  4  7780 3
 .rgw.control   5  0  0 8
 .rgw.buckets   6  8201M  0 2298
 .rgw.buckets.index 7  0  0 13
 .users.swift   8  7  0 1
 volumes9  1106G  0.08  283387
 images 10 40960k 0 8
 backups11 0  0 0
 .rgw.gc12 0  0 32
 .users.uid 13 8480 5
 .users 14 16 0 2
 .log   15 153k   0 37
16 0  0 0
 hpsl4540   21 110G   0 28152
 hpdl38022 245G   0.02  62688
 EC-2-2 23 6338G  0.48  4843
 cache-pool 24 7E 644301.19 1056   
 ## The used % for cache-pool has become 644301.19 
 ssd25 25196M 0 5464
 #
 
 
 # rados -p cache-pool ls 
 #
 
 
 
 
 Is this a bug , if yes , then is is already known. Do you want me to raise a 
 bug ticket in tracker.ceph.com ?
 
 
 
 
 Karan Singh 
 Cloud computing group
 CSC - IT Center for Science,
 Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
 tel. +358 9 4572001
 fax +358 9 4572302
 http://www.csc.fi/
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http

Re: [ceph-users] Can't start OSD

[Karan Singh]

Try to make these OSD IN

ceph osd in osd.12 osd.13 osd.14 osd.15

Then restart osd services 


- Karan Singh -

On 08 Aug 2014, at 00:55, O'Reilly, Dan daniel.orei...@dish.com wrote:

 # idweight  type name   up/down reweight
 -1  7.2 root default
 -2  1.8 host tm1cldosdl01
 0   0.45osd.0   up  1
 1   0.45osd.1   up  1
 2   0.45osd.2   up  1
 3   0.45osd.3   up  1
 -3  1.8 host tm1cldosdl02
 4   0.45osd.4   up  1
 5   0.45osd.5   up  1
 6   0.45osd.6   up  1
 7   0.45osd.7   up  1
 -4  1.8 host tm1cldosdl03
 8   0.45osd.8   up  1
 9   0.45osd.9   up  1
 10  0.45osd.10  up  1
 11  0.45osd.11  up  1
 -5  1.8 host tm1cldosdl04
 12  0.45osd.12  down0
 13  0.45osd.13  down0
 14  0.45osd.14  down0
 15  0.45osd.15  down0
 [ceph@tm1cldosdl04 ~]$ sudo /etc/init.d/ceph start osd.12
 /etc/init.d/ceph: osd.12 not found (/etc/ceph/ceph.conf defines , 
 /var/lib/ceph defines )
  
 What am I missing?  Specifically, what would need to be in ceph.conf or 
 /var/lib/ceph?
  
 Dan O'Reilly
 UNIX Systems Administration
 image001.jpg
 9601 S. Meridian Blvd.
 Englewood, CO 80112
 720-514-6293
  
  
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph rbd volume can't remove because image still has watchers

[Karan Singh]

Make sure the volume  volume-17d9397b-d6e5-45e0-80fa-4bc7b7998842” does no 
have a snapshot or cloned linked to it  , it might sometimes cause problems 
while deletion.


- Karan Singh

On 07 Aug 2014, at 08:55, 杨万元 yangwanyuan8...@gmail.com wrote:

 Hi all：
 we use ceph rbd with openstack ，recently there are some  dirty data in my 
 cinder-volume databases such as volumes status like error-deleting. So we 
 need manually delete this volumes。
 but when I delete the volume on ceph node，ceph tell me this error
 
   [root@ceph-node3 ~]# rbd -p glance rm 
 volume-17d9397b-d6e5-45e0-80fa-4bc7b7998842
 Removing image: 99% complete...failed.
 rbd: error: image still has watchers
 This means the image is still open or the client using it crashed. 
 Try again after   closing/unmapping it or waiting 30s for the crashed 
 client to timeout.
 2014-08-07 11:25:42.793275 7faf8c58b760 -1 librbd: error removing 
 header: (16) Device or resource busy
 
 
I google this problem and  find this  
 http://comments.gmane.org/gmane.comp.file-systems.ceph.user/9767
I did it and got this:
 
  [root@ceph-node3 ~]# rbd info -p glance 
 volume-17d9397b-d6e5-45e0-80fa-4bc7b7998842
 rbd image 'volume-17d9397b-d6e5-45e0-80fa-4bc7b7998842':
 size 51200 MB in 12800 objects
 order 22 (4096 kB objects)
 block_name_prefix: rbd_data.3b1464f8e96d5
 format: 2
 features: layering
  [root@ceph-node3 ~]# rados -p glance listwatchers 
 rbd_header.3b1464f8e96d5 
 watcher=192.168.39.116:0/1032797 client.252302 cookie=1
 
   192.168.39.116 is my nova compute node ,so i can't reboot this server,
   what can i do to delete this volume without reboot my  compute-node?
 
   my ceph version is 0.72.1.
 
  thanks very much!
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Could not find module rbd. CentOs 6.4

[Karan Singh]

Yes you can use other features like CephFS and Object Store on this kernel 
release that you are running.

- Karan Singh 


On 28 Jul 2014, at 07:45, Pratik Rupala pratik.rup...@calsoftinc.com wrote:

 Hi Karan,
 
 I have basic setup of Ceph storage cluster in active+clean state on Linux 
 kernel 2.6.32. As per your suggestion, RBD support starts from 2.6.34 kernel.
 So, can I use other facilities like object store and Cephfs on this setup 
 with 2.6.32 or they are also not supported for this kernel version and is 
 there any way to have Ceph block devices on Linux kernel 2.6.32?
 
 Regards,
 Pratik Rupala
 
 
 On 7/25/2014 5:51 PM, Karan Singh wrote:
 Hi Pratik
 
 Ceph RBD support has been added in mainline Linux kernel starting 2.6.34 ,  
 The following errors shows that , RBD module is not present in kernel.
 
 Its advisable to run latest stable kernel release if you need RBD to be 
 working.
 
 ERROR: modinfo: could not find module rbd
 FATAL: Module rbd not found.
 rbd: modprobe rbd failed! (256)
 
 
 
 - Karan -
 
 On 25 Jul 2014, at 14:52, Pratik Rupala pratik.rup...@calsoftinc.com wrote:
 
 Hi,
 
 I am deploying firefly version on CentOs 6.4. I am following quick 
 installation instructions available at ceph.com.
 I have my customized kernel version in CentOs 6.4 which is 2.6.32.
 
 I am able to create basic Ceph storage cluster with active+clean state. Now 
 I am trying to create block device image on ceph client but it is giving 
 messages as shown below:
 
 [ceph@ceph-client1 ~]$ rbd create foo --size 1024
 2014-07-25 22:31:48.519218 7f6721d43700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x6a7c50 sd=4 :0 s=1 pgs=0 cs=0 l=1 
 c=0x6a8050).fault
 2014-07-25 22:32:18.536771 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718006310 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f6718006580).fault
 2014-07-25 22:33:09.598763 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f67180063e0 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f6718007e70).fault
 2014-07-25 22:34:08.621655 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718007e70 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f67180080e0).fault
 2014-07-25 22:35:19.581978 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718007e70 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f67180080e0).fault
 2014-07-25 22:36:23.694665 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718007e70 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f67180080e0).fault
 2014-07-25 22:37:28.868293 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718007e70 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f67180080e0).fault
 2014-07-25 22:38:29.159830 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718007e70 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f67180080e0).fault
 2014-07-25 22:39:28.854441 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718001db0 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f6718006990).fault
 2014-07-25 22:40:14.581055 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718001ac0 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f671800c950).fault
 2014-07-25 22:41:03.794903 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718004d30 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f671800c950).fault
 2014-07-25 22:42:12.537442 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x6a4640 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x6a4a00).fault
 2014-07-25 22:43:18.912430 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718008300 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f67180080e0).fault
 2014-07-25 22:44:24.129258 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718008300 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f6718008f80).fault
 2014-07-25 22:45:29.174719 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f671800a150 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f671800a620).fault
 2014-07-25 22:46:34.032246 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718008390 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f671800a620).fault
 2014-07-25 22:47:39.551973 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718008390 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f67180077e0).fault
 2014-07-25 22:48:39.342226 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718001db0 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f6718003040).fault
 
 I am not sure whether block device image has been created or not. Further I 
 tried below command which fails:
 [ceph@ceph-client1 ~]$ sudo rbd map foo
 ERROR: modinfo: could not find module rbd
 FATAL: Module rbd not found.
 rbd: modprobe rbd failed! (256)
 
 If I check the health of cluster it looks fine.
 [ceph@node1 ~]$ ceph -s
cluster 98f22f5d-783b-43c2-8ae7-b97a715c9c86
 health HEALTH_OK
 monmap e1: 1 mons

Re: [ceph-users] firefly osds stuck in state booting

[Karan Singh]

The output that you have provided says that OSDs are not IN , Try the below

ceph osd in osd.0
ceph osd in osd.1

service ceph start osd.0
service ceph start osd.1

If you have 1 more host with 1 disk , add it , starting Ceph Firefly default 
rep size is 3


- Karan -

On 27 Jul 2014, at 11:17, 10 minus t10te...@gmail.com wrote:

 Hi Sage, 
 
 I have dropped all unset .. and even restarted the osd 
 No dice .. OSDs are still stuck .
 
 
 
 --snip--
  ceph daemon osd.0 status│rtt 
 min/avg/max/mdev = 0.095/0.120/0.236/0.015 ms 
 
 { cluster_fsid: 99babb8f-c880-4b32-a227-94aa483d4871, 
│[root@ceph2 ~]#  ceph daemon osd.1 status 
   
   
   osd_fsid: 1ad28bde-c23c-44ba-a3b7-0fd3372e, 
│{ cluster_fsid: 99babb8f-c880-4b32-a227-94aa483d4871, 
   
   
   whoami: 0,
│  osd_fsid: becc3252-6977-47d6-87af-7b1337e591d8, 
   
   
   state: booting, 
│  whoami: 1,
   
   
   oldest_map: 1,
│  state: booting, 
   
   
   newest_map: 24,   
│  oldest_map: 1,
   
   
   num_pgs: 0}   
│  newest_map: 21,   
   
   
  --snip--
 
 --snip-- 
 ceph osd tree 
  
 # idweight  type name   up/down reweight
 -1  2   root default
 -3  1   host ceph1
 0   1   osd.0   down0
 -2  1   host ceph2
 1   1   osd.1   down0
 
  --snip--
 
 --snip--
  ceph -s
 cluster 2929fa80-0841-4cb6-a133-90b2098fc802
  health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean
  monmap e2: 3 mons at 
 {ceph0=10.0.12.220:6789/0,ceph1=10.0.12.221:6789/0,ceph2=10.0.12.222:6789/0}, 
 election epoch 50, quorum 0,1,2 ceph0,ceph1,ceph2
  osdmap e24: 2 osds: 0 up, 0 in
   pgmap v25: 192 pgs, 3 pools, 0 bytes data, 0 objects
 0 kB used, 0 kB / 0 kB avail
  192 creating
 --snip--
 
 
 
 
 On Sat, Jul 26, 2014 at 5:57 PM, Sage Weil sw...@redhat.com wrote:
 On Sat, 26 Jul 2014, 10 minus wrote:
  Hi,
 
  I just setup a test ceph installation on 3 node Centos 6.5  .
  two of the nodes are used for hosting osds and the third acts as mon .
 
  Please note I'm using LVM so had to set up the osd using the manual install
  guide.
 
  --snip--
  ceph -s
  cluster 2929fa80-0841-4cb6-a133-90b2098fc802
   health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean;
  noup,nodown,noout flag(s) set
   monmap e2: 3 mons 
  at{ceph0=10.0.12.220:6789/0,ceph1=10.0.12.221:6789/0,ceph2=10.0.12.222:6789/0
  }, election epoch 46, quorum 0,1,2 ceph0,ceph1,ceph2
   osdmap e21: 2 osds: 0 up, 0 in
  flags noup,nodown,noout
 
 
 Do 'ceph osd unset noup' and they should start up.  You likely also want
 to clear nodown and noout as well.
 
 sage
 
 
pgmap v22: 192 pgs, 3 pools, 0 bytes data, 0 objects
  0 kB used, 0 kB / 0 kB avail
   192 creating
  --snip--
 
  osd tree
 
  --snip--
  ceph osd tree
  # idweight  type name   up/down reweight
  -1  2   root default
  -3  1   host ceph1
  0   1   osd.0   down0
  -2  1   host ceph2
  1   1   osd.1   down0
  --snip--
 
  --snip--
   ceph daemon osd.0 status
  { cluster_fsid: 99babb8f-c880-4b32-a227-94aa483d4871,
osd_fsid: 1ad28bde-c23c-44ba-a3b7-0fd3372e,
whoami: 0,
state: booting,
oldest_map: 1,
newest_map: 21,
num_pgs: 0}
 
  --snip--
 
  --snip--
   ceph daemon osd.1 status
  { cluster_fsid: 99babb8f-c880-4b32-a227-94aa483d4871,
osd_fsid: 

Re: [ceph-users] OSD weight 0

[Karan Singh]

Looks like osd.1 has a valid auth ID , which was defined previously.

Trust this is your test cluster , try this

ceph osd crush rm osd.1
ceph osd rm osd.1
ceph auth del osd.1

Once again try to add osd.1 using ceph-deploy ( prepare and then activate 
commands ) , check the logs carefully for any other clues.

- Karan  Singh -

On 25 Jul 2014, at 12:49, Kapil Sharma ksha...@suse.com wrote:

 Hi,
 
 I am using ceph-deploy to deploy my cluster. Whenever I try to add more
 than one osd in a node, except the first osd, all the other osds get a
 weight of 0, and they are in a state of down and out.
 
 So, if I have three nodes in my cluster, I can successfully add 1 node
 each in the three nodes, but the moment I try to add a second node in
 any of the nodes, it gets a weight of 0 and goes down and out.
 
 The capacity of all the disks is same.
 
 
 cephdeploy@node-1:~/cluster ceph osd tree
 # idweight  type name   up/down reweight
 -1  1.82root default
 -2  1.82host node-1
 0   1.82osd.0   up  1
 1   0   osd.1   down0
 
 There is no error as such after I run ceph-deploy activate command.
 
 Has anyone seen this issue before ? 
 
 
 
 Kind Regards,
 Kapil.
 
 
 
 
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Could not find module rbd. CentOs 6.4

[Karan Singh]

Hi Pratik

Ceph RBD support has been added in mainline Linux kernel starting 2.6.34 ,  The 
following errors shows that , RBD module is not present in kernel.

Its advisable to run latest stable kernel release if you need RBD to be working.

 ERROR: modinfo: could not find module rbd
 FATAL: Module rbd not found.
 rbd: modprobe rbd failed! (256)



- Karan -

On 25 Jul 2014, at 14:52, Pratik Rupala pratik.rup...@calsoftinc.com wrote:

 Hi,
 
 I am deploying firefly version on CentOs 6.4. I am following quick 
 installation instructions available at ceph.com.
 I have my customized kernel version in CentOs 6.4 which is 2.6.32.
 
 I am able to create basic Ceph storage cluster with active+clean state. Now I 
 am trying to create block device image on ceph client but it is giving 
 messages as shown below:
 
 [ceph@ceph-client1 ~]$ rbd create foo --size 1024
 2014-07-25 22:31:48.519218 7f6721d43700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x6a7c50 sd=4 :0 s=1 pgs=0 cs=0 l=1 
 c=0x6a8050).fault
 2014-07-25 22:32:18.536771 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718006310 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f6718006580).fault
 2014-07-25 22:33:09.598763 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f67180063e0 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f6718007e70).fault
 2014-07-25 22:34:08.621655 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718007e70 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f67180080e0).fault
 2014-07-25 22:35:19.581978 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718007e70 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f67180080e0).fault
 2014-07-25 22:36:23.694665 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718007e70 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f67180080e0).fault
 2014-07-25 22:37:28.868293 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718007e70 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f67180080e0).fault
 2014-07-25 22:38:29.159830 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718007e70 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f67180080e0).fault
 2014-07-25 22:39:28.854441 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718001db0 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f6718006990).fault
 2014-07-25 22:40:14.581055 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718001ac0 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f671800c950).fault
 2014-07-25 22:41:03.794903 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718004d30 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f671800c950).fault
 2014-07-25 22:42:12.537442 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x6a4640 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x6a4a00).fault
 2014-07-25 22:43:18.912430 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718008300 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f67180080e0).fault
 2014-07-25 22:44:24.129258 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718008300 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f6718008f80).fault
 2014-07-25 22:45:29.174719 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f671800a150 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f671800a620).fault
 2014-07-25 22:46:34.032246 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718008390 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f671800a620).fault
 2014-07-25 22:47:39.551973 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718008390 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f67180077e0).fault
 2014-07-25 22:48:39.342226 7f6721b41700  0 -- 172.17.35.20:0/1003053  
 172.17.35.22:6800/1875 pipe(0x7f6718001db0 sd=5 :0 s=1 pgs=0 cs=0 l=1 
 c=0x7f6718003040).fault
 
 I am not sure whether block device image has been created or not. Further I 
 tried below command which fails:
 [ceph@ceph-client1 ~]$ sudo rbd map foo
 ERROR: modinfo: could not find module rbd
 FATAL: Module rbd not found.
 rbd: modprobe rbd failed! (256)
 
 If I check the health of cluster it looks fine.
 [ceph@node1 ~]$ ceph -s
cluster 98f22f5d-783b-43c2-8ae7-b97a715c9c86
 health HEALTH_OK
 monmap e1: 1 mons at {node1=172.17.35.17:6789/0}, election epoch 1, 
 quorum 0 node1
 osdmap e5972: 3 osds: 3 up, 3 in
  pgmap v20011: 192 pgs, 3 pools, 142 bytes data, 2 objects
190 MB used, 45856 MB / 46046 MB avail
 192 active+clean
 
 Please let me know if I am doing anything wrong.
 
 Regards,
 Pratik Rupala
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] basic questions about pool

[Karan Singh]

Hi Pragya

Let me try to answer these.

1#  The decisions is based on your use case ( performance , reliability ) .If 
you need high performance out of your cluster , the deployer will create a pool 
on SSD and assign this pool to applications which require higher I/O. For Ex : 
if you integrate openstack with Ceph , you can instruct openstack configuration 
files to write data to a specific ceph pool.  
(http://ceph.com/docs/master/rbd/rbd-openstack/#configuring-glance) , similarly 
you can instruct CephFS and RadosGW with pool to use for data storage.

2#  Usually the end user (client to ceph cluster) does not bother about where 
the data is getting stored , which pool its using , and what is the real 
physical locate of data. End user will demand for specific performance , 
reliability and availability. It is the job of Ceph admin to fulfil  their 
storage requirements, out of Ceph functionalities of SSD , Erausre codes , 
replication level etc.


Block Device :- End user will instruct the application ( Qemu / KVM , OpenStack 
etc ) , which pool it should for data storage. rbd is the default pool for 
block device.
CephFS :- End user will mount this pool as filesystem and can use further. 
Default pool are data and metadata .
 RadosGW :- End user will storage objects using S3 or Swift API. 



- Karan Singh -

On 15 Jul 2014, at 07:42, pragya jain prag_2...@yahoo.co.in wrote:

 thank you very much, Craig, for your clear explanation against my questions. 
 
 Now I am very clear about the concept of pools in ceph.
 
 But I have two small questions:
 1. How does the deployer decide that a particular type of information will be 
 stored in a particular pool? Are there any settings at the time of creation 
 of pool that a deployer should make to ensure that which type of data will be 
 stored in which pool?
 
 2. How does an end-user specify that his/her data will be stored in which 
 pool? how can an end-user come to know which pools are stored on SSDs or on 
 HDDs, what are the properties of a particular pool? 
 
 Thanks again, Please help to clear these confusions also. 
 
 Regards
 Pragya Jain
 
 
 On Sunday, 13 July 2014 5:04 AM, Craig Lewis cle...@centraldesktop.com 
 wrote:
 
 
 I'll answer out of order.
 
 #2: rdb is used for RDB images.  data and metadata are used by CephFS.  
 RadosGW's default pools will be created the first time radosgw starts up.  If 
 you aren't using RDB or CephFS, you can ignore those pools.
 
 #1: RadosGW will use several pools to segregate it's data.  There are a 
 couple pools for store user/subuser information, as well as pools for storing 
 the actual data.  I'm using federation, and I have a total of 18 pools that 
 RadosGW is using in some form.  Pools are a way to logically separate your 
 data, and pools can also have different replication/storage settings.  For 
 example, I could say that the .rgw.buckets.index pool needs 4x replication 
 and is only stored on SSDs, while .rgw.bucket is 3x replication on HDDs.
 
 #3: In addition to #1, you can setup different pools to actually store user 
 data in RadosGW.  For example, an end user may have some very important data 
 that you want replicated 4 times, and some other data that needs to be stored 
 on SSDs for low latency.  Using CRUSH, you would create the some rados pools 
 with those specs.  Then you'd setup some placement targets in RadosGW that 
 use those pools.  A user that cares will specify a placement target when they 
 create a bucket.  That way they can decide what the storage requirements are. 
  If they don't care, then they can just use the default.
 
 Does that help?
 
 
 
 On Thu, Jul 10, 2014 at 11:34 PM, pragya jain prag_2...@yahoo.co.in wrote:
 hi all,
 
 I have some very basic questions about pools in ceph.
 
 According to ceph documentation, as we deploy a ceph cluster with radosgw 
 instance over it, ceph creates pool by default to store the data or the 
 deployer can also create pools according to the requirement.
 
 Now, my question is:
 1. what is the relevance of multiple pools in a cluster?
 i.e. why should a deployer create multiple pools in a cluster? what should be 
 the benefits of creating multiple pools?
 
 2. according to the docs, the default pools are data, metadata, and rbd.
 what is the difference among these three types of pools?
 
 3. when a system deployer has deployed a ceph cluster with radosgw interface 
 and start providing services to the end-user, such as, end-user can create 
 their account on the ceph cluster and can store/retrieve their data to/from 
 the cluster, then Is the end user has any concern about the pools created in 
 the cluster?
 
 Please somebody help me to clear these confusions.
 
 regards
 Pragya Jain
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 
 
 ___
 ceph-users mailing list
 ceph-users

Re: [ceph-users] inktank-mellanox webinar access ?

[Karan Singh]

Hey i have missed the webinar , is this available for later review  or slides.

- Karan -

On 10 Jul 2014, at 18:27, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote:

 That makes two of us...
 
 G.
 
 On Thu, 10 Jul 2014 17:12:08 +0200 (CEST), Alexandre DERUMIER wrote:
 Ok, sorry, we have finally receive the login a bit late.
 
 Sorry again to have spam the mailing list
 - Mail original -
 
 De: Alexandre DERUMIER aderum...@odiso.com
 À: ceph-users ceph-us...@ceph.com
 Envoyé: Jeudi 10 Juillet 2014 16:55:22
 Objet: [ceph-users] inktank-mellanox webinar access ?
 
 Hi,
 
 sorry to spam the mailing list,
 
 but they are a inktank mellanox webinar in 10minutes,
 
 and I don't have receive access since I have been registered
 yesterday (same for my co-worker).
 
 and the webinar mellanox contact email (conta...@mellanox.com), does
 not exist
 
 Maybe somebody from Inktank or Mellanox could help us ?
 
 Regards,
 
 Alexandre
 
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] inktank-mellanox webinar access ?

[Karan Singh]

Thanks Georgios

I will wait.

- Karan Singh -

On 14 Jul 2014, at 15:37, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote:

 Hi Karan!
 
 Due to the late reception of the login info I 've also missed
 a very big part of the webinar.
 
 They did send me an e-mail though saying that they will let me know as soon as
 a recording of the session will be available.
 
 I will let you know again then.
 
 Best,
 
 G.
 
 On Mon, 14 Jul 2014 12:40:54 +0300, Karan Singh wrote:
 Hey i have missed the webinar , is this available for later review or
 slides.
 
 - Karan -
 
 On 10 Jul 2014, at 18:27, Georgios Dimitrakakis  wrote:
 
 That makes two of us...
 
 G.
 
 On Thu, 10 Jul 2014 17:12:08 +0200 (CEST), Alexandre DERUMIER wrote:
 
 Ok, sorry, we have finally receive the login a bit late.
 
 Sorry again to have spam the mailing list
 - Mail original -
 
 De: Alexandre DERUMIER
 À: ceph-users
 Envoyé: Jeudi 10 Juillet 2014 16:55:22
 Objet: [ceph-users] inktank-mellanox webinar access ?
 
 Hi,
 
 sorry to spam the mailing list,
 
 but they are a inktank mellanox webinar in 10minutes,
 
 and I don't have receive access since I have been registered
 yesterday (same for my co-worker).
 
 and the webinar mellanox contact email (conta...@mellanox.com
 [3]), does
 not exist
 
 Maybe somebody from Inktank or Mellanox could help us ?
 
 Regards,
 
 Alexandre
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com [4]
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com [5]
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 
 Links:
 --
 [1] mailto:aderum...@odiso.com
 [2] mailto:ceph-us...@ceph.com
 [3] mailto:conta...@mellanox.com
 [4] mailto:ceph-users@lists.ceph.com
 [5] mailto:ceph-users@lists.ceph.com
 [6] mailto:gior...@acmac.uoc.gr
 
 -- 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problem installing ceph from package manager / ceph repositories

[Karan Singh]

Hi Dimitri

It was already resolved , moderator took a long time to approve my email to get 
posted to mailing list.

Thanks for your solution .

- Karan -

On 12 Jun 2014, at 00:02, Dimitri Maziuk dmaz...@bmrb.wisc.edu wrote:

 On 06/09/2014 03:08 PM, Karan Singh wrote:
 
1. When installing Ceph using package manger and ceph repositores , the
package manager i.e YUM does not respect the ceph.repo file and takes ceph
package directly from EPEL .
 
 Option 1: install yum-plugin-priorities, add priority = X to ceph.repo.
 X should be less than EPEL's priority, the default is I believe 99.
 
 Option 2: add exclude = ceph_package(s) to epel.repo.
 
 -- 
 Dimitri Maziuk
 Programmer/sysadmin
 BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cannot attach volumes

[Karan Singh]

Hi Kumar

“Clock skew” is just a warning and should not related to this problem. But its 
pretty easy to fix this warning either by setting up NTP on all Ceph cluster 
nodes or by adding  mon clock drift warn backoff = Seconds  into ceph.conf 
(do not do this in production)

WRT to your second problem , try to check out cinder volume logs and scheduler 
logs  , you would surely find something there. If not try increasing debug 
level of cinder and check for some clues.


- Karan Singh -

On 10 Jun 2014, at 09:53, yalla.gnan.ku...@accenture.com wrote:

 Hi All,
  
  
 I have four node ceph cluster.  I have another three node setup for 
 openstack.  I have integrated Ceph with openstack.
 Whenever I try to create storage with ceph as storage backend for the 
 openstack  vm,  the creation process goes on forever in the horizon dashboard.
 It never completes. Also when attaching the ceph  volume to the VM in 
 openstack, it freezes and goes on forever without completion.
  
 To investigate this issue, I typed the ‘ceph –s’ command on the ceph nodes.  
 The health of ceph cluster  is in warning state.  It says it detected clock 
 skew
 on two of the nodes.   Is this time synchronization the reason beyond the VM 
 freezing while attaching volumes to VMs ?
  
  
 Thanks
 Kumar
  
 
 
 This message is for the designated recipient only and may contain privileged, 
 proprietary, or otherwise confidential information. If you have received it 
 in error, please notify the sender immediately and delete the original. Any 
 other use of the e-mail by you is prohibited. Where allowed by local law, 
 electronic communications with Accenture and its affiliates, including e-mail 
 and instant messaging (including content), may be scanned by our systems for 
 the purposes of information security and assessment of internal compliance 
 with Accenture policy. 
 __
 
 www.accenture.com
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Fail to Block Devices and OpenStack

[Karan Singh]

HI Yamashita

First try to create a cinder volume , that should be stored on ceph backend, 
then proceeded with glance image on Ceph.

I assume you have done all the steps correctly as mentioned on ceph 
documentation for ceph and openstack integration , if you are still not able to 
create volumes, then enable debug=true in cinder config file and check 
cinder-volume and scheduler logs . 
You will get some clue where the error is.

- Karan Singh -

On 10 Jun 2014, at 09:21, 山下 良民 yamash...@ossl.co.jp wrote:

 Hi,
 
 I can mount from connect to Ceph BlockDevice mount.
 
 glance image-create is created on the local.
 # glance image-list
 +--+-+-+--+---++
 | ID   | Name| 
 Disk Format | Container Format | Size  | Status |
 +--+-+-+--+---++
 | f4a0f971-437b-4d3f-a0c4-1c82f31e9f1e | cirros  | 
 raw | ovf  | 41126400  | active |
 +--+-+-+--+---++
 
 cindercreate is an error is not created in the Ceph.
 
 Yamashita
 
 
 - 元のメッセージ -
 差出人: 山下 良民 yamash...@ossl.co.jp
 宛先: ceph-users@lists.ceph.com
 送信済み: 2014年6月10日, 火曜日 午前 11:55:48
 件名: [ceph-users] Fail to Block Devices and OpenStack
 
 Hi,
 
 I fail for the cooperation of Openstack and Ceph.
 I was set on the basis of the url.
 http://ceph.com/docs/next/rbd/rbd-openstack/
 
 Can look at the state of cephcluster from Openstack(cephClient)
 Failure occurs at cinder create
 
 Ceph Cluster:
 CentOS release 6.5
 Ceph 0.80.1
 
 OpenStack:
 Ubuntu 12.04.4
 OpenStack DevStack Icehouse
 
 # glance image-create --name cirros --disk-format raw
 --container-format ovf --file
 /usr/local/src/cirros-0.3.2-x86_64-disk.raw --is-public True
 +--+--+
 | Property | Value|
 +--+--+
 | checksum | cf2392db1f59d59ed69a8f8491b670e0 |
 | container_format | ovf  |
 | created_at   | 2014-06-09T05:04:48  |
 | deleted  | False|
 | deleted_at   | None |
 | disk_format  | raw  |
 | id   | f4a0f971-437b-4d3f-a0c4-1c82f31e9f1e |
 | is_public| True |
 | min_disk | 0|
 | min_ram  | 0|
 | name | cirros   |
 | owner| 5a10a1fed82b45a7affaf57f814434bb |
 | protected| False|
 | size | 41126400 |
 | status   | active   |
 | updated_at   | 2014-06-09T05:04:50  |
 | virtual_size | None |
 +--+--+
 
 
 # cinder create --image-id f4a0f971-437b-4d3f-a0c4-1c82f31e9f1e
 --display-name boot-from-rbd 1
 ++--+
 |Property|Value
 ||
 ++--+
 |  attachments   |  []
 |   |
 |   availability_zone| nova
 |   |
 |bootable|false
 ||
 |   created_at   |  2014-06-09T05:12:51.00
 ||
 |  description   | None
 |  |
 |   encrypted|False
 |   |
 |   id   |
 |   30d1eee7-54d6-4911-af06-b35d2f8ef0c4 |
 |metadata|  {}
 | |
 |  name  |boot-from-rbd
 |  |
 | os-vol-host-attr:host  | None
 | |
 | os-vol-mig-status-attr:migstat | None
 | |
 | os-vol-mig-status-attr:name_id | None
 | |
 |  os-vol-tenant-attr:tenant_id  |   5a10a1fed82b45a7affaf57f814434bb
 ||
 |  size  |  1
 ||
 |  snapshot_id   | None
 |  |
 |  source_volid

Re: [ceph-users] Problem installing ceph from package manager / ceph repositories

[Karan Singh]

Thanks Dan / Alfredo

Yep priority=1  helped a little bit and yum-plugin-priority fixed all.

- karan -

On 10 Jun 2014, at 15:22, Alfredo Deza alfredo.d...@inktank.com wrote:

 On Tue, Jun 10, 2014 at 4:34 AM, Dan Van Der Ster
 daniel.vanders...@cern.ch wrote:
 Hi,
 
 On 10 Jun 2014, at 10:30, Karan Singh karan.si...@csc.fi wrote:
 
 Hello Cephers
 
 First of all this problem is not related to ceph-deploy , ceph-deploy 1.5.4
 works like charm :-) , thanks for Alfredo
 
 Problem :
 
 1. When installing Ceph using package manger ( # yum install ceph  or  # yum
 update cehp) that uses ceph repositories (cep.repo) , the package manager
 does not respect the ceph.repo file and takes ceph package directly from
 EPEL . But when i disable EPEL repo and again try to install ceph , it takes
 from ceph repo.  This is a new problem and earlier i have not faced this ,
 though i have done several ceph cluster installation with package manager. I
 don’t want EPEL version of Ceph.
 
 
 You probably need to tweak the repo priorities. We use priority=30 for
 epel.repo, priority=5 for ceph.repo.
 
 
 This is exactly what ceph-deploy is now doing. It is installing the
 priorities plugin for yum
 and then setting a very low number for ceph.repo (1 I believe)
 
 This is explained in the last release announcement for ceph-deploy and
 for more details
 you can see this ticket:  http://tracker.ceph.com/issues/8533
 
 Cheers, Dan
 
 
 -- Dan van der Ster || Data  Storage Services || CERN IT Department --
 
 
 2. When i uninstall ceph , from system its repository file i.e. ceph.repo is
 getting removed . Is this normal , i have not seen this before.
 
 
 Any one else facing the same problem or have solution to this.
 
 
 Regards
 Karan Singh
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-deploy 1.5.4 (addressing packages coming from EPEL)

[Karan Singh]

Thanks Alfredo , happy to see your email.

I was a victim of this problem , hope 1.5.4 will take away my pain :-)


- Karan Sing -

On 09 Jun 2014, at 15:33, Alfredo Deza alfredo.d...@inktank.com wrote:

 http://ceph.com/ceph-deploy/docs/changelog.html#id1

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] pgs incomplete; pgs stuck inactive; pgs stuck unclean

[Karan Singh]

Try increasing the placement groups for pools

ceph osd pool set data pg_num 128  
ceph osd pool set data pgp_num 128

similarly for other 2 pools as well.

- karan -


On 23 May 2014, at 11:50, jan.zel...@id.unibe.ch wrote:

 Dear ceph,
 
 I am trying to setup ceph 0.80.1 with the following components :
 
 1 x mon - Debian Wheezy (i386)
 3 x osds - Debian Wheezy (i386)
 
 (all are kvm powered)
 
 Status after the standard setup procedure :
 
 root@ceph-node2:~# ceph -s
cluster d079dd72-8454-4b4a-af92-ef4c424d96d8
 health HEALTH_WARN 192 pgs incomplete; 192 pgs stuck inactive; 192 pgs 
 stuck unclean
 monmap e1: 1 mons at {ceph-node1=192.168.123.48:6789/0}, election epoch 
 2, quorum 0 ceph-node1
 osdmap e11: 3 osds: 3 up, 3 in
  pgmap v18: 192 pgs, 3 pools, 0 bytes data, 0 objects
103 MB used, 15223 MB / 15326 MB avail
 192 incomplete
 
 root@ceph-node2:~# ceph health
 HEALTH_WARN 192 pgs incomplete; 192 pgs stuck inactive; 192 pgs stuck unclean
 
 root@ceph-node2:~# ceph osd tree
 # idweight  type name   up/down reweight
 -1  0   root default
 -2  0   host ceph-node2
 0   0   osd.0   up  1
 -3  0   host ceph-node3
 1   0   osd.1   up  1
 -4  0   host ceph-node4
 2   0   osd.2   up  1
 
 
 root@ceph-node2:~# ceph osd dump
 epoch 11
 fsid d079dd72-8454-4b4a-af92-ef4c424d96d8
 created 2014-05-23 09:00:08.780211
 modified 2014-05-23 09:01:33.438001
 flags 
 
 pool 0 'data' replicated size 3 min_size 2 crush_ruleset 0 object_hash 
 rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 flags hashpspool 
 crash_replay_interval 45 stripe_width 0
 
 pool 1 'metadata' replicated size 3 min_size 2 crush_ruleset 0 object_hash 
 rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 flags hashpspool 
 stripe_width 0
 
 pool 2 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash 
 rjenkins pg_num 64 pgp_num 64 last_change 1 owner 0 flags hashpspool 
 stripe_width 0 max_osd 3
 
 osd.0 up   in  weight 1 up_from 4 up_thru 5 down_at 0 last_clean_interval 
 [0,0) 192.168.123.49:6800/11373 192.168.123.49:6801/11373 
 192.168.123.49:6802/11373 192.168.123.49:6803/11373 exists,up 
 21a7d2a8-b709-4a28-bc3b-850913fe4c6b
 
 osd.1 up   in  weight 1 up_from 8 up_thru 0 down_at 0 last_clean_interval 
 [0,0) 192.168.123.50:6800/10542 192.168.123.50:6801/10542 
 192.168.123.50:6802/10542 192.168.123.50:6803/10542 exists,up 
 c1cd3ad1-b086-438f-a22d-9034b383a1be
 
 osd.2 up   in  weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval 
 [0,0) 192.168.123.53:6800/6962 192.168.123.53:6801/6962 
 192.168.123.53:6802/6962 192.168.123.53:6803/6962 exists,up 
 aa06d7e4-181c-4d70-bb8e-018b088c5053
 
 
 What am I doing wrong here ?
 Or what kind of additional information should be provided to get 
 troubleshooted.
 
 thanks,
 
 ---
 
 Jan
 
 P.S. with emperor 0.72.2 I had no such problems
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Screencast/tutorial on setting up Ceph

[Karan Singh]

use my blogs if you like   
http://karan-mj.blogspot.fi/2013/12/ceph-storage-part-2.html 

- Karan Singh -

On 23 May 2014, at 12:30, jan.zel...@id.unibe.ch jan.zel...@id.unibe.ch 
wrote:

 -Ursprüngliche Nachricht-
 Von: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] Im Auftrag
 von Sankar P
 Gesendet: Freitag, 23. Mai 2014 11:14
 An: ceph-users@lists.ceph.com
 Betreff: [ceph-users] Screencast/tutorial on setting up Ceph
 
 Hi,
 
 I have four old machines lying around. I would like to setup ceph on these
 machines.
 
 Are there any screencast or tutorial with commands, on how to obtain,
 install and configure on ceph on these machines ?
 
 The official documentation page OS Recommendations seem to list only
 old distros and not the new version of distros (openSUSE and Ubuntu).
 
 So I wanted to ask if there is a screencast or tutorial or techtalk on how to
 setup Ceph for a total newbie ?
 
 --
 Sankar P
 http://psankar.blogspot.com
 
 Hi,
 
 I am rookie too and only used just this : 
 http://ceph.com/docs/master/start/
 
 it's a very nice doc
 
 ---
 
 jan
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 70+ OSD are DOWN and not coming up

[Karan Singh]

Hello Sage

nodown, noout set on cluster

# ceph status
cluster 009d3518-e60d-4f74-a26d-c08c1976263c
 health HEALTH_WARN 1133 pgs degraded; 44 pgs incomplete; 42 pgs stale; 45 
pgs stuck inactive; 42 pgs stuck stale; 2602 pgs stuck unclean; recovery 
206/2199 objects degraded (9.368%); 40/165 in osds are down; nodown,noout 
flag(s) set
 monmap e4: 4 mons at 
{storage0101-ib=192.168.100.101:6789/0,storage0110-ib=192.168.100.110:6789/0,storage0114-ib=192.168.100.114:6789/0,storage0115-ib=192.168.100.115:6789/0},
 election epoch 18, quorum 0,1,2,3 
storage0101-ib,storage0110-ib,storage0114-ib,storage0115-ib
 osdmap e358031: 165 osds: 125 up, 165 in
flags nodown,noout
  pgmap v604305: 4544 pgs, 6 pools, 4309 MB data, 733 objects
3582 GB used, 357 TB / 361 TB avail
206/2199 objects degraded (9.368%)
   1 inactive
   5 stale+active+degraded+remapped
1931 active+clean
   2 stale+incomplete
  21 stale+active+remapped
 380 active+degraded+remapped
  38 incomplete
1403 active+remapped
   2 stale+active+degraded
   1 stale+remapped+incomplete
 746 active+degraded
  11 stale+active+clean
   3 remapped+incomplete


 Here is my ceph.conf  http://pastebin.com/KZdgPJm7  (debus osd , ms set )
I tried restarting all OSD services of  node-13 , services came up after 
several attempts of “service ceph restart”   http://pastebin.com/yMk86YHh
For Node : 14
All services are up

[root@storage0114-ib ~]# service ceph status
=== osd.142 ===
osd.142: running {version:0.80-475-g9e80c29}
=== osd.36 ===
osd.36: running {version:0.80-475-g9e80c29}
=== osd.83 ===
osd.83: running {version:0.80-475-g9e80c29}
=== osd.107 ===
osd.107: running {version:0.80-475-g9e80c29}
=== osd.47 ===
osd.47: running {version:0.80-475-g9e80c29}
=== osd.130 ===
osd.130: running {version:0.80-475-g9e80c29}
=== osd.155 ===
osd.155: running {version:0.80-475-g9e80c29}
=== osd.60 ===
osd.60: running {version:0.80-475-g9e80c29}
=== osd.118 ===
osd.118: running {version:0.80-475-g9e80c29}
=== osd.98 ===
osd.98: running {version:0.80-475-g9e80c29}
=== osd.70 ===
osd.70: running {version:0.80-475-g9e80c29}
=== mon.storage0114-ib ===
mon.storage0114-ib: running {version:0.80-475-g9e80c29}
[root@storage0114-ib ~]#

— But ceph osd tree says , osd.118  is down

-10 29.93   host storage0114-ib
36  2.63osd.36  up  1
47  2.73osd.47  up  1
60  2.73osd.60  up  1
70  2.73osd.70  up  1
83  2.73osd.83  up  1
98  2.73osd.98  up  1
107 2.73osd.107 up  1
118 2.73osd.118 down1
130 2.73osd.130 up  1
142 2.73osd.142 up  1
155 2.73osd.155 up  1

— I restarted osd.118 service and it was successful , But still its showing as 
down in ceph osd tree . I waited for 30 minutes to get it stable but still not 
showing UP in ceph osd tree. 
Moreover its generating HUGE logs http://pastebin.com/mDYnjAni



The problem now is if i manually visit every host and check for “service ceph 
status “ all services are running on all 15 hosts. But this is not getting 
reflected to ceph osd tree and ceph -s and they continue to show as OSD DOWN.

My irc id is ksingh , let me know by email once you are available on IRC (my 
time zone is Finland +2)



- Karan Singh - 

On 20 May 2014, at 18:18, Sage Weil s...@inktank.com wrote:

 On Tue, 20 May 2014, Karan Singh wrote:
 Hello Cephers , need your suggestion for troubleshooting.
 
 My cluster is terribly struggling , 70+ osd are down out of 165
 
 Problem ?OSD are getting marked out of cluster and are down. The cluster is
 degraded. On checking logs of failed OSD we are getting wired entries that
 are continuously getting generated.
 
 Tracking this at http://tracker.ceph.com/issues/8387
 
 The most recent bits you posted in the ticket don't quite make sense: the 
 OSD is trying to connect to an address for an OSD that is currently marked 
 down.  I suspect this is just timing between when the logs were captured 
 and when teh ceph osd dump was captured.  To get a complete pictures, 
 please:
 
 1) add
 
 debug osd = 20
 debug ms = 1
 
 in [osd] and restart all osds
 
 2) ceph osd set nodown
 
 (to prevent flapping)
 
 3) find some OSD that is showing these messages
 
 4) capture a 'ceph osd dump' output.
 
 Also happy to debug this interactively over IRC; that will likely be 
 faster!
 
 Thanks-
 sage
 
 
 
 
 Osd Debug logs ::  http://pastebin.com/agTKh6zB
 
 
 1. 2014-05-20 10:19:03.699886 7f2328e237a0  0 osd.158 357532 done with
init, starting boot process
 2

[ceph-users] 70+ OSD are DOWN and not coming up

[Karan Singh]

Hello Cephers , need your suggestion for troubleshooting.

My cluster is terribly struggling , 70+ osd are down out of 165

Problem — OSD are getting marked out of cluster and are down. The cluster is 
degraded. On checking logs of failed OSD we are getting wired entries that are 
continuously getting generated.

Osd Debug logs ::  http://pastebin.com/agTKh6zB


2014-05-20 10:19:03.699886 7f2328e237a0  0 osd.158 357532 done with init, 
starting boot process
2014-05-20 10:19:03.700093 7f22ff621700  0 -- 192.168.1.112:6802/3807  
192.168.1.109:6802/910005982 pipe(0x8698500 sd=35 :33500 s=1 pgs=0 cs=0 l=0 
c=0x83018c0).connect claims to be 192.168.1.109:6802/63896 not 
192.168.1.109:6802/910005982 - wrong node!
2014-05-20 10:19:03.700152 7f22ff621700  0 -- 192.168.1.112:6802/3807  
192.168.1.109:6802/910005982 pipe(0x8698500 sd=35 :33500 s=1 pgs=0 cs=0 l=0 
c=0x83018c0).fault with nothing to send, going to standby
2014-05-20 10:19:09.551269 7f22fdd12700  0 -- 192.168.1.112:6802/3807  
192.168.1.109:6803/1176009454 pipe(0x56aee00 sd=53 :40060 s=1 pgs=0 cs=0 l=0 
c=0x533fd20).connect claims to be 192.168.1.109:6803/63896 not 
192.168.1.109:6803/1176009454 - wrong node!
2014-05-20 10:19:09.551347 7f22fdd12700  0 -- 192.168.1.112:6802/3807  
192.168.1.109:6803/1176009454 pipe(0x56aee00 sd=53 :40060 s=1 pgs=0 cs=0 l=0 
c=0x533fd20).fault with nothing to send, going to standby
2014-05-20 10:19:09.703901 7f22fd80d700  0 -- 192.168.1.112:6802/3807  
192.168.1.113:6802/13870 pipe(0x56adf00 sd=137 :42889 s=1 pgs=0 cs=0 l=0 
c=0x8302aa0).connect claims to be 192.168.1.113:6802/24612 not 
192.168.1.113:6802/13870 - wrong node!
2014-05-20 10:19:09.704039 7f22fd80d700  0 -- 192.168.1.112:6802/3807  
192.168.1.113:6802/13870 pipe(0x56adf00 sd=137 :42889 s=1 pgs=0 cs=0 l=0 
c=0x8302aa0).fault with nothing to send, going to standby
2014-05-20 10:19:10.243139 7f22fd005700  0 -- 192.168.1.112:6802/3807  
192.168.1.112:6800/14114 pipe(0x56a8f00 sd=146 :43726 s=1 pgs=0 cs=0 l=0 
c=0x8304780).connect claims to be 192.168.1.112:6800/2852 not 
192.168.1.112:6800/14114 - wrong node!
2014-05-20 10:19:10.243190 7f22fd005700  0 -- 192.168.1.112:6802/3807  
192.168.1.112:6800/14114 pipe(0x56a8f00 sd=146 :43726 s=1 pgs=0 cs=0 l=0 
c=0x8304780).fault with nothing to send, going to standby
2014-05-20 10:19:10.349693 7f22fc7fd700  0 -- 192.168.1.112:6802/3807  
192.168.1.109:6800/13492 pipe(0x8698c80 sd=156 :0 s=1 pgs=0 cs=0 l=0 
c=0x83070c0).fault with nothing to send, going to standby


ceph -v
ceph version 0.80-469-g991f7f1 (991f7f15a6e107b33a24bbef1169f21eb7fcce2c) #
ceph osd stat
osdmap e357073: 165 osds: 91 up, 165 in
flags noout #
I have tried doing :

1. Restarting the problematic OSDs , but no luck
2.  i restarted entire host but no luck, still osds are down and getting the 
same mesage

2014-05-20 10:19:10.243139 7f22fd005700  0 -- 192.168.1.112:6802/3807  
192.168.1.112:6800/14114 pipe(0x56a8f00 sd=146 :43726 s=1 pgs=0 cs=0 l=0 
c=0x8304780).connect claims to be 192.168.1.112:6800/2852 not 
192.168.1.112:6800/14114 - wrong node!
2014-05-20 10:19:10.243190 7f22fd005700  0 -- 192.168.1.112:6802/3807  
192.168.1.112:6800/14114 pipe(0x56a8f00 sd=146 :43726 s=1 pgs=0 cs=0 l=0 
c=0x8304780).fault with nothing to send, going to standby
2014-05-20 10:19:10.349693 7f22fc7fd700  0 -- 192.168.1.112:6802/3807  
192.168.1.109:6800/13492 pipe(0x8698c80 sd=156 :0 s=1 pgs=0 cs=0 l=0 
c=0x83070c0).fault with nothing to send, going to standby
2014-05-20 10:22:23.312473 7f2307e61700  0 osd.158 357781 do_command r=0
2014-05-20 10:22:23.326110 7f2307e61700  0 osd.158 357781 do_command r=0 
debug_osd=0/5
2014-05-20 10:22:23.326123 7f2307e61700  0 log [INF] : debug_osd=0/5
2014-05-20 10:34:08.161864 7f230224d700  0 -- 192.168.1.112:6802/3807  
192.168.1.102:6808/13276 pipe(0x8698280 sd=22 :41078 s=2 pgs=603 cs=1 l=0 
c=0x8301600).fault with nothing to send, going to standby

3. Disks do not have errors , no message in dmesg and /var/log/messages

4. there was a bug in the past http://tracker.ceph.com/issues/4006 , dont know 
it again came bacin in Firefly

5. Recently no activity performed on cluster , except some pool and keys 
creation for cinder /glance integration

6. Nodes have enough free resources for osds.

7. No issues with network , osds are down on all cluster nodes. not from a 
single node.



Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Working at RedHat Ceph User Committee

[Karan Singh]

Nice to hear.

I would say its a good move from RH to hire you :-)   , good luck and keep 
contributing to ceph *core features*
 as you already been doing.

- Karan Singh -

On 19 May 2014, at 12:21, Wido den Hollander w...@42on.com wrote:

 On 05/19/2014 10:46 AM, Loic Dachary wrote:
 Hi Ceph,
 
 TL;DR: I'm starting a new position at RedHat in the Ceph development team, 
 mid-august. I will keep working under the newly elected head of the Ceph 
 User Committee.
 
 Cloudwatt is a company based in France which provides cloud services. My 
 participation in Ceph (and erasure code specifically) was driven by their 
 need for cheaper storage. I'm extremely grateful for the opportunity 
 Cloudwatt gave me to exclusively work upstream. Now that Firefly is released 
 and packaged, Cloudwatt will be able to deploy it in production and indeed 
 save money. A lot more money than my salary, which is a sign of a sound 
 relationship. There still is work to be done and I'll focus on pyramid codes 
 for Giant. After that I'm not sure but I trust Samuel Just will find ways to 
 keep me busy.
 
 When RedHat acquired Inktank my first reaction was : Calamari will be Free 
 Software, no more proprietary software !. The Atlanta OpenStack summit was 
 the opportunity to discuss the consequences of the acquisition with dozens 
 of people and I've not heard anyone say it was bad for the Ceph project. 
 However  I've heard about four problems worth considering: a) communication 
 about the Ceph project is going to be driven by RedHat marketing b) there is 
 no incentive for RedHat to establish a foundation, c) RedHat has much less 
 incentive than Inktank to support multiple GNU/Linux distributions, d) 
 Inktank customers are worried about the transition and need to be reassured.
 
 I also selfishly thought that RedHat/Inktank became a very appealing 
 workplace. I'm committed to exclusively work on Free Software and RedHat is 
 one of the few companies in the world where that is possible. I discussed it 
 with Ian Colle  Sage Weil, sent a job application and was accepted. Since 
 I've been working with the core team for some time now, it will not change 
 much in terms of what I do. But one difference matters to me : I will get to 
 hear about a variety of Ceph use cases that were previously kept secret 
 because Inktank would not discuss about their customers with an external 
 contributor.
 
 
 Congratulations Loic!
 
 The downside of being hired by RedHat is that I'm no longer a community 
 member and my judgment will be biased. I'm told RedHat employees are 
 encouraged to have their own opinion and express them as I just did. But no 
 matter how hard I try, I will gradually become less objective about the 
 company paying my salary: it is the root of all conflicts of interest. To 
 acknowledge that and keep working for the Ceph User Committee, I will not be 
 a candidate to the next elections. I however pledge to assist the newly 
 elected head of the Ceph User Committee, i.e. doing what I've done so far, 
 only without the title nor the authority.
 
 
 I very much appreciate that you are brave enough to admit this yourself. 
 Indeed, your vision on certain things will be 'clouded' ;) by the company who 
 is paying your salary, but that's logical as well.
 
 Good luck at Red Hat and keep improving Ceph!
 
 Cheers
 
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 
 -- 
 Wido den Hollander
 42on B.V.
 Ceph trainer and consultant
 
 Phone: +31 (0)20 700 9902
 Skype: contact42on
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph manual deploy doesnt start at step 15

[Karan Singh]

Hi Alexander

Try adding  your monitor details in  /etc/ceph/ceph.conf file (please check for 
typos)


[mon]

[mon.nfs2.abboom.world]
host = nfs2.abboom.world
mon addr = 10.60.0.111:6789

[mon.nfs3.abboom.world]
host = nfs3.abboom.world
mon addr = 10.60.0.112:6789

[mon.nfs4.abboom.world]
host = nfs4.abboom.world
mon addr = 10.60.0.113:6789


- karan -

On 21 Apr 2014, at 14:46, *sm1Ly st.uz...@gmail.com wrote:

 re2all. I use centos 6.5. this is my ceph.conf http://pastebin.com/0UCevzF5  
 I doing manual deploying with this guide 
 http://ceph.com/docs/master/install/manual-deployment/ when I get to step 15 
 I cant start my node, cause it doesnt defined in my conf. like this: 
 /etc/init.d/ceph start mon.nfs2.abboom.world /etc/init.d/ceph: 
 mon.nfs2.abboom.world not found (/etc/ceph/ceph.conf defines , /var/lib/ceph 
 defines )  what I m doing wrong?
 
 
 
 -- 
 yours respectfully, Alexander Vasin.
 
 8 926 1437200
 icq: 9906064
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd: add failed: (34) Numerical result out of range ( Please help me)

[Karan Singh]

You ceph.conf please 


Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


On 16 Apr 2014, at 13:13, Srinivasa Rao Ragolu srag...@mvista.com wrote:

 Thanks. Please see the output of above command
 
 root@mon:/etc/ceph# rbd ls -l 
 rbd: error opening blk2: (95) Operation not supported2014-04-16 
 10:12:13.947625 7f3a2a0c7780 -1 librbd: Error listing snapshots: (95) 
 Operation not supported
 
 rbd: error opening blk3: (95) Operation not supported2014-04-16 
 10:12:13.961595 7f3a2a0c7780 -1 librbd: Error listing snapshots: (95) 
 Operation not supported
 
 rbd: error opening ceph-block1: (95) Operation not supported2014-04-16 
 10:12:13.974869 7f3a2a0c7780 -1 librbd: Error listing snapshots: (95) 
 Operation not supported
 
 rbd: error opening sample: (95) Operation not supported
 NAME SIZE PARENT FMT PROT LOCK 
 2014-04-16 10:12:13.986056 7f3a2a0c7780 -1 librbd: Error listing snapshots: 
 (95) Operation not supported
 
 
 Thanks,
 Srinivas.
 
 
 On Wed, Apr 16, 2014 at 3:37 PM, Ирек Фасихов malm...@gmail.com wrote:
 Show command output rbd ls -l.
 
 
 2014-04-16 13:59 GMT+04:00 Srinivasa Rao Ragolu srag...@mvista.com:
 
 Hi Wido,
 
 Output of info command is given below
 
 root@mon:/etc/ceph# rbd info sample
 rbd: error opening image sample: (95) Operation not supported2014-04-16 
 09:57:24.575279 7f661c6e5780 -1 librbd: Error listing snapshots: (95) 
 Operation not supported
 
 root@mon:/etc/ceph# ceph status
 cluster a7f64266-0894-4f1e-a635-d0aeaca0e993
  health HEALTH_OK
  monmap e1: 1 mons at {mon=192.168.0.102:6789/0}, election epoch 1, 
 quorum 0 mon
  osdmap e13: 2 osds: 2 up, 2 in
   pgmap v68: 192 pgs, 3 pools, 513 bytes data, 5 objects
 2077 MB used, 9113 MB / 11837 MB avail
  192 active+clean
   client io 13 B/s rd, 0 op/s
 
 After this monitor daemon getting killed. Need to start it again.
 
 Thanks,
 Srinivas.
 
 
 On Wed, Apr 16, 2014 at 3:18 PM, Wido den Hollander w...@42on.com wrote:
 On 04/16/2014 11:41 AM, Srinivasa Rao Ragolu wrote:
 HI all,
 
 I have created ceph cluster with 1 monitor node and 2 OSd nodes. Cluster
 health is OK and Active.
 
 My deployment is on our private distribution of Linux kernel 3.10.33 and
 ceph version is 0.72.2
 
 I could able to create image with command  rbd create sample --size 200.
 
 What is the RBD format of the image?
 
 $ rbd info sample
 
 I don't think it's the problem, but it could be that the krbd doesn't support 
 format 2 yet.
 
 inserted rbd.ko successfully with modprobe command  modprobe rbd
 
 Now when I try to map it with command:
 
 #*rbd map sample
 *
 *[10584.497492] libceph: client4301 fsid
 
 a7f64266-0894-4f1e-a635-d0aeaca0e993
 [10584.535926] libceph: mon0 192.168.0.102:6789
 http://192.168.0.102:6789 session established
 rbd: add failed: (34) Numerical result out of range*
 
 
 
 Please help me in solving this issue.
 
 Thanks,
 Srinivas.
 
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 
 -- 
 Wido den Hollander
 Ceph consultant and trainer
 42on B.V.
 
 Phone: +31 (0)20 700 9902
 Skype: contact42on
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 
 
 -- 
 С уважением, Фасихов Ирек Нургаязович
 Моб.: +79229045757
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph v0.79 Firefly RC :: erasure-code-profile command set not present

[Karan Singh]

Finally everything worked with ceph version 0.79-125 .

I agree with you , version 0.79 does have erasure-code-profile command set , 
But This mess was due to ceph init script which was missing 
/lib/lsb/init-functions” file which is blocking ceph services to get started.

Thanks Sage / Alfredo for your timely support.


Karan Singh 
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


On 08 Apr 2014, at 18:01, Sage Weil s...@inktank.com wrote:

 On Tue, 8 Apr 2014, Karan Singh wrote:
 Thanks Developers for 0.79 , happy to see its announcement.
 I have been using 0.78 in order to test EC and TP , with 0.78 i was not able
 to test erasure code profile feature  ( # ceph osd erasure-code-profile
 set myprofile ??) as erasure-code-profile command set is not present in 0.78 
 but
 its mentioned in documentation.
 
 http://ceph.com/docs/master/dev/erasure-coded-pool/#interface
 https://ceph.com/docs/v0.78/dev/erasure-coded-pool/#interface
 https://ceph.com/docs/v0.79/dev/erasure-coded-pool/#interface
 
 I tried the same thing on 0.79 , but again looks like ? ceph osd
 erasure-code-profile ? command set is not implemented here as well.
 
 Do we need to wait for 0.80 release to test erasure-code-profile  feature
 ?? 
 
 
 # cat /etc/redhat-release
 CentOS release 6.4 (Final)
 
 # ceph -v
 ceph version 0.79 (4c2d73a5095f527c3a2168deb5fa54b3c8991a6e)
 
 # ceph osd erasure-code-profile
 no valid command found; 10 closest matches:
 
 Did you restart the ceph-mon daemons?
 
 sage
 
 
 osd tier add-cache poolname poolname int[0-]
 osd tier set-overlay poolname poolname
 osd tier remove-overlay poolname
 osd tier remove poolname poolname
 osd tier cache-mode poolname none|writeback|forward|readonly
 osd thrash int[0-]
 osd tier add poolname poolname {--force-nonempty}
 osd stat
 osd reweight-by-utilization {int[100-]}
 osd pool stats {name}
 Error EINVAL: invalid command
 #
 
 
 
 
 Karan Singh 
 CSC - IT Center for Science,
 Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
 mobile: +358 503 812758
 tel. +358 9 4572001
 fax +358 9 4572302
 http://www.csc.fi/
 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph v0.79 Firefly RC :: erasure-code-profile command set not present

[Karan Singh]

Thanks Developers for 0.79 , happy to see its announcement.

I have been using 0.78 in order to test EC and TP , with 0.78 i was not able to 
test erasure code profile feature  ( # ceph osd erasure-code-profile set 
myprofile ……) as erasure-code-profile command set is not present in 0.78 but 
its mentioned in documentation.

http://ceph.com/docs/master/dev/erasure-coded-pool/#interface
https://ceph.com/docs/v0.78/dev/erasure-coded-pool/#interface
https://ceph.com/docs/v0.79/dev/erasure-coded-pool/#interface

I tried the same thing on 0.79 , but again looks like “ ceph osd 
erasure-code-profile “ command set is not implemented here as well.

Do we need to wait for 0.80 release to test erasure-code-profile  feature ?? 


# cat /etc/redhat-release
CentOS release 6.4 (Final)

# ceph -v
ceph version 0.79 (4c2d73a5095f527c3a2168deb5fa54b3c8991a6e)

# ceph osd erasure-code-profile
no valid command found; 10 closest matches:
osd tier add-cache poolname poolname int[0-]
osd tier set-overlay poolname poolname
osd tier remove-overlay poolname
osd tier remove poolname poolname
osd tier cache-mode poolname none|writeback|forward|readonly
osd thrash int[0-]
osd tier add poolname poolname {--force-nonempty}
osd stat
osd reweight-by-utilization {int[100-]}
osd pool stats {name}
Error EINVAL: invalid command
#




Karan Singh 
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph v0.79 Firefly RC :: erasure-code-profile command set not present

[Karan Singh]

Thanks Mark for this quick checkup on Ubuntu distro. you saved my time , i was 
also checking the same thing.

All of my ceph components are on 0.79 , this looks to be a problem with ceph 
binaries for RHEL.

@ Ceph Developers :  Any thought on this.

# rpm -qa | egrep -i ceph|rados|rbd
ceph-deploy-1.4.0-0.noarch
librbd1-0.79-0.el6.x86_64
libcephfs1-0.79-0.el6.x86_64
ceph-release-1-0.el6.noarch
python-ceph-0.79-0.el6.x86_64
librados2-0.79-0.el6.x86_64
ceph-0.79-0.el6.x86_64
#


Also updated the latest latest ceph release 0.79-75 ( on a different machine ) 
, but still no LUCK.


# rpm -qa | egrep -i ceph|rados|rbd
libcephfs1-0.79-75.g2a6d962.el6.x86_64
ceph-deploy-1.4.0-0.noarch
librbd1-0.79-75.g2a6d962.el6.x86_64
librados2-0.79-75.g2a6d962.el6.x86_64
ceph-0.79-75.g2a6d962.el6.x86_64
ceph-release-1-0.el6.noarch
python-ceph-0.79-75.g2a6d962.el6.x86_64

# ceph osd erasure-code-profile ls
no valid command found; 10 closest matches:
osd tier add-cache poolname poolname int[0-]
osd tier set-overlay poolname poolname
osd tier remove-overlay poolname
osd tier remove poolname poolname
osd tier cache-mode poolname none|writeback|forward|readonly
osd thrash int[0-]
osd tier add poolname poolname {--force-nonempty}
osd stat
osd reweight-by-utilization {int[100-]}
osd pool stats {name}
Error EINVAL: invalid command
#




Karan Singh 
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


On 08 Apr 2014, at 12:28, Mark Kirkwood mark.kirkw...@catalyst.net.nz wrote:

 Wow - that is a bit strange:
 
 $ cat /etc/issue
 Ubuntu 13.10 \n \l
 
 $ sudo ceph -v
 ceph version 0.78-569-g6a4c50d (6a4c50d7f27d2e7632d8c017d09e864e969a05f7)
 
 $ sudo ceph osd erasure-code-profile ls
 default
 myprofile
 profile
 profile1
 
 I'd hazard a guess that some of your ceph components are at version  0.78 
 still (partial upgrade or similar).
 
 Regards
 
 Mark
 
 On 08/04/14 20:57, Karan Singh wrote:
 Thanks Developers for 0.79 , happy to see its announcement.
 
 I have been using 0.78 in order to test EC and TP , with 0.78 i was not
 able to test erasure code profile feature  ( # ceph osd
 erasure-code-profile set myprofile……) as erasure-code-profile command
 set is not present in 0.78 but its mentioned in documentation.
 
 http://ceph.com/docs/master/dev/erasure-coded-pool/#interface
 https://ceph.com/docs/v0.78/dev/erasure-coded-pool/#interface
 https://ceph.com/docs/v0.79/dev/erasure-coded-pool/#interface
 
 I tried the same thing on 0.79 , but again looks like “ ceph osd
 erasure-code-profile “ command set is not implemented here as well.
 
 Do we need to wait for 0.80 release to test erasure-code-profile
  feature ??
 
 
 # cat /etc/redhat-release
 CentOS release 6.4 (Final)
 
 # ceph -v
 ceph version 0.79 (4c2d73a5095f527c3a2168deb5fa54b3c8991a6e)
 
 # ceph osd erasure-code-profile
 no valid command found; 10 closest matches:
 osd tier add-cache poolname poolname int[0-]
 osd tier set-overlay poolname poolname
 osd tier remove-overlay poolname
 osd tier remove poolname poolname
 osd tier cache-mode poolname none|writeback|forward|readonly
 osd thrash int[0-]
 osd tier add poolname poolname {--force-nonempty}
 osd stat
 osd reweight-by-utilization {int[100-]}
 osd pool stats {name}
 Error EINVAL: invalid command
 #
 
 
 
 
 Karan Singh
 CSC - IT Center for Science,
 Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
 mobile: +358 503 812758
 tel. +358 9 4572001
 fax +358 9 4572302
 http://www.csc.fi/
 
 
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph v0.79 Firefly RC :: erasure-code-profile command set not present

[Karan Singh]

Hi Loic

Here is the output

# ceph --version
ceph version 0.79 (4c2d73a5095f527c3a2168deb5fa54b3c8991a6e)
#

# ceph osd erasure-code-profile
no valid command found; 10 closest matches:
osd tier add-cache poolname poolname int[0-]
osd tier set-overlay poolname poolname
osd tier remove-overlay poolname
osd tier remove poolname poolname
osd tier cache-mode poolname none|writeback|forward|readonly
osd thrash int[0-]
osd tier add poolname poolname {--force-nonempty}
osd stat
osd reweight-by-utilization {int[100-]}
osd pool stats {name}
Error EINVAL: invalid command

# cat /etc/redhat-release
CentOS release 6.4 (Final)



# ceph status
cluster e8768ef6-93a1-4e8c-acda-3ac995e35003
 health HEALTH_OK
 monmap e3: 3 mons at 
{storage0101-ib=192.168.100.101:6789/0,storage0107-ib=192.168.100.107:6789/0,storage0115-ib=192.168.100.115:6789/0},
 election epoch 10, quorum 0,1,2 storage0101-ib,storage0107-ib,storage0115-ib
 mdsmap e14: 0/0/1 up
 osdmap e656: 165 osds: 165 up, 165 in
  pgmap v2818: 4288 pgs, 21 pools, 51212 kB data, 23 objects
15854 MB used, 418 TB / 418 TB avail
 395 stale+active+clean
3893 active+clean



Karan Singh 
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


On 08 Apr 2014, at 13:08, Loic Dachary l...@dachary.org wrote:

 Hi Karan,
 
 What is the output of  ceph --version ?
 
 Cheers
 
 On 08/04/2014 11:44, Karan Singh wrote:
 Thanks Mark for this quick checkup on Ubuntu distro. you saved my time , i 
 was also checking the same thing.
 
 All of my ceph components are on 0.79 , this looks to be a problem with ceph 
 binaries for RHEL.
 
 @ Ceph Developers :  Any thought on this.
 
 # rpm -qa | egrep -i ceph|rados|rbd
 ceph-deploy-1.4.0-0.noarch
 librbd1-0.79-0.el6.x86_64
 libcephfs1-0.79-0.el6.x86_64
 ceph-release-1-0.el6.noarch
 python-ceph-0.79-0.el6.x86_64
 librados2-0.79-0.el6.x86_64
 ceph-0.79-0.el6.x86_64
 #
 
 
 Also updated the latest latest ceph release 0.79-75 ( on a different machine 
 ) , but still no LUCK.
 
 
 # rpm -qa | egrep -i ceph|rados|rbd
 libcephfs1-0.79-75.g2a6d962.el6.x86_64
 ceph-deploy-1.4.0-0.noarch
 librbd1-0.79-75.g2a6d962.el6.x86_64
 librados2-0.79-75.g2a6d962.el6.x86_64
 ceph-0.79-75.g2a6d962.el6.x86_64
 ceph-release-1-0.el6.noarch
 python-ceph-0.79-75.g2a6d962.el6.x86_64
 
 # ceph osd erasure-code-profile ls
 no valid command found; 10 closest matches:
 osd tier add-cache poolname poolname int[0-]
 osd tier set-overlay poolname poolname
 osd tier remove-overlay poolname
 osd tier remove poolname poolname
 osd tier cache-mode poolname none|writeback|forward|readonly
 osd thrash int[0-]
 osd tier add poolname poolname {--force-nonempty}
 osd stat
 osd reweight-by-utilization {int[100-]}
 osd pool stats {name}
 Error EINVAL: invalid command
 #
 
 
 
 
 Karan Singh 
 CSC - IT Center for Science,
 Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
 mobile: +358 503 812758
 tel. +358 9 4572001
 fax +358 9 4572302
 http://www.csc.fi/
 
 
 On 08 Apr 2014, at 12:28, Mark Kirkwood mark.kirkw...@catalyst.net.nz 
 mailto:mark.kirkw...@catalyst.net.nz wrote:
 
 Wow - that is a bit strange:
 
 $ cat /etc/issue
 Ubuntu 13.10 \n \l
 
 $ sudo ceph -v
 ceph version 0.78-569-g6a4c50d (6a4c50d7f27d2e7632d8c017d09e864e969a05f7)
 
 $ sudo ceph osd erasure-code-profile ls
 default
 myprofile
 profile
 profile1
 
 I'd hazard a guess that some of your ceph components are at version  0.78 
 still (partial upgrade or similar).
 
 Regards
 
 Mark
 
 On 08/04/14 20:57, Karan Singh wrote:
 Thanks Developers for 0.79 , happy to see its announcement.
 
 I have been using 0.78 in order to test EC and TP , with 0.78 i was not
 able to test erasure code profile feature  ( # ceph osd
 erasure-code-profile set myprofile……) as erasure-code-profile command
 set is not present in 0.78 but its mentioned in documentation.
 
 http://ceph.com/docs/master/dev/erasure-coded-pool/#interface
 https://ceph.com/docs/v0.78/dev/erasure-coded-pool/#interface
 https://ceph.com/docs/v0.79/dev/erasure-coded-pool/#interface
 
 I tried the same thing on 0.79 , but again looks like “ ceph osd
 erasure-code-profile “ command set is not implemented here as well.
 
 Do we need to wait for 0.80 release to test erasure-code-profile
 feature ??
 
 
 # cat /etc/redhat-release
 CentOS release 6.4 (Final)
 
 # ceph -v
 ceph version 0.79 (4c2d73a5095f527c3a2168deb5fa54b3c8991a6e)
 
 # ceph osd erasure-code-profile
 no valid command found; 10 closest matches:
 osd tier add-cache poolname poolname int[0-]
 osd tier set-overlay poolname poolname
 osd tier remove-overlay poolname
 osd tier remove

Re: [ceph-users] Backup Restore?

[Karan Singh]

Hi Robert

Thanks for raising this question , backup and restores options has always been 
interesting to discuss. i too have a connected question for Inktank.

— Is there any work going for support of ceph cluster getting backed by 
enterprise *proprietary* backup solutions available today 



Karan Singh 
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


On 02 Apr 2014, at 10:08, Robert Sander r.san...@heinlein-support.de wrote:

 Hi,
 
 what are the options to consistently backup and restore
 data out of a ceph cluster?
 
 - RBDs can be snapshotted.
 - Data on RBDs used inside VMs can be backed up using tools from the guest.
 - CephFS data can be backed up using rsync are similar tools
 
 What about object data in other pools?
 
 There are two scenarios where a backup is needed:
 
 - disaster recovery, i.e. the while cluster goes nuts
 - single item restore, because PEBKAC or application error
 
 Is there any work on progress to cover these?
 
 Regards
 -- 
 Robert Sander
 Heinlein Support GmbH
 Schwedter Str. 8/9b, 10119 Berlin
 
 http://www.heinlein-support.de
 
 Tel: 030 / 405051-43
 Fax: 030 / 405051-19
 
 Zwangsangaben lt. §35a GmbHG: 
 HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
 Geschäftsführer: Peer Heinlein -- Sitz: Berlin
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RDO - CEPH

[Karan Singh]

Check if RDO is properly configured to use ceph , also look for MON and OSD 
logs at cep bend , you should get some messages from when qemu fails to crate 
volume



Karan Singh 
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 503 812758
tel. +358 9 4572001
fax +358 9 4572302
http://www.csc.fi/


On 31 Mar 2014, at 22:29, Vilobh Meshram vilob...@yahoo-inc.com wrote:

 What logs should I see to explore more about what might be going wrong
 here ?
 
 Thanks,
 Vilobh
 
 On 3/30/14, 3:00 PM, Vilobh Meshram vilob...@yahoo-inc.com wrote:
 
 Hi Loic,
 
 Thanks for your reply.
 Not really I have setup 3 nodes as storage nodes ³ceph osd tree² output
 also confirms that.
 
 I was more concerned about the authentication aspect meaning the request
 not able to reach the MON node and hence not getting forwarded to the
 storage nodes ?
 
 Thanks,
 Vilobh
 
 On 3/30/14, 7:21 AM, Loic Dachary l...@dachary.org wrote:
 
 
 
 On 30/03/2014 14:23, Vilobh Meshram wrote:
 I was trying to play around with RDO and CEPH at
 http://openstack.redhat.com/Using_Ceph_for_Cinder_with_RDO_Havana but
 after I install QEMU for CEPH and when I run
 
 qemu-img create ­f raw rbd:data/foo 1G
 
 The command just hangs without creating any raw device. Am I missing
 something here ?
 
 Could it be that you only have one machine running Ceph ? What does ceph
 osd tree prints ?
 
 
 Thanks,
 Vilobh
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 -- 
 Loïc Dachary, Artisan Logiciel Libre
 
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] firefly timing

[Karan Singh]

Hello Everyone

I am looking forward to test new features of 0.78 , it would be nice if erasure 
coding and tiering implementation notes available in ceph documentation.

Ceph documentation is in a good shape and its always nice to follow.

A humble request to add erasure coding and tiering in documentation if its not 
already in your pipeline. 


Karan Singh
CSC - IT Center for Science Ltd.
P.O. Box 405, FI-02101 Espoo, FINLAND
http://www.csc.fi/ | +358 (0) 503 812758

On 18 Mar 2014, at 09:13, Ирек Фасихов malm...@gmail.com wrote:

 I'm ready to test the tiering.
 
 
 2014-03-18 11:07 GMT+04:00 Stefan Priebe - Profihost AG 
 s.pri...@profihost.ag:
 Hi Sage,
 
 i really would like to test the tiering. Is there any detailed
 documentation about it and how it works?
 
 Greets,
 Stefan
 
 Am 18.03.2014 05:45, schrieb Sage Weil:
  Hi everyone,
 
  It's taken longer than expected, but the tests for v0.78 are calming down
  and it looks like we'll be able to get the release out this week.
 
  However, we've decided NOT to make this release firefly.  It will be a
  normal development release.  This will be the first release that includes
  some key new functionality (erasure coding and cache tiering) and although
  it is passing our tests we'd like to have some operational experience with
  it in more users' hands before we commit to supporting it long term.
 
  The tentative plan is to freeze and then release v0.79 after a normal two
  week cycle.  This will serve as a 'release candidate' that shaves off a
  few rough edges from the pending release (including some improvements with
  the API for setting up erasure coded pools).  It is possible that 0.79
  will turn into firefly, but more likely that we will opt for another two
  weeks of hardening and make 0.80 the release we name firefly and maintain
  for the long term.
 
  Long story short: 0.78 will be out soon, and you should test it!  It is
  will vary from the final firefly in a few subtle ways, but any feedback or
  usability and bug reports at this point will be very helpful in shaping
  things.
 
  Thanks!
  sage
  --
  To unsubscribe from this list: send the line unsubscribe ceph-devel in
  the body of a message to majord...@vger.kernel.org
  More majordomo info at  http://vger.kernel.org/majordomo-info.html
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 
 -- 
 С уважением, Фасихов Ирек Нургаязович
 Моб.: +79229045757
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] No more Journals ?

[Karan Singh]

Hello Everyone

If you see ceph day presentation delivered by Sebastien ( slide number 23 )   
http://www.slideshare.net/Inktank_Ceph/ceph-performance

It looks like Firefly has dropped support to Journals , How concrete is this 
news ???


-Karan-


On 14 Mar 2014, at 15:35, Jake Young jak3...@gmail.com wrote:

 You should take a look at this blog post:
 
 http://ceph.com/community/ceph-performance-part-2-write-throughput-without-ssd-journals/
 
 The test results shows that using a RAID card with a write-back cache without 
 journal disks can perform better or equivalent to using journal disks with 
 XFS. 
 
 As to whether or not it’s better to buy expensive controllers and use all of 
 your drive bays for spinning disks or cheap controllers and use some portion 
 of your bays for SSDs/Journals, there are trade-offs.  If built right, 
 systems with SSD journals provide higher large block write throughput, while 
 putting journals on the data disks provides higher storage density.  Without 
 any tuning both solutions currently provide similar IOP throughput.
 
 Jake
 
 
 On Friday, March 14, 2014, Markus Goldberg goldb...@uni-hildesheim.de wrote:
 Sorry,
 i should have asked a little bit clearer:
 Can ceph (or OSDs) be used without journals now ?
 The Journal-Parameter seems to be optional ( because of '[...]' )
 
 Markus
 Am 14.03.2014 12:19, schrieb John Spray:
 Journals have not gone anywhere, and ceph-deploy still supports
 specifying them with exactly the same syntax as before.
 
 The page you're looking at is the simplified quick start, the detail
 on osd creation including journals is here:
 http://eu.ceph.com/docs/v0.77/rados/deployment/ceph-deploy-osd/
 
 Cheers,
 John
 
 On Fri, Mar 14, 2014 at 9:47 AM, Markus Goldberg
 goldb...@uni-hildesheim.de wrote:
 Hi,
 i'm a little bit surprised. I read through the new manuals of 0.77
 (http://eu.ceph.com/docs/v0.77/start/quick-ceph-deploy/)
 In the section of creating the osd the manual says:
 
 Then, from your admin node, use ceph-deploy to prepare the OSDs.
 
 ceph-deploy osd prepare {ceph-node}:/path/to/directory
 
 For example:
 
 ceph-deploy osd prepare node2:/var/local/osd0 node3:/var/local/osd1
 
 Finally, activate the OSDs.
 
 ceph-deploy osd activate {ceph-node}:/path/to/directory
 
 For example:
 
 ceph-deploy osd activate node2:/var/local/osd0 node3:/var/local/osd1
 
 
 In former versions the osd was created like:
 
 ceph-deploy -v --overwrite-conf osd --fs-type btrfs prepare
 bd-0:/dev/sdb:/dev/sda5
 
 ^^ Journal
 As i remember defining and creating a journal for each osd was a must.
 
 So the question is: Are Journals obsolet now ?
 
 --
 MfG,
Markus Goldberg
 
 --
 Markus Goldberg   Universität Hildesheim
Rechenzentrum
 Tel +49 5121 88392822 Marienburger Platz 22, D-31141 Hildesheim, Germany
 Fax +49 5121 88392823 email goldb...@uni-hildesheim.de
 --
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 
 -- 
 MfG,
   Markus Goldberg
 
 --
 Markus Goldberg   Universität Hildesheim
   Rechenzentrum
 Tel +49 5121 88392822 Marienburger Platz 22, D-31141 Hildesheim, Germany
 Fax +49 5121 88392823 email goldb...@uni-hildesheim.de
 --
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Issue with Openstack Glance integration with CEPH

[Karan Singh]

Hi Srinivasa

Few things for troubleshooting 


1)  check in glance-api.conf  , it should have

rbd_store_ceph_conf = /etc/ceph/ceph.conf

2) if not already done 

cp /etc/ceph/ceph.client.images.keyring /etc/glance
3)  i am not sure if there is any difference between  glance image-create and 
glance add  , but try with another image as well.  ( download new image and 
move it to glance )

[root@rdo var(keystone_admin)]# wget 
http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img

 glance add name=ceph-glance-image is_public=True disk_format=qcow2 
container_format=ovf architecture=x86_64   ubuntu-12.04.3-desktop-amd64.iso

4) If you are still getting Internal server error , make sure your openstack is 
working fine , if i remember correctly i too seen same problem , which was 
related to openstack ,  check openstack logs also ceph logs.

—Regards
karan


On 28 Feb 2014, at 16:24, Srinivasa Rao Ragolu srag...@mvista.com wrote:

 Hi Karan,
 
 First of all many many thanks for your blog written on openstack integration 
 with CEPH
 
 I could able to integrate Openstack Cinder with CEPH successfully and attach 
 volumes to running VMs 
 
 But facing the issue with Glance service while uploading image as shown below
 
 
 [controller@controller images]$ glance image-create --name=CirrOS 0.3.1 
 --disk-format=qcow2 --container-format=bare --is-public=true  
 cirros-0.3.1-x86_64-disk.img
 Request returned failure status.
 500 Internal Server Error
 Failed to upload image be2b5d58-3a17-4544-8a4e-8c5b31ff5546
 (HTTP 500)
 
 On stand alone openstack installation i could able to do the above 
 successfully
 
 changes in /etc/glance/glance-api.conf
 
 default_store=rbd
 rbd_store_user=images
 rbd_store_pool=images
 Output of ceph auth list is 
 [controller@controller images]$ ceph auth list
 installed auth entries:
 
 osd.0
 key: AQBOx/1S+IovNBAAupF3jUQliOg8HXwEuwNrIw==
 caps: [mon] allow profile osd
 caps: [osd] allow *
 osd.1
 key: AQClyP1SgEY4LhAA7t0ylE1M+JzMz8uYklbmHg==
 caps: [mon] allow profile osd
 caps: [osd] allow *
 client.admin
 key: AQChw/1SmDhNDxAAwEd5o8Di1paSPsvUJSOGVQ==
 caps: [mds] allow
 caps: [mon] allow *
 caps: [osd] allow *
 client.bootstrap-mds
 key: AQChw/1SuG98LhAAjMFVX7Zhqgsgn8SnR2rx9A==
 caps: [mon] allow profile bootstrap-mds
 client.bootstrap-osd
 key: AQChw/1SgGFVHhAAG/C5UNaXexYFR14Ocs+SPw==
 caps: [mon] allow profile bootstrap-osd
 client.image
 key: AQBalRBTIDDnBBAA/edyQ2lrLOOyol2452GKQg==
 caps: [mon] allow rw
 caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
 pool=image
 client.images
 key: AQBWzwVTsBclExAAc36/Z1eNZ5k4jQxhMX2gaw==
 caps: [mon] allow r
 caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
 pool=images
 client.radosgw.gateway
 key: AQAJHw9TOGLoLBAAx6ItvnW3Vqu/If2xtvcoUg==
 caps: [mon] allow rw
 caps: [osd] allow rwx
 client.volumes
 key: AQAuzwVTUHSkFBAAdNmC5Ql73WEjFkl/sBi6/g==
 caps: [mon] allow r
 caps: [osd] allow class-read object_prefix rbd_children, allow rwx 
 pool=volumes, allow rx pool=images
 
 Restarted the glance-api, glance-registry services.
 
 Could you please help me in resolving above error?
 
 Thanks in advance,
 Srinivas.
 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Issue with Openstack Glance integration with CEPH

[Karan Singh]

https://ask.openstack.org/en/question/9570/glance-image-create-returns-httpinternalservererror-http-500/


check this also try to upload a new image , i know its crazy but just a try.

—karan

On 28 Feb 2014, at 16:24, Srinivasa Rao Ragolu srag...@mvista.com wrote:

 Failed to upload image

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Unable to add monitor nodes

[Karan Singh]

Hi Kumar

Follow this  

http://karan-mj.blogspot.fi/2013/12/ceph-openstack-part-2.html   ( part 3 , 4 
and 5  as well )


http://ceph.com/docs/master/rbd/libvirt/


—karan singh

On 19 Feb 2014, at 08:28, yalla.gnan.ku...@accenture.com wrote:

 Hi,
 
 I want to implement two scenarios:
 ---
 1. Store the sample cirrOS image in ceph block device and use it to boot the 
 VMs.
 2. Use the ceph block device as a volume and attach it to the VMs.
 
 
 Thanks
 Kumar
 
 
 -Original Message-
 From: Gnan Kumar, Yalla
 Sent: Wednesday, February 19, 2014 11:15 AM
 To: 'Alfredo Deza'; ceph-users@lists.ceph.com
 Cc: ceph-users@lists.ceph.com
 Subject: RE: [ceph-users] Unable to add monitor nodes
 
 Hi,
 
 Now I am integrating  ceph with openstack.   I have installed openstack . Now 
 to test ceph,
 how do I create a ceph block device and store the sample cirrOS image in  it 
 and use it for booting VMs  ?
 
 
 Thanks
 Kumar
 
 -Original Message-
 From: Alfredo Deza [mailto:alfredo.d...@inktank.com]
 Sent: Tuesday, February 18, 2014 8:22 PM
 To: Gnan Kumar, Yalla
 Cc: ceph-users@lists.ceph.com
 Subject: Re: [ceph-users] Unable to add monitor nodes
 
 ceph-deploy is not able to add new monitors, this is a feature that is 
 actually being worked on at the moment
 
 See http://tracker.ceph.com/issues/6638
 
 You will need to do this manually and specify all the things you are missing 
 following the Ceph docs here 
 http://ceph.com/docs/master/rados/operations/add-or-rm-mons/
 
 On Mon, Feb 17, 2014 at 7:43 AM,  yalla.gnan.ku...@accenture.com wrote:
 Hi,
 
 
 
 I am new user of ceph. I have installed a three node cluster following
 the ceph document. I have added OSDs and initial monitor.
 
 But while adding additional monitors, I am receiving this error as
 shown below.
 
 
 
 
 
 user1@cephadmin:~/my-cluster$ ceph-deploy mon create   cephnode2
 
 [ceph_deploy.cli][INFO  ] Invoked (1.3.5): /usr/bin/ceph-deploy mon
 create
 cephnode2
 
 [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts cephnode2
 
 [ceph_deploy.mon][DEBUG ] detecting platform for host cephnode2 ...
 
 [cephnode2][DEBUG ] connected to host: cephnode2
 
 [cephnode2][DEBUG ] detect platform information from remote host
 
 [cephnode2][DEBUG ] detect machine type
 
 [ceph_deploy.mon][INFO  ] distro info: Ubuntu 12.04 precise
 
 [cephnode2][DEBUG ] determining if provided host has same hostname in
 remote
 
 [cephnode2][DEBUG ] get remote short hostname
 
 [cephnode2][DEBUG ] deploying mon to cephnode2
 
 [cephnode2][DEBUG ] get remote short hostname
 
 [cephnode2][DEBUG ] remote hostname: cephnode2
 
 [cephnode2][DEBUG ] write cluster configuration to
 /etc/ceph/{cluster}.conf
 
 [cephnode2][DEBUG ] create the mon path if it does not exist
 
 [cephnode2][DEBUG ] checking for done path:
 /var/lib/ceph/mon/ceph-cephnode2/done
 
 [cephnode2][DEBUG ] done path does not exist:
 /var/lib/ceph/mon/ceph-cephnode2/done
 
 [cephnode2][INFO  ] creating keyring file:
 /var/lib/ceph/tmp/ceph-cephnode2.mon.keyring
 
 [cephnode2][DEBUG ] create the monitor keyring file
 
 [cephnode2][INFO  ] Running command: sudo ceph-mon --cluster ceph
 --mkfs -i
 cephnode2 --keyring /var/lib/ceph/tmp/ceph-cephnode2.mon.keyring
 
 [cephnode2][DEBUG ] ceph-mon: set fsid to
 b3d4e423-25a2-4380-8595-8b3fae4f8806
 
 [cephnode2][DEBUG ] ceph-mon: created monfs at
 /var/lib/ceph/mon/ceph-cephnode2 for mon.cephnode2
 
 [cephnode2][INFO  ] unlinking keyring file
 /var/lib/ceph/tmp/ceph-cephnode2.mon.keyring
 
 [cephnode2][DEBUG ] create a done file to avoid re-doing the mon
 deployment
 
 [cephnode2][DEBUG ] create the init path if it does not exist
 
 [cephnode2][DEBUG ] locating the `service` executable...
 
 [cephnode2][INFO  ] Running command: sudo initctl emit ceph-mon
 cluster=ceph
 id=cephnode2
 
 [cephnode2][INFO  ] Running command: sudo ceph --cluster=ceph
 --admin-daemon /var/run/ceph/ceph-mon.cephnode2.asok mon_status
 
 [cephnode2][ERROR ] admin_socket: exception getting command descriptions:
 [Errno 2] No such file or directory
 
 [cephnode2][WARNIN] monitor: mon.cephnode2, might not be running yet
 
 [cephnode2][INFO  ] Running command: sudo ceph --cluster=ceph
 --admin-daemon /var/run/ceph/ceph-mon.cephnode2.asok mon_status
 
 [cephnode2][ERROR ] admin_socket: exception getting command descriptions:
 [Errno 2] No such file or directory
 
 [cephnode2][WARNIN] cephnode2 is not defined in `mon initial members`
 
 [cephnode2][WARNIN] monitor cephnode2 does not exist in monmap
 
 [cephnode2][WARNIN] neither `public_addr` nor `public_network` keys
 are defined for monitors
 
 [cephnode2][WARNIN] monitors may not be able to form quorum
 
 -
 
 
 
 What is the error about ?
 
 
 
 
 
 Thanks
 
 Kumar
 
 
 
 
 This message is for the designated recipient only and may contain
 privileged, proprietary, or otherwise confidential information

Re: [ceph-users] newbie / Error connecting to cluster: PermissionError

[Karan Singh]

Hello

Can you try this

1)  From monitor node  scp /etc/ceph/ceph.client.admin.keyring node1:/etc/ceph
2)  From monitor node scp /var/lib/ceph/bootstrap-osd/ceph.keyring 
node1:/var/lib/ceph/bootstrap-osd

I encountered in the same issue yesterday ( on centOS ) and fixed in this 
manner.


—Karan Singh


On 19 Feb 2014, at 12:23, João Pagaime j...@fccn.pt wrote:

 hello Alfredo
 
 here's some additional information and log output attached
 
 had some trouble with yum (Resolving dl.fedoraproject.org... failed: Name or 
 service not known), but only with node3 and ceph-admin.  looked ok after 
 retrying
 
 -
 [ceph@ceph-admin scripts]$ uname -a
 Linux ceph-admin 2.6.32-431.5.1.el6.x86_64 #1 SMP Wed Feb 12 00:41:43 UTC 
 2014 x86_64 x86_64 x86_64 GNU/Linux
 
 ---
 [ceph@ceph-admin scripts]$ cat /etc/redhat-release
 CentOS release 6.5 (Final)
 
 ---
 [ceph@node1 ~]$ sudo yum update
 
 No Packages marked for Update
 
 
 - iptables
 
 [ceph@node1 ~]$ chkconfig --list|grep tables
 ip6tables 0:off 1:off 2:off 3:off 4:off 5:off 6:off
 iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off
 
 [ceph@node1 ~]$ sudo iptables -L
 Chain INPUT (policy ACCEPT)
 target prot opt source destination
 
 Chain FORWARD (policy ACCEPT)
 target prot opt source destination
 
 Chain OUTPUT (policy ACCEPT)
 target prot opt source destination
 
 
 --- some suspect errors on activate (see file 
 05-activate-typescript)
 
 filestore(/var/local/osd0) could not find 23c2fcde/osd_superblock/0//-1 in 
 index: (2) No such file or directory
 
  some more diagnostics:
 
 [ceph@node1 ~]$ ceph health
 2014-02-19 10:06:56.952386 7ffa7648b700 -1 monclient(hunting): ERROR: missing 
 keyring, cannot use cephx for authentication
 2014-02-19 10:06:56.952438 7ffa7648b700 0 librados: client.admin 
 initialization error (2) No such file or directory
 Error connecting to cluster: ObjectNotFound
 
 - setup
 
 vms addresses
 
 [ceph@ceph-admin scripts]$ cat /etc/hosts
 ...
 192.168.50.144 ceph-admin
 192.168.50.145 node1
 192.168.50.146 node2
 192.168.50.147 node3
 192.168.50.148 client
 
 ceph version--
 
 [ceph@node1 ~]$ rpm -q -a |grep -i ceph
 ceph-deploy-1.3.5-0.noarch
 ceph-release-1-0.el6.noarch
 python-ceph-0.72.2-0.el6.x86_64
 libcephfs1-0.72.2-0.el6.x86_64
 ceph-0.72.2-0.el6.x86_64
 
 hope it's useful
 
 didn't try ubuntu yet
 
 best regards,
 joão
 
 Melhores cumprimentos,
 João Pagaime,
 Área de Infra-estruturas Aplicacionais - FCCN – FCT I.P.
 
 Em 18-02-2014 20:54, Alfredo Deza escreveu:
 On Tue, Feb 18, 2014 at 3:44 PM, João Pagaime j...@fccn.pt wrote:
 Hello João
 
 thanks for the reply
 
 tried again with debian VMs and didn't have any problems (it worked!)
 
 the ceph-deploy script doesn't seem to work perfectly on CentOS 6.5, so I
 guess that was the reason for the troubles (it did throw a few errors, but,
 since it was on debug mode, I didn't pay much attention)
 That sounds odd, we do test ceph-deploy heavily against that distro
 and version. It would be useful to
 see the log output of your issues and determine if ceph-deploy is
 missing something or it needs fixing.
 
 One of the reasons why sometimes getting a cluster up and running
 works with some distros right of the bat is
 that certain things come enabled/disabled that affect cluster
 operation like iptables for example.
 
 
 thanks, best regards,
 
 Melhores cumprimentos,
 João Pagaime,
 Área de Infra-estruturas Aplicacionais - FCCN - FCT I.P.
 
 Em 18-02-2014 16:04, Joao Eduardo Luis escreveu:
 
 On 02/18/2014 08:42 AM, João Pagaime wrote:
 Hello all
 
 We're looking at Ceph for a possible future pilot, and so
 I've just installed ceph-emperor [1] by the Installation (Quick)
 guide, but got stuck on the following error:
 
 authentication error (1) Operation not permitted [2]
 
 could someone give clues as to diagnose (or better yet fix) this problem?
 
 the setup:
 - as prescribed by the current Installation (Quick) guide
 - all vms are CentOS release 6.5 (Final) running on vmware-player:
 - one VM admin node ceph-deploy
 - one VM node1 (monitor)
 - one VM node2
 - one VM node3
 
 any help would be welcome!
 
 best regards
 joao
 
 [1]--- version
 
 $ rpm -q -a |grep -i ceph
 libcephfs1-0.72.2-0.el6.x86_64
 python-ceph-0.72.2-0.el6.x86_64
 ceph-0.72.2-0.el6.x86_64
 ceph-release-1-0.el6.noarch
 
 [2]- specific error
 
 [ceph@ceph-node2 ~]$ sudo ceph-disk-activate --mark-init sysvinit
 --mount /var/local/osd0
 === osd.0 ===
 2014-02-18 08:35:44.769825 7f1659c3c700 0 librados: osd.0
 authentication error (1) Operation not permitted
 Error connecting to cluster: PermissionError
 failed: 'timeout 10 /usr/bin/ceph --name=osd.0
 --keyring=/var/lib/ceph/osd/ceph-0/keyring osd crush create-or-move
 -- 0 0.05 root=default
 host

Re: [ceph-users] Paris Ceph meetup

[Karan Singh]

Sounds great Loic  

Is it possible for you to share these presentations online ?  ,  if they are in 
English ;-)


Regards
karan



On 18 Feb 2014, at 12:08, Loic Dachary l...@dachary.org wrote:

 Hi Ceph,
 
 Yesterday evening was the Paris Ceph meetup and it was great to see everyone.
 
 http://www.meetup.com/Ceph-in-Paris/photos/20178242/334526882/#334529682
 
 There was six Ceph commiters (Christophe, Guilhem, Laurent, Sebastien, 
 Francois and myself) out of twenty seven attendees. We kept the presentations 
 to a minimum :
 
 * How could Ceph be used by an ISP by Thanh-Liêm
 * How Ceph is used, a devop view by Laurent
 * The state of the Chef cookbooks by Ghuilhem
 * Erasure Code theory and implementation by myself
 
 Then our Ceph cluster was divided in pools of five to seven hosts and started 
 peer to peer interactions ;-)
 
 The head count was done by the number of t-shirts distributed. From an 
 organizational point of view, there only was one glitch : we had to hijack a 
 VGA cable from a company located nearby http://goo.gl/M8JyXZ . It is a great 
 ( and free of charge ) spot for meetups.
 
 Cheers
 
 -- 
 Loïc Dachary, Artisan Logiciel Libre
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osd down

[Karan Singh]

Hi Pavel

Try to add at least 1 more OSD ( bare minimum ) and set pool replication to 2 
after that.
For osd.0  try  ,   # ceph osd in osd.0   , once the osd is IN , try to bring 
up osd.0 services up 


Finally your both the OSD should be  IN  and UP , so that your cluster can 
store data.

Regards
Karan


On 16 Feb 2014, at 20:06, Pavel V. Kaygorodov pa...@inasan.ru wrote:

 Hi, All!
 
 I am trying to setup ceph from scratch, without dedicated drive, with one mon 
 and one osd.
 After all, I see following output of ceph osd tree:
 
 # idweight  type name   up/down reweight
 -1  1   root default
 -2  1   host host1
 0   1   osd.0   down0
 
 ceph -w:
 
cluster e90dfd37-98d1-45bb-a847-8590a5ed8e71
 health HEALTH_WARN 192 pgs stuck inactive; 192 pgs stuck unclean
 monmap e1: 1 mons at {host1=172.17.0.4:6789/0}, election epoch 1, quorum 
 0 host1
 osdmap e5: 1 osds: 0 up, 0 in
  pgmap v6: 192 pgs, 3 pools, 0 bytes data, 0 objects
0 kB used, 0 kB / 0 kB avail
 192 creating
 
 2014-02-16 13:27:30.095938 mon.0 [INF] osdmap e5: 1 osds: 0 up, 0 in
 
 What can be wrong?
 I see working daemons, and nothing bad in log files.
 
 
 
 How to reproduce:
 I have cloned and compiled sources on debian/jessie:
 
 git clone --recursive -b v0.75 https://github.com/ceph/ceph.git
 cd /ceph/  ./autogen.sh  ./configure  make  make install
 
 Everything seems ok.
 
 I have created ceph.conf:
 
 [global]
 
 fsid = e90dfd37-98d1-45bb-a847-8590a5ed8e71
 mon initial members = host1
 
 auth cluster required = cephx
 auth service required = cephx
 auth client required = cephx
 
 keyring = /data/ceph.client.admin.keyring
 
 osd pool default size = 1
 osd pool default min size = 1
 osd pool default pg num = 333
 osd pool default pgp num = 333
 osd crush chooseleaf type = 0   
 osd journal size = 1000
 
 filestore xattr use omap = true
 
 ;journal dio = false
 ;journal aio = false
 
 mon addr = ceph.dkctl
 mon host = ceph.dkctl
 
 log file = /data/logs/ceph.log
 
 [mon]
 mon data = /data/mon0
 keyring = /data/ceph.mon.keyring
 log file = /data/logs/mon0.log
 
 [osd.0]
 osd host= host1
 osd data= /data/osd0
 osd journal = /data/osd0.journal
 log file= /data/logs/osd0.log
 keyring = /data/ceph.osd0.keyring
 
 ///
 
 I have initialized mon and osd using following script:
 
 /usr/local/bin/ceph-authtool --create-keyring /data/ceph.mon.keyring 
 --gen-key -n mon. --cap mon 'allow *'
 /usr/local/bin/ceph-authtool --create-keyring /data/ceph.client.admin.keyring 
 --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' 
 --cap mds 'allow'
 /usr/local/bin/ceph-authtool /data/ceph.mon.keyring --import-keyring 
 /data/ceph.client.admin.keyring 
 /usr/local/bin/monmaptool --create --add host1 `grep ceph /etc/hosts | awk 
 '{print $1}'` --fsid de90dfd37-98d1-45bb-a847-8590a5ed8e71 /data/monmap
 /usr/local/bin/ceph-mon --mkfs -i host1 --monmap /data/monmap --keyring 
 /data/ceph.mon.keyring
 /usr/local/bin/ceph-mon -c /ceph.conf --public-addr `grep ceph /etc/hosts | 
 awk '{print $1}'` -i host1
 /usr/local/bin/ceph osd create e90dfd37-98d1-45bb-a847-8590a5ed8e71
 /usr/local/bin/ceph osd create e90dfd37-98d1-45bb-a847-8590a5ed8e71
 /usr/local/bin/ceph-osd -i 0 --mkfs --mkkey
 /usr/local/bin/ceph auth add osd.0 osd 'allow *' mon 'allow rwx'  -i 
 /data/ceph.osd0.keyring
 /usr/local/bin/ceph osd crush add-bucket host1 host
 /usr/local/bin/ceph osd crush move host1 root=default
 /usr/local/bin/ceph osd crush add osd.0 1.0 host=host1
 
 
 
 Script ouptut seems to be ok:
 
 creating /data/ceph.mon.keyring
 creating /data/ceph.client.admin.keyring
 importing contents of /data/ceph.client.admin.keyring into 
 /data/ceph.mon.keyring
 /usr/local/bin/monmaptool: monmap file /data/monmap
 /usr/local/bin/monmaptool: set fsid to e90dfd37-98d1-45bb-a847-8590a5ed8e71
 /usr/local/bin/monmaptool: writing epoch 0 to /data/monmap (1 monitors)
 /usr/local/bin/ceph-mon: set fsid to e90dfd37-98d1-45bb-a847-8590a5ed8e71
 /usr/local/bin/ceph-mon: created monfs at /data/mon0 for mon.host1
 0
 2014-02-16 13:24:37.833469 7f5ef61747c0 -1 journal FileJournal::_open: 
 disabling aio for non-block journal.  Use journal_force_aio to force use of 
 aio anyway
 2014-02-16 13:24:37.94 7f5ef61747c0 -1 journal FileJournal::_open: 
 disabling aio for non-block journal.  Use journal_force_aio to force use of 
 aio anyway
 2014-02-16 13:24:37.948704 7f5ef61747c0 -1 filestore(/data/osd0) could not 
 find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
 2014-02-16 13:24:38.054345 7f5ef61747c0 -1 created object store /data/osd0 
 journal /data/osd0.journal for osd.0 fsid e90dfd37-98d1-45bb-a847-8590a5ed8e71
 2014-02-16 13:24:38.054427 7f5ef61747c0 -1 auth: error reading file: 
 /data/ceph.osd0.keyring: can't open 

[ceph-users] Problems after up gradation from 0.65 to 0.76

[Karan Singh]

-81bbfb2e676a
 health HEALTH_WARN 6 pgs degraded; 262 pgs down; 564 pgs peering; 14 pgs 
stale; 564 pgs stuck inactive; 12 pgs stuck stale; 570 pgs stuck unclean; 
recovery 10/49174 objects degraded (0.020%); mds cluster is degraded; mds 
ceph-mon1 is laggy; 2/10 in osds are down; crush map has non-optimal tunables
 monmap e3: 3 mons at 
{ceph-mon1=192.168.1.38:6789/0,ceph-mon2=192.168.1.33:6789/0,ceph-mon3=192.168.1.31:6789/0},
 election epoch 4218, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3
 mdsmap e8461: 1/1/1 up {0=ceph-mon1=up:replay(laggy or crashed)}
 osdmap e249808: 10 osds: 8 up, 10 in
  pgmap v583944: 576 pgs, 6 pools, 98505 MB data, 24587 objects
342 GB used, 5424 GB / 5767 GB avail
10/49174 objects degraded (0.020%)
   6 active+clean
 184 peering
 223 down+peering
   6 active+degraded
  12 stale+peering
   2 stale+down+peering
 106 remapped+peering
  37 down+remapped+peering
[root@ceph-node2 ceph]#




Karan Singh
CSC - IT Center for Science Ltd.
P.O. Box 405, FI-02101 Espoo, FINLAND
http://www.csc.fi/





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] PG not getting clean

[Karan Singh]

Hello Cephers

I am struggling with my ceph cluster health  ,  PGS are not getting clean , i 
waited for recovery process to get end was hoping after recovery PG will become 
clean , but it didn’t. Can you please share your suggestions.

cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a
 health HEALTH_WARN 119 pgs down; 303 pgs peering; 303 pgs stuck inactive; 
303 pgs stuck unclean; mds cluster is degraded; crush map has no
n-optimal tunables
 monmap e3: 3 mons at 
{ceph-mon1=192.168.1.38:6789/0,ceph-mon2=192.168.1.33:6789/0,ceph-mon3=192.168.1.31:6789/0},
 election epoch 4226, quo
rum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3
 mdsmap e8465: 1/1/1 up {0=ceph-mon1=up:replay}
 osdmap e250466: 10 osds: 10 up, 10 in
  pgmap v585809: 576 pgs, 6 pools, 101933 MB data, 25453 objects
343 GB used, 5423 GB / 5767 GB avail
 273 active+clean
 108 peering
 119 down+peering
  76 remapped+peering


# idweight  type name   up/down reweight
-1  5.65root default
-2  0   host ceph-node1
-3  1.72host ceph-node2
4   0.43osd.4   up  1
5   0.43osd.5   up  1
6   0.43osd.6   up  1
7   0.43osd.7   up  1
-4  1.31host ceph-node4
8   0.88osd.8   up  1
1   0.43osd.1   up  1
-5  1.31host ceph-node5
9   0.88osd.9   up  1
2   0.43osd.2   up  1
-6  0.88host ceph-node6
10  0.88osd.10  up  1
-7  0.43host ceph-node3
0   0.43osd.0   up  1



Regards
karan

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Is it possible to have One Ceph-OSD-Daemon managing more than one OSD

[Karan Singh]

Hi vikrant

Yes , its possible to have this , but if you follow best practices its not 
recommended.

Under this scenario , if you have let suppose 5 disks tied to one OSD daemon  
and by any reason if the OSD daemon stops working , you would loose all 5 
drives in a row.

Good to know the reason for this setup , why you really wants this ??


Regards
karan



On 13 Feb 2014, at 09:37, Ashish Chandra mail.ashishchan...@gmail.com wrote:

 Hi,
 
  With Ceph, an OSD is generally one Ceph ceph-osd daemon for one storage 
 drive within a host machine. If your host has multiple storage drives, you 
 may map one ceph-osd daemon for each drive.
 
 But , in your condition what I would suggest that make a volume group out of 
 all physical hard drives and create a logical volume of roughly the same size 
 as of your volume group , and use it as one single osd.
 
 I have tried this and it works for me.
 
 Thanks and Regards
 Ashish Chandra
 
 
 On Wed, Feb 12, 2014 at 11:33 PM, Vikrant Verma vikrantverm...@gmail.com 
 wrote:
 yes, I want to use multiple hard drives with a single OSD.
 Is it possible to have it?
 
 Regards,
 Vikrant
 
 
 On Wed, Feb 12, 2014 at 10:14 PM, Loic Dachary l...@dachary.org wrote:
 
 
 On 12/02/2014 12:28, Vikrant Verma wrote:
  Hi All,
 
  I have one quick question -
 
  Is it possible to have One Ceph-OSD-Daemon managing more than one Object 
  Storage Device in a Ceph  Cluster?
 
 Hi,
 
 Do you want to use multiple hard drives with a single OSD ?
 
 Cheers
 
  Regards,
  Vikrant
 
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 --
 Loïc Dachary, Artisan Logiciel Libre
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph MON can no longer join quorum

[Karan Singh]

Hi Greg 




I have seen this problem before in my cluster. 




* What ceph version you are running 
* Did you made any change recently in the cluster , that resulted in this 
problem 





You identified correct , the only problem is ceph-mon-2003 is listening to 
incorrect port , it should listen on port 6789 ( like the other two monitors ) 
. How i resolved is cleanly removing the infected monitor node and adding it 
back to cluster. 




Regards 

Karan 
- Original Message -

From: Greg Poirier greg.poir...@opower.com 
To: ceph-users@lists.ceph.com 
Sent: Tuesday, 4 February, 2014 10:50:21 PM 
Subject: [ceph-users] Ceph MON can no longer join quorum 

I have a MON that at some point lost connectivity to the rest of the cluster 
and now cannot rejoin. 

Each time I restart it, it looks like it's attempting to create a new MON and 
join the cluster, but the rest of the cluster rejects it, because the new one 
isn't in the monmap. 

I don't know why it suddenly decided it needed to be a new MON. 

I am not really sure where to start. 

root@ceph-mon-2003:/var/log/ceph# ceph -s 
cluster 4167d5f2-2b9e-4bde-a653-f24af68a45f8 
health HEALTH_ERR 1 pgs inconsistent; 2 pgs peering; 126 pgs stale; 2 pgs stuck 
inactive; 126 pgs stuck stale; 2 pgs stuck unclean; 10 requests are blocked  
32 sec; 1 scrub errors; 1 mons down, quorum 0,1 ceph-mon-2001,ceph-mon-2002 
monmap e2: 3 mons at {ceph-mon-2001= 
10.30.66.13:6789/0,ceph-mon-2002=10.30.66.14:6789/0,ceph-mon-2003=10.30.66.15:6800/0
 }, election epoch 12964, quorum 0,1 ceph-mon-2001,ceph-mon-2002 

Notice ceph-mon-2003:6800 

If I try to start ceph-mon-all, it will be listening on some other port... 

root@ceph-mon-2003:/var/log/ceph# start ceph-mon-all 
ceph-mon-all start/running 
root@ceph-mon-2003:/var/log/ceph# ps -ef | grep ceph 
root 6930 1 31 15:49 ? 00:00:00 /usr/bin/ceph-mon --cluster=ceph -i 
ceph-mon-2003 -f 
root 6931 1 3 15:49 ? 00:00:00 python /usr/sbin/ceph-create-keys --cluster=ceph 
-i ceph-mon-2003 

root@ceph-mon-2003:/var/log/ceph# ceph -s 
2014-02-04 15:49:56.854866 7f9cf422d700 0 -- :/1007028  10.30.66.15:6789/0 
pipe(0x7f9cf0021370 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f9cf00215d0).fault 
cluster 4167d5f2-2b9e-4bde-a653-f24af68a45f8 
health HEALTH_ERR 1 pgs inconsistent; 2 pgs peering; 126 pgs stale; 2 pgs stuck 
inactive; 126 pgs stuck stale; 2 pgs stuck unclean; 10 requests are blocked  
32 sec; 1 scrub errors; 1 mons down, quorum 0,1 ceph-mon-2001,ceph-mon-2002 
monmap e2: 3 mons at {ceph-mon-2001= 
10.30.66.13:6789/0,ceph-mon-2002=10.30.66.14:6789/0,ceph-mon-2003=10.30.66.15:6800/0
 }, election epoch 12964, quorum 0,1 ceph-mon-2001,ceph-mon-2002 

Suggestions? 

___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Meetup in Frankfurt, before the Ceph day

[Karan Singh]

+1   

It would be nice to meet you guys @ Ceph Day Frankfurt.  

Anyone else from ceph community , willing to join.


Loic — Same here , first time visiting to Frankfurt.



Regards
Karan


On 05 Feb 2014, at 10:44, Loic Dachary l...@dachary.org wrote:

 Hi Ceph,
 
 I'll be in Frankfurt for the Ceph day February 27th 
 http://www.eventbrite.com/e/ceph-day-frankfurt-tickets-10173269523 and I will 
 attend the meetup organized the evening before 
 http://www.meetup.com/Ceph-Frankfurt/events/164620852/
 
 Anyone interested to join ? Not sure where we should meet ... I've never been 
 to Frankfurt before :-)
 
 Cheers
 
 -- 
 Loïc Dachary, Artisan Logiciel Libre
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph MON can no longer join quorum

[Karan Singh]

Hi Greg

I have seen this problem before in my cluster.

What ceph version you are running 
Did you made any change recently in the cluster , that resulted in this problem

You identified correct , the only problem is ceph-mon-2003  is listening to 
incorrect port , it should listen on port 6789 ( like the other two monitors ) 
. How i resolved is cleanly removing the infected monitor node and adding it 
back to cluster.

Regards
Karan


On 04 Feb 2014, at 22:50, Greg Poirier greg.poir...@opower.com wrote:

 I have a MON that at some point lost connectivity to the rest of the cluster 
 and now cannot rejoin.
 
 Each time I restart it, it looks like it's attempting to create a new MON and 
 join the cluster, but the rest of the cluster rejects it, because the new one 
 isn't in the monmap.
 
 I don't know why it suddenly decided it needed to be a new MON.
 
 I am not really sure where to start. 
 
 root@ceph-mon-2003:/var/log/ceph# ceph -s
 cluster 4167d5f2-2b9e-4bde-a653-f24af68a45f8
  health HEALTH_ERR 1 pgs inconsistent; 2 pgs peering; 126 pgs stale; 2 
 pgs stuck inactive; 126 pgs stuck stale; 2 pgs stuck unclean; 10 requests are 
 blocked  32 sec; 1 scrub errors; 1 mons down, quorum 0,1 
 ceph-mon-2001,ceph-mon-2002
  monmap e2: 3 mons at 
 {ceph-mon-2001=10.30.66.13:6789/0,ceph-mon-2002=10.30.66.14:6789/0,ceph-mon-2003=10.30.66.15:6800/0},
  election epoch 12964, quorum 0,1 ceph-mon-2001,ceph-mon-2002
 
 Notice ceph-mon-2003:6800
 
 If I try to start ceph-mon-all, it will be listening on some other port...
 
 root@ceph-mon-2003:/var/log/ceph# start ceph-mon-all
 ceph-mon-all start/running
 root@ceph-mon-2003:/var/log/ceph# ps -ef | grep ceph
 root  6930 1 31 15:49 ?00:00:00 /usr/bin/ceph-mon 
 --cluster=ceph -i ceph-mon-2003 -f
 root  6931 1  3 15:49 ?00:00:00 python 
 /usr/sbin/ceph-create-keys --cluster=ceph -i ceph-mon-2003
 
 root@ceph-mon-2003:/var/log/ceph# ceph -s
 2014-02-04 15:49:56.854866 7f9cf422d700  0 -- :/1007028  10.30.66.15:6789/0 
 pipe(0x7f9cf0021370 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f9cf00215d0).fault
 cluster 4167d5f2-2b9e-4bde-a653-f24af68a45f8
  health HEALTH_ERR 1 pgs inconsistent; 2 pgs peering; 126 pgs stale; 2 
 pgs stuck inactive; 126 pgs stuck stale; 2 pgs stuck unclean; 10 requests are 
 blocked  32 sec; 1 scrub errors; 1 mons down, quorum 0,1 
 ceph-mon-2001,ceph-mon-2002
  monmap e2: 3 mons at 
 {ceph-mon-2001=10.30.66.13:6789/0,ceph-mon-2002=10.30.66.14:6789/0,ceph-mon-2003=10.30.66.15:6800/0},
  election epoch 12964, quorum 0,1 ceph-mon-2001,ceph-mon-2002
 
 Suggestions?
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How can I set the warning level?

[Karan Singh]

Hello Greg

Where can we find this config option , any command for this.

Many Thanks
Karan Singh


- Original Message -
From: Gregory Farnum g...@inktank.com
To: vernon1...@126.com
Cc: ceph-users ceph-users@lists.ceph.com
Sent: Monday, 6 January, 2014 9:49:22 PM
Subject: [ceph-users] How can I set the warning level?

On Wed, Dec 25, 2013 at 6:13 PM, vernon1...@126.com vernon1...@126.com wrote:
 Hello,  my Mon's always HEALTH_WARN, and I run ceph health detail, it show
 me like this:

 HEALTH_WARN
 mon.2 addr 192.168.0.7:6789/0 has 30% avail disk space -- low disk space!

 I want to know how to set this warning level? I have to made it give out the
 alarm with the space remaining no more than 10%.

There's not any option to disable specific ceph health warnings. You
can change the threshold at which that warning occurs with the mon
data avail [warn|crit] config options, though (default to 30 and 5).
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] The Kraken has been released!

[Karan Singh]

Excellent Stuff Donald . 

Thanks :-) 

Many Thanks 
Karan Singh 


- Original Message -

From: Don Talton (dotalton) dotal...@cisco.com 
To: ceph-us...@ceph.com 
Sent: Thursday, 9 January, 2014 7:31:16 AM 
Subject: [ceph-users] The Kraken has been released! 



The first phase of Kraken (free) dashboard for Ceph cluster monitoring is 
complete. You can grab it here ( https://github.com/krakendash/krakendash ) 



Pictures here http://imgur.com/a/JoVPy 



Current features: 



MON statuses 

OSD statuses 

OSD detail drilldown 

Pool statuses 

Pool detail drilldown 



Upcoming features: 

Advanced metrics via collectd 

Cluster management (eg write) operations 

Multi-cluster support 

Hardware node monitoring 



Dave Simard has contributed a wrapper for the Ceph API here ( 
https://github.com/dmsimard/python-cephclient ) which Kraken will begin using 
shortly. 



Pull requests are welcome! The more the merrier, I’d love to get more features 
developed. 



Donald Talton 

Cloud Systems Development 

Cisco Systems 





___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] After reboot nothing worked

[Karan Singh]

Umar 

Ceph is stable for production , there are a large number of ceph clusters 
deployed and running smoothly in PRODUCTIONS and countless in testing / 
pre-production. 

Since you are facing problems with your ceph testing , it does not mean CEPH is 
unstable. 

I would suggest put some time troubleshooting your problem. 

What i see from your logs -- 

1) you have 2 Mons thats a problem ( either have 1 or have 3 to form quorum ) . 
Add 1 more monitor node 
2) out of 2 OSD , only 1 is IN , check where is the other one and try bringing 
both of them UP . Add few more OSD's to remove health warning . 2 OSD is a very 
less numbers for OSD 

Many Thanks 
Karan Singh 


- Original Message -

From: Umar Draz unix...@gmail.com 
To: ceph-us...@ceph.com 
Sent: Tuesday, 17 December, 2013 8:51:27 AM 
Subject: [ceph-users] After reboot nothing worked 

Hello, 

I have 2 node ceph cluster, I just rebooted both of the host just for testing 
that after rebooting the cluster remain work or not, and the result was cluster 
unable to start. 

here is ceph -s output 

health HEALTH_WARN 704 pgs stale; 704 pgs stuck stale; mds cluster is degraded; 
1/1 in osds are down; clock skew detected on mon.kvm2 
monmap e2: 2 mons at {kvm1= 192.168.214.10:6789/0,kvm2=192.168.214.11:6789/0 }, 
election epoch 16, quorum 0,1 kvm1,kvm2 
mdsmap e13: 1/1/1 up {0=kvm1=up:replay} 
osdmap e29: 2 osds: 0 up, 1 in 
pgmap v68: 704 pgs, 4 pools, 9603 bytes data, 23 objects 
1062 MB used, 80816 MB / 81879 MB avail 
704 stale+active+clean 

according to this useless documentation. 

http://ceph.com/docs/master/rados/operations/monitoring-osd-pg/ 

I tried ceph osd tree 

the output was 

# id weight type name up/down reweight 
-1 0.16 root default 
-2 0.07999 host kvm1 
0 0.07999 osd.0 down 1 
-3 0.07999 host kvm2 
1 0.07999 osd.1 down 0 

Then i tried 

sudo /etc/init.d/ceph -a start osd.0 
sudo /etc/init.d/ceph -a start osd.1 

to start the osd on both host the result was 

/etc/init.d/ceph: osd.0 not found (/etc/ceph/ceph.conf defines , /var/lib/ceph 
defines ) 

/etc/init.d/ceph: osd.1 not found (/etc/ceph/ceph.conf defines , /var/lib/ceph 
defines ) 

Now question is what is this? is really ceph is stable? can we use this for 
production environment? 

My both host has ntp running the time is upto date. 

Br. 

Umar 

___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] After reboot nothing worked

[Karan Singh]

Thanks Joao for information.

Many Thanks
Karan Singh 


- Original Message -
From: Joao Eduardo Luis joao.l...@inktank.com
To: ceph-users@lists.ceph.com
Sent: Tuesday, 17 December, 2013 2:56:23 PM
Subject: Re: [ceph-users] After reboot nothing worked

On 12/17/2013 09:54 AM, Karan Singh wrote:
 Umar

 *Ceph is stable for production* , there are a large number of ceph
 clusters deployed and running smoothly in PRODUCTIONS and countless in
 testing / pre-production.

 Since you are facing problems with your ceph testing , it does not mean
 CEPH is unstable.

 I would suggest put some time troubleshooting your problem.

 What i see from your logs  --

   1) you have 2 Mons thats a problem ( either have 1  or have 3 to form
 quorum ) . Add 1 more monitor node

Just to clarify this point a bit, one doesn't need an odd number of 
monitors in a ceph cluster to reach quorum.  This is a common misconception.

The requirement to reach quorum is simply to have a majority of monitors 
able to talk to each other.  If one has 2 monitors and both are able to 
talk to each other they'll be able to form a quorum.

Odd-numbers are advised however because one can tolerate as much 
failures with less infrastructure. E.g.,

- for n = 1, failure of 1 monitor means loss of quorum
- for n = 2, failure of 1 monitor means loss of quorum
- for n = 3, failure of 1 monitor is okay; failure of 2 monitors means 
loss of quorum
- for n = 4, failure of 1 monitor is okay; failure of 2 monitors means 
loss of quorum
- for n = 5, failure of 2 monitors is okay; failure of 3 monitors means 
loss of quorum
- for n = 6, failure of 2 monitors is okay; failure of 3 monitors means 
loss of quorum

etc.

So you can see how you don't get any benefits, from an availability 
perspective, by having either 2, 4 or 6 monitors when compared to having 
1, 3, 5.  If your target however is replication, then 2 is better than 1.

   -Joao



-- 
Joao Eduardo Luis
Software Engineer | http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph not responding after trying to add a new MON

[Karan Singh]

Hello German 

Can you check the following and let us know. 

1. After you execute service ceph start , are the service getting started ?? 
what is the output of service ceph status 
2. what does cehp status says 
3. check on ceph-node02 what are things mounted. 

Many Thanks 
Karan Singh 


- Original Message -

From: German Anders gand...@despegar.com 
To: ceph-users@lists.ceph.com 
Sent: Friday, 13 December, 2013 7:11:41 PM 
Subject: [ceph-users] Ceph not responding after trying to add a new MON 

Hi to All, 
I've a situation where i can't run any ceph command on the cluster. Initially 
the cluster had only one MON daemon, with three OSD daemons running. Everything 
were ok, but someone from the team try to add a new MON daemon, and then when i 
try to start the ceph service I'm getting this error message (I've try it on 
every node): 

root@ceph-node02:/tmp/ceph-node02# service ceph start 
=== mon.ceph-node02 === 
Starting Ceph mon.ceph-node02 on ceph-node02... 
failed: 'ulimit -n 32768; /usr/bin/ceph-mon -i ceph-node02 --pid-file 
/var/run/ceph/mon.ceph-node02.pid -c /etc/ceph/ceph.conf ' 
Starting ceph-create-keys on ceph-node02... 
INFO:ceph-disk:Activating 
/dev/disk/by-parttypeuuid/4fbd7e29-9d25-41b8-afd0-062c0ceff05d.dbf17a68-e94e-4dc7-bcc4-60263e4b0a7c
 
INFO:ceph-disk:ceph osd.0 already mounted in position; unmounting ours. 

root@ceph-node01:/var/log/ceph# service ceph start 
=== mon.ceph-node01 === 
Starting Ceph mon.ceph-node01 on ceph-node01... 
failed: 'ulimit -n 32768; /usr/bin/ceph-mon -i ceph-node01 --pid-file 
/var/run/ceph/mon.ceph-node01.pid -c /etc/ceph/ceph.conf ' 
Starting ceph-create-keys on ceph-node01... 
INFO:ceph-disk:Activating 
/dev/disk/by-parttypeuuid/4fbd7e29-9d25-41b8-afd0-062c0ceff05d.fcf613c6-ae4a-4a44-b890-6d77dac3818b
 
INFO:ceph-disk:ceph osd.2 already mounted in position; unmounting ours. 
root@ceph-node01:/var/log/ceph# 

root@ceph-node03:~# service ceph start 
INFO:ceph-disk:Activating 
/dev/disk/by-parttypeuuid/4fbd7e29-9d25-41b8-afd0-062c0ceff05d.7ba458b7-bd58-4373-b4b7-a0b1cffec548
 
INFO:ceph-disk:ceph osd.1 already mounted in position; unmounting ours. 
root@ceph-node03:~# 

The initial monitor was ceph-node01. 

Here's the /etc/ceph/ceph.conf file from the three nodes: 

[global] 
fsid = cd60ab37-23bd-4c17-9470-404cb3b31112 
mon_initial_members = ceph-node01 
mon_host = ceph-node01 
auth_supported = cephx 
osd_journal_size = 1024 
filestore_xattr_use_omap = true 

[mon.ceph-node01] 
host = ceph-node01 
mon addr = 10.111.82.242:6789 

[mon.ceph-node02] 
host = ceph-node02 
mon aggr = 10.111.82.245:6789 


Someone could point me out here to solve this issue? 

Thanks in advance, 

Best regards, 


German Anders 








___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph not responding after trying to add a new MON

[Karan Singh]

can you come to #ceh irc channel , we will troubleshoot real time ?? 

Many Thanks 
Karan Singh 


- Original Message -

From: German Anders gand...@despegar.com 
To: Karan Singh ksi...@csc.fi 
Cc: ceph-users@lists.ceph.com 
Sent: Friday, 13 December, 2013 8:05:00 PM 
Subject: Re: [ceph-users] Ceph not responding after trying to add a new MON 

Also here's the output from the mount command: 

ceph-node01: 

root@ceph-node01:/var/lib/ceph/osd/ceph-2# mount 
/dev/mapper/ubuntu-root on / type ext4 (rw,errors=remount-ro) 
proc on /proc type proc (rw,noexec,nosuid,nodev) 
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) 
none on /sys/fs/fuse/connections type fusectl (rw) 
none on /sys/kernel/debug type debugfs (rw) 
none on /sys/kernel/security type securityfs (rw) 
udev on /dev type devtmpfs (rw,mode=0755) 
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) 
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) 
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) 
none on /run/shm type tmpfs (rw,nosuid,nodev) 
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755) 
/dev/sda1 on /boot type ext2 (rw) 
/dev/sdb1 on /var/lib/ceph/osd/ceph-2 type xfs (rw,noatime) 


ceph-node02: 

root@ceph-node02:/var/lib/ceph/osd# mount 
proc on /proc type proc (rw,noexec,nosuid,nodev) 
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) 
none on /sys/fs/fuse/connections type fusectl (rw) 
none on /sys/kernel/security type securityfs (rw) 
udev on /dev type devtmpfs (rw,mode=0755) 
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) 
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) 
none on /sys/kernel/debug type debugfs (rw) 
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) 
none on /run/shm type tmpfs (rw,nosuid,nodev) 
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755) 
/dev/mapper/ceph--deploy02-root on / type ext4 (rw,errors=remount-ro) 
/dev/sda1 on /boot type ext2 (rw) 
/dev/sdb1 on /var/lib/ceph/osd/ceph-0 type xfs (rw,noatime) 


ceph-node03: 

root@ceph-node03:/var/lib/ceph/osd# mount 
proc on /proc type proc (rw,noexec,nosuid,nodev) 
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) 
none on /sys/fs/fuse/connections type fusectl (rw) 
none on /sys/kernel/security type securityfs (rw) 
udev on /dev type devtmpfs (rw,mode=0755) 
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) 
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) 
none on /sys/kernel/debug type debugfs (rw) 
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) 
none on /run/shm type tmpfs (rw,nosuid,nodev) 
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755) 
/dev/mapper/ceph--deploy03-root on / type ext4 (rw,errors=remount-ro) 
/dev/sda1 on /boot type ext2 (rw) 
/dev/sdb1 on /var/lib/ceph/osd/ceph-1 type xfs (rw,noatime) 



German Anders 







--- Original message --- 
Asunto: Re: [ceph-users] Ceph not responding after trying to add a new MON 
De: Karan Singh ksi...@csc.fi 
Para: German Anders gand...@despegar.com 
Cc: ceph-users@lists.ceph.com 
Fecha: Friday, 13/12/2013 14:56 

Hello German 

Can you check the following and let us know. 

1. After you execute service ceph start , are the service getting started ?? 
what is the output of service ceph status 
2. what does cehp status says 
3. check on ceph-node02 what are things mounted. 

Many Thanks 
Karan Singh 



From: German Anders gand...@despegar.com 
To: ceph-users@lists.ceph.com 
Sent: Friday, 13 December, 2013 7:11:41 PM 
Subject: [ceph-users] Ceph not responding after trying to add a new MON 

Hi to All, 
I've a situation where i can't run any ceph command on the cluster. Initially 
the cluster had only one MON daemon, with three OSD daemons running. Everything 
were ok, but someone from the team try to add a new MON daemon, and then when i 
try to start the ceph service I'm getting this error message (I've try it on 
every node): 

root@ceph-node02:/tmp/ceph-node02# service ceph start 
=== mon.ceph-node02 === 
Starting Ceph mon.ceph-node02 on ceph-node02... 
failed: 'ulimit -n 32768; /usr/bin/ceph-mon -i ceph-node02 --pid-file 
/var/run/ceph/mon.ceph-node02.pid -c /etc/ceph/ceph.conf ' 
Starting ceph-create-keys on ceph-node02... 
INFO:ceph-disk:Activating 
/dev/disk/by-parttypeuuid/4fbd7e29-9d25-41b8-afd0-062c0ceff05d.dbf17a68-e94e-4dc7-bcc4-60263e4b0a7c
 
INFO:ceph-disk:ceph osd.0 already mounted in position; unmounting ours. 

root@ceph-node01:/var/log/ceph# service ceph start 
=== mon.ceph-node01 === 
Starting Ceph mon.ceph-node01 on ceph-node01... 
failed: 'ulimit -n 32768; /usr/bin/ceph-mon -i ceph-node01 --pid-file 
/var/run/ceph/mon.ceph-node01.pid -c /etc/ceph/ceph.conf ' 
Starting ceph-create-keys on ceph-node01... 
INFO:ceph-disk:Activating 
/dev/disk/by-parttypeuuid/4fbd7e29-9d25-41b8-afd0-062c0ceff05d.fcf613c6-ae4a-4a44-b890-6d77dac3818b
 
INFO:ceph-disk:ceph osd.2

Re: [ceph-users] Openstack glance ceph rbd_store_user authentification problem

[Karan Singh]

Hi Vikrant 

I managed to boot openstack from ceph volume , You can see my work , hope this 
will help you. http://karan-mj.blogspot.fi/ 



* Change efault_store to rbd in glance-api.conf 
* Make sure user with name  glance exists in ceph auth list output. 


Many Thanks 
Karan Singh 


- Original Message -

From: Vikrant Verma vikrantverm...@gmail.com 
To: Karan Singh ksi...@csc.fi 
Cc: thorh...@iti.cs.uni-magdeburg.de, ceph-users@lists.ceph.com 
Sent: Wednesday, 11 December, 2013 10:42:55 AM 
Subject: Re: [ceph-users] Openstack glance ceph rbd_store_user authentification 
problem 

Hi Karan, 

I want to set up a ceph boot volume, i am following the below link for that 
http://ceph.com/docs/master/rbd/rbd-openstack/#configuring-glance 


Here is the glance-api.conf contents - 

[DEFAULT] 
# Show more verbose log output (sets INFO log level output) 
verbose = True 

# Show debugging output in logs (sets DEBUG log level output) 
debug = True 

# Which backend scheme should Glance use by default is not specified 
# in a request to add a new image to Glance? Known schemes are determined 
# by the known_stores option below. 
# Default: 'file' 
default_store = file 

# List of which store classes and store class locations are 
# currently known to glance at startup. 
#known_stores = glance.store.filesystem.Store, 
# glance.store.http.Store, 
# glance.store.rbd.Store, 
# glance.store.s3.Store, 
# glance.store.swift.Store, 
# glance.store.sheepdog.Store, 
# glance.store.cinder.Store, 


# Maximum image size (in bytes) that may be uploaded through the 
# Glance API server. Defaults to 1 TB. 
# WARNING: this value should only be increased after careful consideration 
# and must be set to a value under 8 EB (9223372036854775808). 
#image_size_cap = 1099511627776 

# Address to bind the API server 
bind_host = 0.0.0.0 

# Port the bind the API server to 
bind_port = 9292 

# Log to this file. Make sure you do not set the same log 
# file for both the API and registry servers! 
log_file = /var/log/glance/api.log 

# Backlog requests when creating socket 
backlog = 4096 

# TCP_KEEPIDLE value in seconds when creating socket. 
# Not supported on OS X. 
#tcp_keepidle = 600 

# API to use for accessing data. Default value points to sqlalchemy 
# package, it is also possible to use: glance.db.registry.api 
# data_api = glance.db.sqlalchemy.api 

# SQLAlchemy connection string for the reference implementation 
# registry server. Any valid SQLAlchemy connection string is fine. 
# See: 
http://www.sqlalchemy.org/docs/05/reference/sqlalchemy/connections.html#sqlalchemy.create_engine
 
sql_connection = mysql://glance:intel123@vikrant/glance 

# Period in seconds after which SQLAlchemy should reestablish its connection 
# to the database. 
# 
# MySQL uses a default `wait_timeout` of 8 hours, after which it will drop 
# idle connections. This can result in 'MySQL Gone Away' exceptions. If you 
# notice this, you can lower this value to ensure that SQLAlchemy reconnects 
# before MySQL can drop the connection. 
sql_idle_timeout = 3600 

# Number of Glance API worker processes to start. 
# On machines with more than one CPU increasing this value 
# may improve performance (especially if using SSL with 
# compression turned on). It is typically recommended to set 
# this value to the number of CPUs present on your machine. 
workers = 1 

# Role used to identify an authenticated user as administrator 
#admin_role = admin 

# Allow unauthenticated users to access the API with read-only 
# privileges. This only applies when using ContextMiddleware. 
#allow_anonymous_access = False 

# Allow access to version 1 of glance api 
#enable_v1_api = True 

# Allow access to version 2 of glance api 
#enable_v2_api = True 

# Return the URL that references where the data is stored on 
# the backend storage system. For example, if using the 
# file system store a URL of 'file:///path/to/image' will 
# be returned to the user in the 'direct_url' meta-data field. 
# The default value is false. 
#show_image_direct_url = False 

# Send headers containing user and tenant information when making requests to 
# the v1 glance registry. This allows the registry to function as if a user is 
# authenticated without the need to authenticate a user itself using the 
# auth_token middleware. 
# The default value is false. 
#send_identity_headers = False 

# Supported values for the 'container_format' image attribute 
#container_formats=ami,ari,aki,bare,ovf 

# Supported values for the 'disk_format' image attribute 
#disk_formats=ami,ari,aki,vhd,vmdk,raw,qcow2,vdi,iso 

# Directory to use for lock files. Default to a temp directory 
# (string value). This setting needs to be the same for both 
# glance-scrubber and glance-api. 
#lock_path=None 
# 
# Property Protections config file 
# This file contains the rules for property protections and the roles 
# associated with it. 
# If this config value is not specified, by default, property

Re: [ceph-users] Openstack glance ceph rbd_store_user authentification problem

[Karan Singh]

Hi Vikrant 

Can you share ceph auth list and your glance-api.conf file output. 

What are your plans with respect to configuration , what you want to achieve. 

Many Thanks 
Karan Singh 


- Original Message -

From: Vikrant Verma vikrantverm...@gmail.com 
To: thorh...@iti.cs.uni-magdeburg.de 
Cc: ceph-users@lists.ceph.com 
Sent: Tuesday, 10 December, 2013 1:20:32 PM 
Subject: [ceph-users] Openstack glance ceph rbd_store_user authentification 
problem 

Hi Steffen, 

WIth respect to your post as mentioned in the below link 

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-August/003370.html 

I am facing the same issue, here is my error log from api.log 

2013-12-10 02:47:36.156 32509 TRACE glance.api.v1.upload_utils File 
/usr/lib/python2.7/dist-packages/rados.py, line 367, in connect 
2013-12-10 02:47:36.156 32509 TRACE glance.api.v1.upload_utils raise 
make_ex(ret, error calling connect) 
2013-12-10 02:47:36.156 32509 TRACE glance.api.v1.upload_utils ObjectNotFound: 
error calling connect 
2013-12-10 02:47:36.156 32509 TRACE glance.api.v1.upload_utils 
2013-12-10 02:47:36.157 32509 DEBUG glance.registry.client.v1.api 
[5fb8c48e-73a8-4d65-b921-9bcc8ca3488f None None] Updating image metadata for 
image a66d0475-70fd-49f0-86c3-ddc47cffdcaf... update_image_metadata 
/usr/lib/python2.7/dist-packages/glance/registry/client/v1/api.py:168 
2013-12-10 02:47:36.157 32509 DEBUG glance.common.client 
[5fb8c48e-73a8-4d65-b921-9bcc8ca3488f None None] Constructed URL: 
http://0.0.0.0:9191/images/a66d0475-70fd-49f0-86c3-ddc47cffdcaf _construct_url 
/usr/lib/python2.7/dist-packages/glance/common/client.py:408 
2013-12-10 02:47:36.221 32509 DEBUG glance.registry.client.v1.client 
[5fb8c48e-73a8-4d65-b921-9bcc8ca3488f None None] Registry request PUT 
/images/a66d0475-70fd-49f0-86c3-ddc47cffdcaf HTTP 200 request id 
req-8aba73e7-3390-4201-acdb-73ae1ddf80f7 do_request 
/usr/lib/python2.7/dist-packages/glance/registry/client/v1/client.py:115 


As per your suggestion  configured a client.glance auth and it worked with the 
implicit glance user 

Please let me know to configure it. 

Regards, 
Vikrant 

___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] My experience with ceph now documentted

[Karan Singh]

Hello Cephers 

I would like to say a BIG THANKS to ceph community for helping me in setting up 
and learning ceph. 

I have created a small documentation http://karan-mj.blogspot.fi/ of my 
experience with ceph till now , i belive it would help beginners in installing 
ceph and integrating it with openstack. I would keep updating this blog. 


PS -- i recommend original ceph documentation http://ceph.com/docs/master/ and 
other original content published by Ceph community , INKTANK and other 
partners. My attempt http://karan-mj.blogspot.fi/ is just to contribute for a 
regular online content about ceph. 



Karan Singh 
CSC - IT Center for Science Ltd. 
P.O. Box 405, FI-02101 Espoo, FINLAND 
http://www.csc.fi/ | +358 (0) 503 812758 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable

[Karan Singh]

Hello Jan

I faced similar kind of errors and these are really annoying. I tried this and 
worked for me.

 1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
 2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will 
give you more strength.
 3. You might need to manually add monitor in cluster. 


  Below is my solution ( you should change it as per your hostname and IP etc)



 login on monitor node and execute commands like below 
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph

In case ceph service is not showing as running in service ceph status command , 
however ps -ef | grep ceph , shows monitor running , kill the process manually 
and restart ceph services , monitor now should start working

# service ceph status
# ps -ef | grep ceph
# kill -9 6554

### Finally you would see your ceph cluster is healthy with all the OSD and 
monitors UP and Running

[root@ceph-mon1 ~]# ceph status
  cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a
   health HEALTH_OK
   monmap e3: 3 mons at 
{ceph-mon1=192.168.1.38:6789/0,ceph-mon2=192.168.1.33:6789/0,ceph-mon3=192.168.1.31:6789/0},
 election epoch 10, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3
   osdmap e97: 8 osds: 8 up, 8 in
pgmap v246: 192 pgs: 192 active+clean; 0 bytes data, 2352 MB used, 3509 GB 
/ 3512 GB avail
   mdsmap e1: 0/0/1 up
[root@ceph-mon1 ~]#


 All the best :-)

Many Thanks
Karan Singh


- Original Message -
From: Jan Kalcic jan...@gmail.com
To: ceph-users@lists.ceph.com
Sent: Thursday, 5 December, 2013 11:16:32 AM
Subject: [ceph-users] IO error: lock 
/var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily 
unavailable

Hi all,

approaching ceph today for the first time, so apologize for the basic 
questions I promise I will do all my homework :-)

Following the documentation storage cluster quick start I am soon 
stuck with the issue below while creating a first mon:

ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO  ] Invoked (1.3.3): /usr/bin/ceph-deploy mon 
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO  ] distro info: SUSE Linux Enterprise Server 11 
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in 
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path: 
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO  ] Running command: sudo rcceph -c /etc/ceph/ceph.conf 
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] failed: 'ulimit -n 32768;  /usr/bin/ceph-mon -i 
ceph-node1 --pid-file /var/run/ceph/mon.ceph-node1.pid -c 
/etc/ceph/ceph.conf '
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO  ] Running command: sudo ceph --cluster=ceph 
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command 
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor: mon.ceph-node1, might not be running yet
[ceph-node1][INFO  ] Running command: sudo ceph --cluster=ceph 
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command 
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor ceph-node1 does not exist in monmap
[ceph-node1][WARNIN] neither `public_addr` nor `public_network` keys are 
defined for monitors
[ceph-node1][WARNIN] monitors may not be able to form quorum

It seems ceph-mon does not exit with success, in fact:

ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file 
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780  0 ceph version 0.72.1 
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource 
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource

[ceph-users] Openstack--instance-boot-from-ceph-volume:: error could not open disk image rbd

[Karan Singh]

 to | 
+--+---+---+--+--+--+--+
 
| 0e2bfced-be6a-44ec-a3ca-22c771c66cdc | in-use | nova-vol_1 | 2 | None | false 
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 | 
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available | ceph-vol1 | 5 | 
ceph-storage | false | | 
| 5e09ce7b-b874-4c24-97ae-341a305841d5 | available | precise-cloudimg_vol_boot 
| 5 | None | true | | 
| 9671edaa-62c8-4f98-a36c-d6e59612141b | available | boot_from_volume | 20 | 
None | false | | 
| a75d435b-ebce-4013-9c68-8df5d2c917e1 | available | dashboard-volume | 1 | 
ceph-storage | false | | 
| dd315dda-b22a-4cf8-8b77-7c2b2f163155 | available | new_boot_from_vol | 10 | 
None | true | | 
+--+---+---+--+--+--+--+
 
[root@rdo nova(keystone_admin)]# 













[root@rdo nova(keystone_admin)]# 
[root@rdo nova(keystone_admin)]# nova boot --flavor 2 --image 
be62a5bf-879f-4d1f-846c-fdef960224ff --block_device_mapping 
vda=dd315dda-b22a-4cf8-8b77-7c2b2f163155:::0 --security_groups=default --nic 
net-id=4fe5909e-02db-4517-89f2-1278248fa26c myInstanceFromVolume 
+--++
 
| Property | Value | 
+--++
 
| OS-EXT-STS:task_state | scheduling | 
| image | precise-cloudimg.raw | 
| OS-EXT-STS:vm_state | building | 
| OS-EXT-SRV-ATTR:instance_name | instance-001d | 
| OS-SRV-USG:launched_at | None | 
| flavor | m1.small | 
| id | f324e9b8-ec3a-4174-8b97-bf78dba62932 | 
| security_groups | [{u'name': u'default'}] | 
| user_id | 99f8019ba2694d78a680a5de46aa1afd | 
| OS-DCF:diskConfig | MANUAL | 
| accessIPv4 | | 
| accessIPv6 | | 
| progress | 0 | 
| OS-EXT-STS:power_state | 0 | 
| OS-EXT-AZ:availability_zone | nova | 
| config_drive | | 
| status | BUILD | 
| updated | 2013-12-05T12:42:22Z | 
| hostId | | 
| OS-EXT-SRV-ATTR:host | None | 
| OS-SRV-USG:terminated_at | None | 
| key_name | None | 
| OS-EXT-SRV-ATTR:hypervisor_hostname | None | 
| name | myInstanceFromVolume | 
| adminPass | eish5pu56CiE | 
| tenant_id | 0dafe42cfde242ddbb67b681f59bdb00 | 
| created | 2013-12-05T12:42:21Z | 
| os-extended-volumes:volumes_attached | [{u'id': 
u'dd315dda-b22a-4cf8-8b77-7c2b2f163155'}] | 
| metadata | {} | 
+--++
 
[root@rdo nova(keystone_admin)]# 
[root@rdo nova(keystone_admin)]# 
[root@rdo nova(keystone_admin)]# 
[root@rdo nova(keystone_admin)]# nova list 
+--+--+-++-+-+
 
| ID | Name | Status | Task State | Power State | Networks | 
+--+--+-++-+-+
 
| 0043a8be-60d1-43ed-ba43-1ccd0bba7559 | instance2 | SHUTOFF | None | Shutdown 
| public=172.24.4.228 | 
| f324e9b8-ec3a-4174-8b97-bf78dba62932 | myInstanceFromVolume | ERROR | None | 
NOSTATE | private=10.0.0.3 | 
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 | small-ubuntu | ACTIVE | None | Running 
| public=172.24.4.230 | 
+--+--+-++-+-+
 
[root@rdo nova(keystone_admin)]# 







= 




Logs from /var/log/libvirt/qemu :: 


= 



qemu-kvm: -drive 
file=rbd:ceph-volumes/volume-dd315dda-b22a-4cf8-8b77-7c2b2f163155:id=volumes:key=AQC804xS8HzFJxAAD/zzQ8LMzq9wDLq/5a472g==:auth_supported=cephx\;none:mon_host=192.168.1.31\:6789\;192.168.1.33\:6789\;192.168.1.38\:6789,if=none,id=drive-virtio-disk0,format=raw,serial=dd315dda-b22a-4cf8-8b77-7c2b2f163155,cache=none:
 could not open disk image 
rbd:ceph-volumes/volume-dd315dda-b22a-4cf8-8b77-7c2b2f163155:id=volumes:key=AQC804xS8HzFJxAAD/zzQ8LMzq9wDLq/5a472g==:auth_supported=cephx\;none:mon_host=192.168.1.31\:6789\;192.168.1.33\:6789\;192.168.1.38\:6789:
 No such file or directory 
2013-12-05 12:42:29.544+: shutting down 


Any idea why its throwing this error. 

Many Thanks 
Karan Singh 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Openstack+ceph volume mounting to vm

[Karan Singh]

Hello Cephers 

Need your guidance 

In my setup ceph cluster and openstack are working good , i am able to create 
volumes using cinder as well. 

What i want is to mount ceph volume to VM instance. But getting deadly errors 
like this . Expecting your help in this 




[root@rdo /(keystone_admin)]# virsh attach-device instance-0018 disk.xml 
error: Failed to attach device from disk.xml 
error: internal error unable to execute QEMU command '__com.redhat_drive_add': 
Device 'drive-virtio-disk5' could not be initialized 

[root@rdo /(keystone_admin)]# 







My Setup details :- 




[root@rdo /(keystone_admin)]# rpm -qa | grep -i qemu 
qemu-img-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
qemu-kvm-tools-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
qemu-guest-agent-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
qemu-kvm-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
gpxe-roms-qemu-0.9.7-6.10.el6.noarch 
[root@rdo /(keystone_admin)]# 







[root@rdo /(keystone_admin)]# uname -a 
Linux rdo 3.10.18-1.el6.elrepo.x86_64 #1 SMP Mon Nov 4 19:12:54 EST 2013 x86_64 
x86_64 x86_64 GNU/Linux 
[root@rdo /(keystone_admin)]# 
[root@rdo /(keystone_admin)]# cat /etc/redhat-release 
CentOS release 6.5 (Final) 
[root@rdo /(keystone_admin)]# 







[root@rdo /(keystone_admin)]# cinder list 
+--+---+--+--+--+--+-+
 
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | 
+--+---+--+--+--+--+-+
 
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available | ceph-vol1 | 5 | 
ceph-storage | false | | 
| 9671edaa-62c8-4f98-a36c-d6e59612141b | available | boot_from_volume | 20 | 
None | false | | 
+--+---+--+--+--+--+-+
 
[root@rdo /(keystone_admin)]# 







[root@rdo /(keystone_admin)]# ceph status 
cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a 
health HEALTH_OK 
monmap e3: 3 mons at 
{ceph-mon1=192.168.1.38:6789/0,ceph-mon2=192.168.1.33:6789/0,ceph-mon3=192.168.1.31:6789/0},
 election epoch 30, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3 
osdmap e157: 11 osds: 11 up, 11 in 
pgmap v12102: 448 pgs: 448 active+clean; 135 GB data, 272 GB used, 5935 GB / 
6207 GB avail 
mdsmap e27: 1/1/1 up {0=ceph-mon1=up:active} 

[root@rdo /(keystone_admin)]# 





[root@rdo /(keystone_admin)]# cat disk.xml 
disk type='network' 
driver name=qemu type=raw/ 
source protocol=rbd 
name=ceph-volumes/volume-10cc0855-652a-4a9b-baa1-80bc86dc12ac 
host name='192.168.1.38' port='6789'/ 
host name='192.168.1.31' port='6789'/ 
host name='192.168.1.33' port='6789'/ 
/source 
target dev=vdf bus=virtio/ 
auth username='admin' 
secret type='ceph' uuid='801a42ec-aec1-3ea8-d869-823c2de56b83'/ 
/auth 
/disk 
[root@rdo /(keystone_admin)]# 




[root@rdo /(keystone_admin)]# service libvirtd status 
libvirtd (pid 17947) is running... 
[root@rdo /(keystone_admin)]# 







virsh # list 
Id Name State 
 
2 instance-0018 running 

virsh # 





Karan Singh 
Systems Specialist, Computing Environments Group 
CSC - IT Center for Science Ltd. 
P.O. Box 405, FI-02101 Espoo, FINLAND 
http://www.csc.fi/ | +358 (0) 503 812758 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Openstack+ceph volume mounting to vm

[Karan Singh]

Hello Every 

Still waiting .. Any help in this would be highly appreciated. 

Many Thanks 
Karan Singh 


- Original Message -

From: Karan Singh ksi...@csc.fi 
To: ceph-users@lists.ceph.com 
Sent: Tuesday, 3 December, 2013 6:27:29 PM 
Subject: [ceph-users] Openstack+ceph volume mounting to vm 

Hello Cephers 

Need your guidance 

In my setup ceph cluster and openstack are working good , i am able to create 
volumes using cinder as well. 

What i want is to mount ceph volume to VM instance. But getting deadly errors 
like this . Expecting your help in this 




[root@rdo /(keystone_admin)]# virsh attach-device instance-0018 disk.xml 
error: Failed to attach device from disk.xml 
error: internal error unable to execute QEMU command '__com.redhat_drive_add': 
Device 'drive-virtio-disk5' could not be initialized 

[root@rdo /(keystone_admin)]# 







My Setup details :- 




[root@rdo /(keystone_admin)]# rpm -qa | grep -i qemu 
qemu-img-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
qemu-kvm-tools-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
qemu-guest-agent-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
qemu-kvm-0.12.1.2-2.355.el6.2.cuttlefish.async.x86_64 
gpxe-roms-qemu-0.9.7-6.10.el6.noarch 
[root@rdo /(keystone_admin)]# 







[root@rdo /(keystone_admin)]# uname -a 
Linux rdo 3.10.18-1.el6.elrepo.x86_64 #1 SMP Mon Nov 4 19:12:54 EST 2013 x86_64 
x86_64 x86_64 GNU/Linux 
[root@rdo /(keystone_admin)]# 
[root@rdo /(keystone_admin)]# cat /etc/redhat-release 
CentOS release 6.5 (Final) 
[root@rdo /(keystone_admin)]# 







[root@rdo /(keystone_admin)]# cinder list 
+--+---+--+--+--+--+-+
 
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | 
+--+---+--+--+--+--+-+
 
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available | ceph-vol1 | 5 | 
ceph-storage | false | | 
| 9671edaa-62c8-4f98-a36c-d6e59612141b | available | boot_from_volume | 20 | 
None | false | | 
+--+---+--+--+--+--+-+
 
[root@rdo /(keystone_admin)]# 







[root@rdo /(keystone_admin)]# ceph status 
cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a 
health HEALTH_OK 
monmap e3: 3 mons at 
{ceph-mon1=192.168.1.38:6789/0,ceph-mon2=192.168.1.33:6789/0,ceph-mon3=192.168.1.31:6789/0},
 election epoch 30, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3 
osdmap e157: 11 osds: 11 up, 11 in 
pgmap v12102: 448 pgs: 448 active+clean; 135 GB data, 272 GB used, 5935 GB / 
6207 GB avail 
mdsmap e27: 1/1/1 up {0=ceph-mon1=up:active} 

[root@rdo /(keystone_admin)]# 





[root@rdo /(keystone_admin)]# cat disk.xml 
disk type='network' 
driver name=qemu type=raw/ 
source protocol=rbd 
name=ceph-volumes/volume-10cc0855-652a-4a9b-baa1-80bc86dc12ac 
host name='192.168.1.38' port='6789'/ 
host name='192.168.1.31' port='6789'/ 
host name='192.168.1.33' port='6789'/ 
/source 
target dev=vdf bus=virtio/ 
auth username='admin' 
secret type='ceph' uuid='801a42ec-aec1-3ea8-d869-823c2de56b83'/ 
/auth 
/disk 
[root@rdo /(keystone_admin)]# 




[root@rdo /(keystone_admin)]# service libvirtd status 
libvirtd (pid 17947) is running... 
[root@rdo /(keystone_admin)]# 







virsh # list 
Id Name State 
 
2 instance-0018 running 

virsh # 





Karan Singh 
Systems Specialist, Computing Environments Group 
CSC - IT Center for Science Ltd. 
P.O. Box 405, FI-02101 Espoo, FINLAND 
http://www.csc.fi/ | +358 (0) 503 812758 


___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] how to Testing cinder and glance with CEPH

[Karan Singh]

Hello Cephers 

I was following http://ceph.com/docs/master/rbd/rbd-openstack/ for ceph and 
openstack Integration , using this document ih ave done all the changes 
required for this integration. 

I am not sure how should i test my configuration , how should i make sure 
integration is successful. Can you suggest some test that i can perform to 
check my ceph and openstack integration . 

FYI , in the document http://ceph.com/docs/master/rbd/rbd-openstack/ , Nothing 
is mentioned after  Restart Openstack Services  heading , but there should be 
steps to test this inttegration , please suggest me here , i am new to 
openstack great if you can give me some commanes used for testing. 



Karan Singh 
CSC - IT Center for Science Ltd. 
P.O. Box 405, FI-02101 Espoo, FINLAND 
http://www.csc.fi/ | +358 (0) 503 812758 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to Testing cinder and glance with CEPH

[Karan Singh]

Hello Sebastien / Community


I tried the commands mentioned in below email.


[root@rdo ~]#
[root@rdo ~]# cinder create 1
+-+--+
|   Property  |Value |
+-+--+
| attachments |  []  |
|  availability_zone  | nova |
|   bootable  |false |
|  created_at |  2013-11-27T07:40:54.161478  |
| display_description | None |
| display_name| None |
|  id | ae8cd686-5f1d-4c05-8c42-cb7622122a3e |
|   metadata  |  {}  |
| size|  1   |
| snapshot_id | None |
| source_volid| None |
|status   |   creating   |
| volume_type | None |
+-+--+
[root@rdo ~]#
[root@rdo ~]# cinder list
+--++--+--+-+--+-+
|  ID  | Status | Display Name | Size | Volume 
Type | Bootable | Attached to |
+--++--+--+-+--+-+
| ae8cd686-5f1d-4c05-8c42-cb7622122a3e | error  | None |  1   | 
None|  false   | |
+--++--+--+-+--+-+
[root@rdo ~]#
[root@rdo ~]#
[root@rdo ~]#
[root@rdo ~]# rbd -p ceph-volumes ls
rbd: pool ceph-volumes doesn't contain rbd images
[root@rdo ~]#
[root@rdo ~]#
[root@rdo ~]# rados lspools
data
metadata
rbd
ceph-images
ceph-volumes
[root@rdo ~]# rbd -p rbd ls
[root@rdo ~]# rbd -p data ls
foo
foo1
[root@rdo ~]#




I checked in cinder.log and got the below errors.


2013-11-27 09:44:14.830 3273 INFO cinder.volume.manager [-] Updating volume 
status
2013-11-27 09:44:14.830 3273 WARNING cinder.volume.manager [-] Unable to update 
stats, driver is uninitialized
2013-11-27 09:44:42.407 12007 INFO cinder.volume.manager [-] Updating volume 
status
2013-11-27 09:44:42.408 12007 WARNING cinder.volume.manager [-] Unable to 
update stats, driver is uninitialized
2013-11-27 09:44:51.799 4943 INFO cinder.volume.manager [-] Updating volume 
status
2013-11-27 09:44:51.799 4943 WARNING cinder.volume.manager [-] Unable to update 
stats, driver is uninitialized
2013-11-27 09:45:14.834 3273 INFO cinder.volume.manager [-] Updating volume 
status
2013-11-27 09:45:14.834 3273 WARNING cinder.volume.manager [-] Unable to update 
stats, driver is uninitialized
[root@rdo cinder]#




Output from my cinder.conf file



# Options defined in cinder.volume.utils
#

# The default block size used when copying/clearing volumes
# (string value)
#volume_dd_blocksize=1M


# Total option count: 382
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_pool=ceph-volumes
glance_api_version=2
rbd_user=volumes
rbd_secret_uuid=801a42ec-aec1-3ea8-d869-823c2de56b83

rootwrap_config=/etc/cinder/rootwrap.conf
sql_connection = mysql://cinder:root@localhost/cinder
api_paste_config = /etc/cinder/api-paste.ini

iscsi_helper=tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
auth_strategy = keystone
#osapi_volume_listen_port=5900
[keystone_authtoken]
admin_tenant_name = admin
admin_user = admin
admin_password = root@123
auth_host = 192.168.1.49
auth_port = 35357
auth_protocol = http
signing_dirname = /tmp/keystone-signing-cinder
[root@rdo cinder]#



Can you suggest what is wrong with this configuration , i followed everything 
mentioned on http://ceph.com/docs/master/rbd/rbd-openstack/   and there is not 
much information available on internet WRT this error.

Please help.



Many Thanks
Karan Singh


- Original Message -
From: Sebastien Han sebastien@enovance.com
To: Karan Singh ksi...@csc.fi
Cc: ceph-users@lists.ceph.com
Sent: Wednesday, 27 November, 2013 1:08:35 AM
Subject: Re: [ceph-users] how to Testing cinder and glance with CEPH

Hi,

Well after restarting the services run:

$ cinder create 1

Then you can check both status in Cinder and Ceph:

For Cinder run:
$ cinder list

For Ceph run:
$ rbd -p cinder-pool ls

If the image is there, you’re good.

Cheers.

 
Sébastien Han 
Cloud Engineer 

Always give 100%. Unless you're giving blood.” 

Phone: +33 (0)1 49 70 99 72 
Mail: sebastien@enovance.com 
Address : 10, rue de la Victoire - 75009 Paris 
Web : www.enovance.com - Twitter : @enovance 

On 27 Nov 2013, at 00:04, Karan Singh ksi...@csc.fi wrote:

 Hello Cephers
 
 I was following http://ceph.com/docs/master/rbd/rbd-openstack/  for ceph

Re: [ceph-users] v0.72 Emperor released

[Karan Singh]

Congratulations Cephers its a great news.

@sage - Can we consider cephfs production ready now.

Many Thanks
Karan Singh


- Original Message -
From: Sage Weil s...@inktank.com
To: ceph-de...@vger.kernel.org, ceph-us...@ceph.com
Sent: Saturday, 9 November, 2013 7:40:04 AM
Subject: [ceph-users] v0.72 Emperor released

This is the fifth major release of Ceph, the fourth since adopting a
3-month development cycle. This release brings several new features,
including multi-datacenter replication for the radosgw, improved
usability, and lands a lot of incremental performance and internal
refactoring work to support upcoming features in Firefly.

Thank you to every who contributed to this release!  There were 46 authors 
in all.

Highlights include:

 * common: improved crc32c performance
 * librados: new example client and class code
 * mds: many bug fixes and stability improvements
 * mon: health warnings when pool pg_num values are not reasonable
 * mon: per-pool performance stats
 * osd, librados: new object copy primitives
 * osd: improved interaction with backend file system to reduce latency
 * osd: much internal refactoring to support ongoing erasure coding and 
   tiering support
 * rgw: bucket quotas
 * rgw: improved CORS support
 * rgw: performance improvements
 * rgw: validate S3 tokens against Keystone

Coincident with core Ceph, the Emperor release also brings:

 * radosgw-agent: support for multi-datacenter replication for disaster 
   recovery (buliding on the multi-site features that appeared in 
   Dumpling)
 * tgt: improved support for iSCSI via upstream tgt

Upgrading

There are no specific upgrade restrictions on the order or sequence of
upgrading from 0.67.x Dumpling.  We normally suggest a rolling upgrade
of monitors first, and then OSDs, followed by the radosgw and ceph-mds
daemons (if any).

It is also possible to do a rolling upgrade from 0.61.x Cuttlefish, but
there are ordering restrictions. (This is the same set of restrictions
for Cuttlefish to Dumpling.)

 1. Upgrade ceph-common on all nodes that will use the command line 
ceph utility.
 2. Upgrade all monitors (upgrade ceph package, restart ceph-mon 
daemons). This can happen one daemon or host at a time. Note that
because cuttlefish and dumpling monitors can't talk to each other,
all monitors should be upgraded in relatively short succession to
minimize the risk that an a untimely failure will reduce availability.
 3. Upgrade all osds (upgrade ceph package, restart ceph-osd daemons). 
This can happen one daemon or host at a time.
 4. Upgrade radosgw (upgrade radosgw package, restart radosgw daemons).

There are several minor compatibility changes in the librados API that
direct users of librados should be aware of.  For a full summary of
those changes, please see the complete release notes:

 * http://ceph.com/docs/master/release-notes/#v0-72-emperor

The next major release of Ceph, Firefly, is scheduled for release in
February of 2014.

You can download v0.72 Emperor from the usual locations:

 * Git at git://github.com/ceph/ceph.git
 * Tarball at http://ceph.com/download/ceph-0.72.tar.gz
 * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
 * For RPMs, see http://ceph.com/docs/master/install/rpm
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] please help me.problem with my ceph

[Karan Singh]

Hello Joseph

This sounds like a solution , BTW how to set replication level to 1 , is there 
any direct command or need to edit configuration file.

Many Thanks
Karan Singh


- Original Message -
From: Joseph R Gruher joseph.r.gru...@intel.com
To: ceph-users@lists.ceph.com
Sent: Thursday, 7 November, 2013 9:14:45 PM
Subject: Re: [ceph-users] please help me.problem with my ceph

From: ceph-users-boun...@lists.ceph.com [mailto:ceph-users-
boun...@lists.ceph.com] On Behalf Of ??
Sent: Wednesday, November 06, 2013 10:04 PM
To: ceph-users
Subject: [ceph-users] please help me.problem with my ceph

1.  I have installed ceph with one mon/mds and one osd.When i use 'ceph -
s',there si a warning:health HEALTH_WARN 384 pgs degraded; 384 pgs stuck
unclean; recovery 21/42 degraded (50.000%) 

I would think this is because Ceph defaults to a replication level of 2 and you 
only have one OSD (nowhere to write a second copy) so you are degraded?  You 
could add a second OSD or perhaps you could set the replication level to 1?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Unable to find bootstrap-osd and bootstrap-mds ceph.keyring

[Karan Singh]

Hello Vikrant 

You can try creating directories manually on the monitor node 

mkdir -p  /var/lib/ceph/{tmp,mon,mds,bootstrap-osd} 


* Important Do not call ceph-deploy with sudo or run it as root if you are 
logged in as a different user, because it will not issue sudo commands needed 
on the remote host. 

Try this hope this helps you. 

Many Thanks 
Karan Singh 


- Original Message -

From: Vikrant Verma vikrantverm...@gmail.com 
To: ceph-users@lists.ceph.com 
Sent: Friday, 8 November, 2013 2:41:36 PM 
Subject: [ceph-users] Unable to find bootstrap-osd and bootstrap-mds 
ceph.keyring 

Hi All, 



I am able to Add a Ceph Monitor (step 3) as per the link 
http://ceph.com/docs/master/start/quick-ceph-deploy/ (Setting Up Ceph Storage 
Cluster) 



But when I am executing the gatherkey command, I am getting the 
warnings(highlighted in yellow). Please find the details – 



Command – “ ceph-deploy gatherkeys vikrant” (vikrant is the hostname of the 
ceph-node1) 

Output – 

[ceph_deploy.cli][INFO ] Invoked (1.3.1): /usr/bin/ceph-deploy gatherkeys 
vikrant 
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.client.admin.keyring 
[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring 
[ceph_deploy.gatherkeys][DEBUG ] Checking vikrant for 
/var/lib/ceph/bootstrap-osd/ceph.keyring 
[vikrant][DEBUG ] connected to host: vikrant 
[vikrant][DEBUG ] detect platform information from remote host 
[vikrant][DEBUG ] detect machine type 
[vikrant][DEBUG ] fetch remote file 
[ceph_deploy.gatherkeys][WARNIN] Unable to find 
/var/lib/ceph/bootstrap-osd/ceph.keyring on ['vikrant'] 
[ceph_deploy.gatherkeys][DEBUG ] Checking vikrant for 
/var/lib/ceph/bootstrap-mds/ceph.keyring 
[vikrant][DEBUG ] connected to host: vikrant 
[vikrant][DEBUG ] detect platform information from remote host 
[vikrant][DEBUG ] detect machine type 
[vikrant][DEBUG ] fetch remote file 
[ceph_deploy.gatherkeys][WARNIN] Unable to find 
/var/lib/ceph/bootstrap-mds/ceph.keyring on ['vikrant'] 



I checked ceph.keyring is not generated for bootstrap-osd and bootstrap-mds in 
ceph-node1, due to which the next command “ ceph-deploy osd prepare ceph-node2” 
is giving error. 



Please find the setup details – 

One Admin Node – from where I am executing ceph-deploy commands 

Ceph-node1 – this is the ceph monitor, (hostname is vikrant) 

Ceph-node2 – Ceph OSD, this is on a separate machine ( as of now I am trying to 
configure one OSD, in the link they have mentioned the example for two OSD) 



Content of ceph.conf (this is same for admin node and ceph-node1) 

[global] 
fsid = eb4099a6-d2ab-437c-94f2-f3b43b3170d1 
mon_initial_members = vikrant 
mon_host = 10.XX.XX.XX 
auth_supported = cephx 
osd_journal_size = 1024 
filestore_xattr_use_omap = true 





Output of “ceph-deploy mon create vikrant” command (vikrant is the hostname of 
the ceph-node1) -- 

-ceph-deploy mon create vikrant 
-o/p-- 

[ceph_deploy.cli][INFO ] Invoked (1.3.1): /usr/bin/ceph-deploy mon create 
vikrant 

[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts vikrant 

[ceph_deploy.mon][DEBUG ] detecting platform for host vikrant ... 

[vikrant][DEBUG ] connected to host: vikrant 

[vikrant][DEBUG ] detect platform information from remote host 

[vikrant][DEBUG ] detect machine type 

[ceph_deploy.mon][INFO ] distro info: Ubuntu 12.04 precise 

[vikrant][DEBUG ] determining if provided host has same hostname in remote 

[vikrant][DEBUG ] get remote short hostname 

[vikrant][DEBUG ] deploying mon to vikrant 

[vikrant][DEBUG ] get remote short hostname 

[vikrant][DEBUG ] remote hostname: vikrant 

[vikrant][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf 

[vikrant][DEBUG ] create the mon path if it does not exist 

[vikrant][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-vikrant/done 

[vikrant][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-vikrant/done 

[vikrant][INFO ] creating tmp path: /var/lib/ceph/tmp 

[vikrant][INFO ] creating keyring file: 
/var/lib/ceph/tmp/ceph-vikrant.mon.keyring 

[vikrant][DEBUG ] create the monitor keyring file 

[vikrant][INFO ] Running command: sudo ceph-mon --cluster ceph --mkfs -i 
vikrant --keyring /var/lib/ceph/tmp/ceph-vikrant.mon.keyring 

[vikrant][DEBUG ] ceph-mon: mon.noname-a 10.XX.XX.XX:6789/0 is local, renaming 
to mon.vikrant 

[vikrant][DEBUG ] ceph-mon: set fsid to eb4099a6-d2ab-437c-94f2-f3b43b3170d1 

[vikrant][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-vikrant for 
mon.vikrant 

[vikrant][INFO ] unlinking keyring file 
/var/lib/ceph/tmp/ceph-vikrant.mon.keyring 

[vikrant][DEBUG ] create a done file to avoid re-doing the mon deployment 

[vikrant][DEBUG ] create the init path if it does not exist 

[vikrant][DEBUG ] locating the `service` executable... 

[vikrant][INFO ] Running command: sudo initctl emit ceph-mon cluster=ceph 
id=vikrant 

[vikrant][INFO ] Running command: sudo ceph --cluster=ceph

Re: [ceph-users] Puppet Modules for Ceph

[Karan Singh]

Dear Cephers 

I have a running ceph cluster that was deployed using ceph-deploy , our next 
objective is to build a Puppet setup that can be used for long term scaling of 
ceph infrastructure.

It would be a great help if any one can 

1) Provide ceph modules for (centos OS)
2) Guidance on how to proceed

Many Thanks
Karan Singh


- Original Message -
From: Karan Singh ksi...@csc.fi
To: Loic Dachary l...@dachary.org
Cc: ceph-users@lists.ceph.com
Sent: Monday, 4 November, 2013 5:01:26 PM
Subject: Re: [ceph-users] Ceph deployment using puppet

Hello Loic

Thanks for your reply , Ceph-deploy works good to me.

My next objective is to deploy ceph using puppet. Can you guide me now i can 
proceed.

Regards
karan

- Original Message -
From: Loic Dachary l...@dachary.org
To: ceph-users@lists.ceph.com
Sent: Monday, 4 November, 2013 4:45:06 PM
Subject: Re: [ceph-users] Ceph deployment using puppet

Hi,

Unless you're force to use puppet for some reason, I suggest you give 
ceph-deploy a try:

http://ceph.com/docs/master/start/quick-ceph-deploy/

Cheers

On 04/11/2013 19:00, Karan Singh wrote:
 Hello Everyone
 
 Can  someone guide me how i can start for  ceph deployment using puppet  , 
 what all things i need to have for this .
 
 I have no prior idea of using puppet , hence need your help to getting 
 started with it.
 
 
 Regards
 Karan Singh
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 

-- 
Loïc Dachary, Artisan Logiciel Libre


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Puppet Modules for Ceph

[Karan Singh]

A Big thanks Don for creating puppet modules .

Need your guidance on -

1) Did you manage to run this on centos
2) What all things can be installed using these modules ( mon , osd , mds OR 
All )
3) What all things i need to change in this module


Many Thanks
Karan Singh


- Original Message -
From: Don Talton (dotalton) dotal...@cisco.com
To: Karan Singh ksi...@csc.fi, ceph-users@lists.ceph.com, 
ceph-users-j...@lists.ceph.com, ceph-us...@ceph.com
Sent: Wednesday, 6 November, 2013 6:49:16 PM
Subject: RE: [ceph-users] Puppet Modules for Ceph

This will work https://github.com/dontalton/puppet-cephdeploy

Just change the unless statements (should only be two) from testing dpkg to 
testing rpm instead.
I'll add an OS check myself, or you can fork and send me a pull request.

 -Original Message-
 From: ceph-users-boun...@lists.ceph.com [mailto:ceph-users-
 boun...@lists.ceph.com] On Behalf Of Karan Singh
 Sent: Wednesday, November 06, 2013 7:56 PM
 To: ceph-users@lists.ceph.com; ceph-users-j...@lists.ceph.com; ceph-
 us...@ceph.com
 Subject: Re: [ceph-users] Puppet Modules for Ceph
 
 Dear Cephers
 
 I have a running ceph cluster that was deployed using ceph-deploy , our next
 objective is to build a Puppet setup that can be used for long term scaling of
 ceph infrastructure.
 
 It would be a great help if any one can
 
 1) Provide ceph modules for (centos OS)
 2) Guidance on how to proceed
 
 Many Thanks
 Karan Singh
 
 
 - Original Message -
 From: Karan Singh ksi...@csc.fi
 To: Loic Dachary l...@dachary.org
 Cc: ceph-users@lists.ceph.com
 Sent: Monday, 4 November, 2013 5:01:26 PM
 Subject: Re: [ceph-users] Ceph deployment using puppet
 
 Hello Loic
 
 Thanks for your reply , Ceph-deploy works good to me.
 
 My next objective is to deploy ceph using puppet. Can you guide me now i
 can proceed.
 
 Regards
 karan
 
 - Original Message -
 From: Loic Dachary l...@dachary.org
 To: ceph-users@lists.ceph.com
 Sent: Monday, 4 November, 2013 4:45:06 PM
 Subject: Re: [ceph-users] Ceph deployment using puppet
 
 Hi,
 
 Unless you're force to use puppet for some reason, I suggest you give ceph-
 deploy a try:
 
 http://ceph.com/docs/master/start/quick-ceph-deploy/
 
 Cheers
 
 On 04/11/2013 19:00, Karan Singh wrote:
  Hello Everyone
 
  Can  someone guide me how i can start for  ceph deployment using
 puppet  , what all things i need to have for this .
 
  I have no prior idea of using puppet , hence need your help to getting
 started with it.
 
 
  Regards
  Karan Singh
 
 
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 --
 Loïc Dachary, Artisan Logiciel Libre
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph deployment using puppet

[Karan Singh]

Hello Everyone 

Can someone guide me how i can start for  ceph deployment using puppet  , 
what all things i need to have for this . 

I have no prior idea of using puppet , hence need your help to getting started 
with it. 


Regards 
Karan Singh 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph deployment using puppet

[Karan Singh]

Hello Loic

Thanks for your reply , Ceph-deploy works good to me.

My next objective is to deploy ceph using puppet. Can you guide me now i can 
proceed.

Regards
karan

- Original Message -
From: Loic Dachary l...@dachary.org
To: ceph-users@lists.ceph.com
Sent: Monday, 4 November, 2013 4:45:06 PM
Subject: Re: [ceph-users] Ceph deployment using puppet

Hi,

Unless you're force to use puppet for some reason, I suggest you give 
ceph-deploy a try:

http://ceph.com/docs/master/start/quick-ceph-deploy/

Cheers

On 04/11/2013 19:00, Karan Singh wrote:
 Hello Everyone
 
 Can  someone guide me how i can start for  ceph deployment using puppet  , 
 what all things i need to have for this .
 
 I have no prior idea of using puppet , hence need your help to getting 
 started with it.
 
 
 Regards
 Karan Singh
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 

-- 
Loïc Dachary, Artisan Logiciel Libre


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Install Guide

[Karan Singh]

Hello Raghavendra 

I would recommend you to follow Inktank Webinar and ceph documentation to go 
with the basics of ceph first. 


As a answer to your question : You would need ADMIN-NODE , MONITOR-NODE , 
OSD-NODE and CLIENT-NODE ( as testing you can configure in 1 or 2 VMs ) 

Ceph Documentation : http://ceph.com/docs/master/ 

Regards 
Karan Singh 

- Original Message -

From: Raghavendra Lad raghavendra_...@rediffmail.com 
To: ceph-users@lists.ceph.com 
Sent: Friday, 1 November, 2013 7:27:26 AM 
Subject: [ceph-users] Ceph Install Guide 

Hi, 

Please can you help with the Ceph Install guide. 

Do we need to install Ceph server or client? 

Regards, 
Raghavendra Lad 
Get your own FREE website, FREE domain  FREE mobile app with Company email. 
Know More  
___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph-deploy 1.3 searching for packages in incorrect path

[Karan Singh]

 ] error: skipping 
http://ceph.com/rpm-testing/noarch/ceph-release-1-0.el6.noarch.rpm - transfer 
failed 
[ceph-client1][DEBUG ] Retrieving 
http://ceph.com/rpm-testing/noarch/ceph-release-1-0.el6.noarch.rpm 







4) And Finally i tried ceph-deploy from my other node running versino 1.2.7 
using command ceph-deploy install ceph-client1 it went FINE and every thing is 
working 







Regards 

Karan Singh 








___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com