Re: [ceph-users] [PG] Slow request *** seconds old,v4 currently waiting for pg to exist locally

2014-09-25 Thread Udo Lembke
Hi, looks that some osds are down?! What is the output of ceph osd tree Udo Am 25.09.2014 04:29, schrieb Aegeaner: The cluster healthy state is WARN: health HEALTH_WARN 118 pgs degraded; 8 pgs down; 59 pgs incomplete; 28 pgs peering; 292 pgs stale; 87 pgs stuck inactive;

Re: [ceph-users] [PG] Slow request *** seconds old,v4 currently waiting for pg to exist locally

2014-09-25 Thread Udo Lembke
Hi again, sorry - forgot my post... see osdmap e421: 9 osds: 9 up, 9 in shows that all your 9 osds are up! Do you have trouble with your journal/filesystem? Udo Am 25.09.2014 08:01, schrieb Udo Lembke: Hi, looks that some osds are down?! What is the output of ceph osd tree Udo Am

Re: [ceph-users] [PG] Slow request *** seconds old,v4 currently waiting for pg to exist locally

2014-09-25 Thread Irek Fasikhov
osd_op(client.4625.1:9005787) . This is due to external factors. For example, the network settings. 2014-09-25 10:05 GMT+04:00 Udo Lembke ulem...@polarzone.de: Hi again, sorry - forgot my post... see osdmap e421: 9 osds: 9 up, 9 in shows that all your 9 osds are up! Do you have

Re: [ceph-users] [PG] Slow request *** seconds old,v4 currently waiting for pg to exist locally

2014-09-25 Thread Aegeaner
Yeah, three of nine OSDs went down but I recreated them, but the pgs cannot be recovered. I don't know how to erase all the pgs, so I deleted all the osd pools, including data and metadata … Now all pgs are active and clean... I'm not sure if there are more elegant ways to deal with this.

Re: [ceph-users] [Ceph-community] Pgs are in stale+down+peering state

2014-09-25 Thread Sahana Lokeshappa
Replies Inline : Sahana Lokeshappa Test Development Engineer I SanDisk Corporation 3rd Floor, Bagmane Laurel, Bagmane Tech Park C V Raman nagar, Bangalore 560093 T: +918042422283 sahana.lokesha...@sandisk.com -Original Message- From: Sage Weil [mailto:sw...@redhat.com] Sent: Wednesday,

Re: [ceph-users] [Ceph-community] Pgs are in stale+down+peering state

2014-09-25 Thread Sahana Lokeshappa
Hi Craig, Sorry for late response. Somehow missed this mail. All osds are up and running. There were no specific logs related to this activity. And, there are no IOs running right now. Few osds were made in and out ,removed fully and recreated before these pgs coming to this stage. I had tried

Re: [ceph-users] [Ceph-community] Pgs are in stale+down+peering state

2014-09-25 Thread Sahana Lokeshappa
Hi All, Here are the steps I followed, to get back all pgs to active+clean state. Still don't know what is the root cause for this pg state. 1. Force create pgs which are in stale+down+peering 2. Stop osd.12 3. Mark osd.12 as lost 4. Start osd.12 5. All pgs were back to active+clean state

Re: [ceph-users] Frequent Crashes on rbd to nfs gateway Server

2014-09-25 Thread Micha Krause
Hi, That's strange. 3.13 is way before any changes that could have had any such effect. Can you by any chance try with older kernels to see where it starts misbehaving for you? 3.12? 3.10? 3.8? my crush tunables are set to bobtail, so I can't go bellow 3.9, I will try 3.12 tomorrow

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

2014-09-25 Thread Alexandre DERUMIER
As Dieter asked, what replication level is this, I guess 1? Yes, replication x1 for theses benchmarks. Now at 3 nodes and 6 OSDs you're getting about the performance of a single SSD, food for thought. yes, sure . I don't have more nodes to test, but I would like to known if it's scale more

Re: [ceph-users] bug: ceph-deploy does not support jumbo frame

2014-09-25 Thread yuelongguang
thanks. i have not configured switch. i just know about it. 在 2014-09-25 12:38:48,Irek Fasikhov malm...@gmail.com 写道: You have configured the switch? 2014-09-25 5:07 GMT+04:00 yuelongguang fasts...@163.com: hi,all after i set mtu=9000, ceph-deply waits reply all the time ,

Re: [ceph-users] Frequent Crashes on rbd to nfs gateway Server

2014-09-25 Thread Andrei Mikhailovsky
Guys, Have done some testing with 3.16.3-031603-generic downloaded from Ubuntu utopic branch. The hang task problem is gone when using large block size (tested with 1M and 4M) and I could no longer preproduce the hang tasks while doing 100 dd tests in a for loop. However, I can confirm that

[ceph-users] pgs stuck in active+clean+replay state

2014-09-25 Thread Pavel V. Kaygorodov
Hi! 16 pgs in our ceph cluster are in active+clean+replay state more then one day. All clients are working fine. Is this ok? root@bastet-mon1:/# ceph -w cluster fffeafa2-a664-48a7-979a-517e3ffa0da1 health HEALTH_OK monmap e3: 3 mons at

Re: [ceph-users] Frequent Crashes on rbd to nfs gateway Server

2014-09-25 Thread Andrei Mikhailovsky
Right, I've stopped the tests because it is just getting ridiculous. Without rbd cache enabled, dd tests run extremely slow: dd if=/dev/zero of=/tmp/mount/1G bs=1M count=1000 oflag=direct 230+0 records in 230+0 records out 241172480 bytes (241 MB) copied, 929.71 s, 259 kB/s Any thoughts

[ceph-users] ceph debian systemd

2014-09-25 Thread zorg
Hi, I'm using ceph version 0.80.5 I trying to make work a ceph cluster using debian and systemd I have already manage to install ceph cluster on debian with sysinit without any problem But after installing all, using ceph deploy without error after rebooting not all my osd start (they are

Re: [ceph-users] Frequent Crashes on rbd to nfs gateway Server

2014-09-25 Thread Ilya Dryomov
On Thu, Sep 25, 2014 at 1:58 PM, Andrei Mikhailovsky and...@arhont.com wrote: Guys, Have done some testing with 3.16.3-031603-generic downloaded from Ubuntu utopic branch. The hang task problem is gone when using large block size (tested with 1M and 4M) and I could no longer preproduce the

Re: [ceph-users] [ceph-calamari] Setting up Ceph calamari :: Made Simple

2014-09-25 Thread Johan Kooijman
Karan, Thanks for the tutorial, great stuff. Please note that in order to get the graphs working, I had to install ipvsadm and create a symlink from /sbin/ipvsadm to /usr/bin/ipvsadm (CentOS 6). On Wed, Sep 24, 2014 at 10:16 AM, Karan Singh karan.si...@csc.fi wrote: Hello Cepher’s Now here

Re: [ceph-users] Frequent Crashes on rbd to nfs gateway Server

2014-09-25 Thread Andrei Mikhailovsky
Ilya, I've not used rbd map on older kernels. Just experimenting with rbd map to have an iscsi and nfs gateway service for hypervisors such as xenserver and vmware. I've tried it with the latest ubuntu LTS kernel 3.13 I believe and noticed the issue. Can you not reproduce the hang tasks when

[ceph-users] v0.67.11 dumpling released

2014-09-25 Thread Sage Weil
v0.67.11 Dumpling === This stable update for Dumpling fixes several important bugs that affect a small set of users. We recommend that all Dumpling users upgrade at their convenience. If none of these issues are affecting your deployment there is no urgency. Notable Changes

Re: [ceph-users] Frequent Crashes on rbd to nfs gateway Server

2014-09-25 Thread Ilya Dryomov
On Thu, Sep 25, 2014 at 7:06 PM, Andrei Mikhailovsky and...@arhont.com wrote: Ilya, I've not used rbd map on older kernels. Just experimenting with rbd map to have an iscsi and nfs gateway service for hypervisors such as xenserver and vmware. I've tried it with the latest ubuntu LTS kernel

[ceph-users] Icehouse Ceph -- live migration fails?

2014-09-25 Thread Daniel Schneller
Hi! We have an Icehouse system running with librbd based Cinder and Glance configurations, storing images and volumes in Ceph. Configuration is (apart from network setup details, of course) by the book / OpenStack setup guide. Works very nicely, including regular migration, but live migration

Re: [ceph-users] v0.67.11 dumpling released

2014-09-25 Thread Mike Dawson
On 9/25/2014 11:09 AM, Sage Weil wrote: v0.67.11 Dumpling === This stable update for Dumpling fixes several important bugs that affect a small set of users. We recommend that all Dumpling users upgrade at their convenience. If none of these issues are affecting your deployment

Re: [ceph-users] v0.67.11 dumpling released

2014-09-25 Thread Sage Weil
On Thu, 25 Sep 2014, Mike Dawson wrote: On 9/25/2014 11:09 AM, Sage Weil wrote: v0.67.11 Dumpling === This stable update for Dumpling fixes several important bugs that affect a small set of users. We recommend that all Dumpling users upgrade at their convenience.

Re: [ceph-users] v0.67.11 dumpling released

2014-09-25 Thread Dan Van Der Ster
Hi Mike, On 25 Sep 2014, at 17:47, Mike Dawson mike.daw...@cloudapt.com wrote: On 9/25/2014 11:09 AM, Sage Weil wrote: v0.67.11 Dumpling === This stable update for Dumpling fixes several important bugs that affect a small set of users. We recommend that all Dumpling

Re: [ceph-users] [Ceph-maintainers] v0.67.11 dumpling released

2014-09-25 Thread Loic Dachary
Hi, On 25/09/2014 17:53, Sage Weil wrote: On Thu, 25 Sep 2014, Mike Dawson wrote: On 9/25/2014 11:09 AM, Sage Weil wrote: v0.67.11 Dumpling === This stable update for Dumpling fixes several important bugs that affect a small set of users. We recommend that all Dumpling

Re: [ceph-users] v0.67.11 dumpling released

2014-09-25 Thread Sage Weil
On Thu, 25 Sep 2014, Dan Van Der Ster wrote: Hi Mike, On 25 Sep 2014, at 17:47, Mike Dawson mike.daw...@cloudapt.com wrote: On 9/25/2014 11:09 AM, Sage Weil wrote: v0.67.11 Dumpling === This stable update for Dumpling fixes several important bugs that affect a

Re: [ceph-users] [ceph-calamari] Setting up Ceph calamari :: Made Simple

2014-09-25 Thread Dan Mick
Can you explain this a little more, Johan? I've never even heard of ipvsadmin or its facilities before today, and it ought not be required... On Sep 25, 2014 7:04 AM, Johan Kooijman m...@johankooijman.com wrote: Karan, Thanks for the tutorial, great stuff. Please note that in order to get the

Re: [ceph-users] pgs stuck in active+clean+replay state

2014-09-25 Thread Gregory Farnum
I imagine you aren't actually using the data/metadata pool that these PGs are in, but it's a previously-reported bug we haven't identified: http://tracker.ceph.com/issues/8758 They should go away if you restart the OSDs that host them (or just remove those pools), but it's not going to hurt

Re: [ceph-users] v0.67.11 dumpling released

2014-09-25 Thread Mike Dawson
Looks like the packages have partially hit the repo, but at least the following are missing: Failed to fetch http://ceph.com/debian-dumpling/pool/main/c/ceph/librbd1_0.67.11-1precise_amd64.deb 404 Not Found Failed to fetch

Re: [ceph-users] v0.67.11 dumpling released

2014-09-25 Thread Alfredo Deza
On Thu, Sep 25, 2014 at 1:27 PM, Mike Dawson mike.daw...@cloudapt.com wrote: Looks like the packages have partially hit the repo, but at least the following are missing: Failed to fetch http://ceph.com/debian-dumpling/pool/main/c/ceph/librbd1_0.67.11-1precise_amd64.deb 404 Not Found Failed

Re: [ceph-users] pgs stuck in active+clean+replay state

2014-09-25 Thread Pavel V. Kaygorodov
Hi! I imagine you aren't actually using the data/metadata pool that these PGs are in, but it's a previously-reported bug we haven't identified: http://tracker.ceph.com/issues/8758 They should go away if you restart the OSDs that host them (or just remove those pools), but it's not going to

Re: [ceph-users] Any way to remove possible orphaned files in a federated gateway configuration

2014-09-25 Thread Lyn Mitchell
Thanks Yehuda for your response, much appreciated. Using the radosgw-admin object stat option I was able to reconcile the objects on master and slave. There are 10 objects on the master that have replicated to the slave, for these 10 objects I was able to confirm by pulling the tag prefix

Re: [ceph-users] RBD import slow

2014-09-25 Thread Josh Durgin
On 09/24/2014 04:57 PM, Brian Rak wrote: I've been doing some testing of importing virtual machine images, and I've found that 'rbd import' is at least 2x as slow as 'qemu-img convert'. Is there anything I can do to speed this process up? I'd like to use rbd import because it gives me a little

[ceph-users] Best practice about using multiple disks on one single OSD

2014-09-25 Thread James Pan
Hi, I have several servers and each server has 4 disks. Now I am going to setup Ceph on these servers and use all the 4 disks but it seems one OSD instance can be configured with one backend storage. So there seems two options to me: 1. Make the 4 disks into a raid0 then setup OSD to use this

Re: [ceph-users] Best practice about using multiple disks on one single OSD

2014-09-25 Thread Jean-Charles LOPEZ
Hi James, the best practice is to set up 1 OSD daemon per physical disk drive. In your case, each OSD node would hence be 4 OSD daemons using one physical drive per daemon, and deploying a minimum of 3 servers so each object copy resides on a separate physical server. JC On Sep 25, 2014,

[ceph-users] iptables

2014-09-25 Thread shiva rkreddy
Hello, On my ceph cluster osd node . there is a rule to REJECT all. As per the documentation, added a rule to allow the trafficon the full range of ports, But, the cluster will not come into clean state. Can you please share your experience with the iptables configuration. Following are the INPUT