Re: [ceph-users] unable to repair PG
Hi Greg, thanks for your help. It's always highly appreciated. :) On Thu, Dec 11, 2014 at 6:41 PM, Gregory Farnum g...@gregs42.com wrote: On Thu, Dec 11, 2014 at 2:57 AM, Luis Periquito periqu...@gmail.com wrote: Hi, I've stopped OSD.16, removed the PG from the local filesystem and started the OSD again. After ceph rebuilt the PG in the removed OSD I ran a deep-scrub and the PG is still inconsistent. What led you to remove it from osd 16? Is that the one hosting the log you snipped from? Is osd 16 the one hosting shard 6 of that PG, or was it the primary? OSD 16 is both the primary for this PG and the one that has the snipped log. The other 3 OSDs has any mention of this PG in their logs. Just some messages about slow requests and the backfill when I removed the object. Actually it came from OSD.6 - currently we don't have OSD.3. this is the output of the pg dump for this PG 9.180256140002330648234830013001 active+clean+inconsistent2014-12-10 17:29:01.93792940242'1108124 40242:23305321[16,10,27,6]16[16,10,27,6]1640242'1071363 2014-12-10 17:29:01.93788140242'10713632014-12-10 17:29:01.937881 Anyway, the message means that shard 6 (which I think is the seventh OSD in the list) of PG 9.180 is missing a bunch of xattrs on object 370cbf80/29145.4_xxx/head//9. I'm actually a little surprised it didn't crash if it's missing the _ attr -Greg Any idea on how to fix it? I'm running out of ideas on trying to solve this. Does this mean that all copies of the object should also be inconsistent? Should I just try to figure which object/bucket this belongs to and delete it/copy it again to the ceph cluster? Also, do you know what the error message means? is it just some sort of metadata for this object that isn't correct, not the object itself? On Wed, Dec 10, 2014 at 11:11 AM, Luis Periquito periqu...@gmail.com wrote: Hi, In the last few days this PG (pool is .rgw.buckets) has been in error after running the scrub process. After getting the error, and trying to see what may be the issue (and finding none), I've just issued a ceph repair followed by a ceph deep-scrub. However it doesn't seem to have fixed the issue and it still remains. The relevant log from the OSD is as follows. 2014-12-10 09:38:09.348110 7f8f618be700 0 log [ERR] : 9.180 deep-scrub 0 missing, 1 inconsistent objects 2014-12-10 09:38:09.348116 7f8f618be700 0 log [ERR] : 9.180 deep-scrub 1 errors 2014-12-10 10:13:15.922065 7f8f618be700 0 log [INF] : 9.180 repair ok, 0 fixed 2014-12-10 10:55:27.556358 7f8f618be700 0 log [ERR] : 9.180 shard 6: soid 370cbf80/29145.4_xxx/head//9 missing attr _, missing attr _user.rgw.acl, missing attr _user.rgw.content_type, missing attr _user.rgw.etag, missing attr _user.rgw.idtag, missing attr _user.rgw.manifest, missing attr _user.rgw.x-amz-meta-md5sum, missing attr _user.rgw.x-amz-meta-stat, missing attr snapset 2014-12-10 10:56:50.597952 7f8f618be700 0 log [ERR] : 9.180 deep-scrub 0 missing, 1 inconsistent objects 2014-12-10 10:56:50.597957 7f8f618be700 0 log [ERR] : 9.180 deep-scrub 1 errors I'm running version firefly 0.80.7. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] AWS SDK and MultiPart Problem
Dear Yehuda, I have installed the patched version as you can see: $ radosgw --version ceph version 0.80.7-1-gbd43759 (bd43759f6e76fa827e2534fa4e61547779ee10a5) $ ceph --version ceph version 0.80.7-1-gbd43759 (bd43759f6e76fa827e2534fa4e61547779ee10a5) $ sudo yum info ceph-radosgw Installed Packages Name: ceph-radosgw Arch: x86_64 Version : 0.80.7 Release : 1.gbd43759.el6 Size: 3.8 M Repo: installed From repo : ceph-source Summary : Rados REST gateway URL : http://ceph.com/ License : GPL-2.0 Description : radosgw is an S3 HTTP REST gateway for the RADOS object store. It is : implemented as a FastCGI module using libfcgi, and can be used in : conjunction with any FastCGI capable web server. Unfortunately the problem on the multipart upload with aws-sdk still remains the same! Here is a part of the apache log: PUT /clients-space/test/iip7.dmg?partNumber=3uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=1uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=2uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 Directly modification of the binary so that the 2%2F be changed to 2- results in success and here is the log: PUT /clients-space/test/iip7.dmg?partNumber=1uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=2uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=4uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 POST /clients-space/test/iip7.dmg?uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 302 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 Can you think of something else?? Best regards, George OK! I will give it some time and will try again later! Thanks a lot for your help! Warmest regards, George The branch I pushed earlier was based off recent development branch. I just pushed one based off firefly (wip-10271-firefly). It will probably take a bit to build. Yehuda On Thu, Dec 11, 2014 at 12:03 PM, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote: Hi again! I have installed and enabled the development branch repositories as described here: http://ceph.com/docs/master/install/get-packages/#add-ceph-development and when I try to update the ceph-radosgw package I get the following: Installed Packages Name: ceph-radosgw Arch: x86_64 Version : 0.80.7 Release : 0.el6 Size: 3.8 M Repo: installed From repo : Ceph Summary : Rados REST gateway URL : http://ceph.com/ License : GPL-2.0 Description : radosgw is an S3 HTTP REST gateway for the RADOS object store. It is : implemented as a FastCGI module using libfcgi, and can be used in : conjunction with any FastCGI capable web server. Available Packages Name: ceph-radosgw Arch: x86_64 Epoch : 1 Version : 0.80.5 Release : 9.el6 Size: 1.3 M Repo: epel Summary : Rados REST gateway URL : http://ceph.com/ License : GPL-2.0 Description : radosgw is an S3 HTTP REST gateway for the RADOS object store. It is : implemented as a FastCGI module using libfcgi, and can be used in : conjunction with any FastCGI capable web server. Is this normal??? I am concerned because the installed version is 0.80.7 and the available update package is 0.80.5 Have I missed something? Regards, George Pushed a fix to wip-10271. Haven't tested it though, let me know if you try it. Thanks, Yehuda On Thu, Dec 11, 2014 at 8:38 AM, Yehuda Sadeh yeh...@redhat.com wrote: I don't think it has been fixed recently. I'm looking at it now, and not sure why it hasn't triggered before in other areas. Yehuda On Thu, Dec 11, 2014 at 5:55 AM, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote: This issue seems very similar to these: http://tracker.ceph.com/issues/8202 http://tracker.ceph.com/issues/8702 Would it make any difference if I try to build CEPH from sources? I mean is someone aware of it been fixed on any of the recent commits and probably hasn't passed yet to the repositories? Regards, George On Mon, 08 Dec 2014 19:47:59 +0200, Georgios Dimitrakakis wrote: I 've just created issues #10271 Best, George On Fri, 5 Dec 2014 09:30:45 -0800, Yehuda Sadeh wrote: It looks like a bug. Can you open an issue on tracker.ceph.com, describing what you see? Thanks, Yehuda On Fri, Dec 5, 2014 at 7:17 AM, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote: It would be nice to see where
[ceph-users] ceph blk-mq
On latest kernels, for most HDDs (for me actual spinning with SCSI interface - IDE/SATA in AHCI mode Megaraid) IO schedulers can be replaced by blk-mq per-CPU queue. Even I put one node with 3.18 kernel into this mode (Megaraid, scsi_mod.use_blk_mq=Y), planned to switch all nodes (include AHCI). So, I have ceph-specific question. Are you prognosed this way effective, according to multiple op threads (I have default 2)? Or in other words: will ceph too frequental split one sequental client stream into multiple op threads inside one PG (and significant reduce merging per-CPU, cause perfomance degradation) or there will be mostly one thread? -- WBR, Dzianis Kahanovich AKA Denis Kaganovich, http://mahatma.bspu.unibel.by/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph Block device and Trim/Discard
I am new to Ceph and start discovering its features. I used ext4 partitions (also mounted with -o discard) to place several osd on them. Then I created an erasure coded pool in this cluster. On top of this there is the rados block device which holds also an ext4 filesystem (of course mounted with -o discard). I started to create some random 1MB files with tempfile and /dev/urandom to mess the filesystem up and use all available space (256MB on my testdrive). After this I deleted everything again. To my surprise the discard-feature did not work. Ceph reports that ~256MB are used for data (after mkfs it was around 0MB). I also tried to use 'fstrim' on the mountpoint but it reports that the discard operation is not supported. But why? It would be great to shrink the used space. Is there a way to achieve this? Or have I done something wrong? In a professional environment you may can live with filesystems that only grow. But on my small home-cluster this really is a problem. Greetings from Germany! P.S.: I use ceph version 0.80.7 as delivered with debian jessie. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Block device and Trim/Discard
On 12/12/2014 12:48 PM, Max Power wrote: I am new to Ceph and start discovering its features. I used ext4 partitions (also mounted with -o discard) to place several osd on them. Then I created an erasure coded pool in this cluster. On top of this there is the rados block device which holds also an ext4 filesystem (of course mounted with -o discard). How are you using Ceph, Kernel RBD, Qemu/KVM? I started to create some random 1MB files with tempfile and /dev/urandom to mess the filesystem up and use all available space (256MB on my testdrive). After this I deleted everything again. To my surprise the discard-feature did not work. Ceph reports that ~256MB are used for data (after mkfs it was around 0MB). I also tried to use 'fstrim' on the mountpoint but it reports that the discard operation is not supported. But why? It depends. Kernel RBD does not support discard/trim yet. Qemu does under certain situations and with special configuration. A search should tell you the parameters. It would be great to shrink the used space. Is there a way to achieve this? Or have I done something wrong? In a professional environment you may can live with filesystems that only grow. But on my small home-cluster this really is a problem. Greetings from Germany! P.S.: I use ceph version 0.80.7 as delivered with debian jessie. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Wido den Hollander Ceph consultant and trainer 42on B.V. Phone: +31 (0)20 700 9902 Skype: contact42on ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Block device and Trim/Discard
Wido den Hollander w...@42on.com hat am 12. Dezember 2014 um 12:53 geschrieben: It depends. Kernel RBD does not support discard/trim yet. Qemu does under certain situations and with special configuration. Ah, Thank you. So this is my problem. I use rbd with the kernel modules. I think I should port my fileserver to qemu/kvm environment then and hope that it is safe to have a big qemu-partition with around 10 TB. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Block device and Trim/Discard
Discard works with virtio-scsi controllers for disks in QEMU. Just use discard=unmap in the disk section (scsi disk). On 12 Dec 2014, at 13:17, Max Power mailli...@ferienwohnung-altenbeken.de wrote: Wido den Hollander w...@42on.com hat am 12. Dezember 2014 um 12:53 geschrieben: It depends. Kernel RBD does not support discard/trim yet. Qemu does under certain situations and with special configuration. Ah, Thank you. So this is my problem. I use rbd with the kernel modules. I think I should port my fileserver to qemu/kvm environment then and hope that it is safe to have a big qemu-partition with around 10 TB. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com Cheers. Sébastien Han Cloud Architect Always give 100%. Unless you're giving blood. Phone: +33 (0)1 49 70 99 72 Mail: sebastien@enovance.com Address : 11 bis, rue Roquépine - 75008 Paris Web : www.enovance.com - Twitter : @enovance signature.asc Description: Message signed with OpenPGP using GPGMail ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] AWS SDK and MultiPart Problem
Ok, I've been digging a bit more. I don't have full radosgw logs for the issue, so if you could provide it (debug rgw = 20), it might help. However, as it is now, I think the issue is with the way the client library is signing the requests. Instead of using the undecoded uploadId, it uses the encoded version for the signature, which doesn't sign correctly. The same would have happened if it would have run against amazon S3 (just tested it). The two solutions that I see are to fix the client library, and/or to modify the character to one that does not require escaping. Sadly the dash character that you were using cannot be used safely in that context. Maybe tilde ('~') would could work. Yehuda On Fri, Dec 12, 2014 at 2:41 AM, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote: Dear Yehuda, I have installed the patched version as you can see: $ radosgw --version ceph version 0.80.7-1-gbd43759 (bd43759f6e76fa827e2534fa4e61547779ee10a5) $ ceph --version ceph version 0.80.7-1-gbd43759 (bd43759f6e76fa827e2534fa4e61547779ee10a5) $ sudo yum info ceph-radosgw Installed Packages Name: ceph-radosgw Arch: x86_64 Version : 0.80.7 Release : 1.gbd43759.el6 Size: 3.8 M Repo: installed From repo : ceph-source Summary : Rados REST gateway URL : http://ceph.com/ License : GPL-2.0 Description : radosgw is an S3 HTTP REST gateway for the RADOS object store. It is : implemented as a FastCGI module using libfcgi, and can be used in : conjunction with any FastCGI capable web server. Unfortunately the problem on the multipart upload with aws-sdk still remains the same! Here is a part of the apache log: PUT /clients-space/test/iip7.dmg?partNumber=3uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=1uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=2uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 Directly modification of the binary so that the 2%2F be changed to 2- results in success and here is the log: PUT /clients-space/test/iip7.dmg?partNumber=1uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=2uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=4uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 POST /clients-space/test/iip7.dmg?uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 302 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 Can you think of something else?? Best regards, George OK! I will give it some time and will try again later! Thanks a lot for your help! Warmest regards, George The branch I pushed earlier was based off recent development branch. I just pushed one based off firefly (wip-10271-firefly). It will probably take a bit to build. Yehuda On Thu, Dec 11, 2014 at 12:03 PM, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote: Hi again! I have installed and enabled the development branch repositories as described here: http://ceph.com/docs/master/install/get-packages/#add-ceph-development and when I try to update the ceph-radosgw package I get the following: Installed Packages Name: ceph-radosgw Arch: x86_64 Version : 0.80.7 Release : 0.el6 Size: 3.8 M Repo: installed From repo : Ceph Summary : Rados REST gateway URL : http://ceph.com/ License : GPL-2.0 Description : radosgw is an S3 HTTP REST gateway for the RADOS object store. It is : implemented as a FastCGI module using libfcgi, and can be used in : conjunction with any FastCGI capable web server. Available Packages Name: ceph-radosgw Arch: x86_64 Epoch : 1 Version : 0.80.5 Release : 9.el6 Size: 1.3 M Repo: epel Summary : Rados REST gateway URL : http://ceph.com/ License : GPL-2.0 Description : radosgw is an S3 HTTP REST gateway for the RADOS object store. It is : implemented as a FastCGI module using libfcgi, and can be used in : conjunction with any FastCGI capable web server. Is this normal??? I am concerned because the installed version is 0.80.7 and the available update package is 0.80.5 Have I missed something? Regards, George Pushed a fix to wip-10271. Haven't tested it though, let me know if you try it. Thanks, Yehuda On Thu, Dec 11, 2014 at 8:38 AM, Yehuda Sadeh yeh...@redhat.com wrote: I don't think it has been fixed recently. I'm looking at it now, and not sure why it hasn't
Re: [ceph-users] AWS SDK and MultiPart Problem
I 'd be more than happy to provide to you all the info but for some unknown reason my radosgw.log is empty. This is the part that I have in ceph.conf [client.radosgw.gateway] host = xxx keyring = /etc/ceph/keyring.radosgw.gateway rgw socket path = /tmp/radosgw.sock rgw dns name = xxx.example.com rgw enable usage log = true rgw usage log tick interval = 30 rgw usage log flush threshold = 1024 rgw usage max shards = 32 rgw usage max user shards = 1 log file = /var/log/ceph/radosgw.log debug ms = 1 debug rgw = 20 but no matter what I put in there the log is empty $ pwd /var/log/ceph $ ls -l radosgw.log -rw-r--r-- 1 root root 0 Nov 30 03:01 radosgw.log I have already started another thread with title Empty Rados log here in ceph-users list since December 4th but haven't heard from anyone yet... If I solve this I will be able to provide you with all the data. Regards, George Ok, I've been digging a bit more. I don't have full radosgw logs for the issue, so if you could provide it (debug rgw = 20), it might help. However, as it is now, I think the issue is with the way the client library is signing the requests. Instead of using the undecoded uploadId, it uses the encoded version for the signature, which doesn't sign correctly. The same would have happened if it would have run against amazon S3 (just tested it). The two solutions that I see are to fix the client library, and/or to modify the character to one that does not require escaping. Sadly the dash character that you were using cannot be used safely in that context. Maybe tilde ('~') would could work. Yehuda On Fri, Dec 12, 2014 at 2:41 AM, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote: Dear Yehuda, I have installed the patched version as you can see: $ radosgw --version ceph version 0.80.7-1-gbd43759 (bd43759f6e76fa827e2534fa4e61547779ee10a5) $ ceph --version ceph version 0.80.7-1-gbd43759 (bd43759f6e76fa827e2534fa4e61547779ee10a5) $ sudo yum info ceph-radosgw Installed Packages Name: ceph-radosgw Arch: x86_64 Version : 0.80.7 Release : 1.gbd43759.el6 Size: 3.8 M Repo: installed From repo : ceph-source Summary : Rados REST gateway URL : http://ceph.com/ License : GPL-2.0 Description : radosgw is an S3 HTTP REST gateway for the RADOS object store. It is : implemented as a FastCGI module using libfcgi, and can be used in : conjunction with any FastCGI capable web server. Unfortunately the problem on the multipart upload with aws-sdk still remains the same! Here is a part of the apache log: PUT /clients-space/test/iip7.dmg?partNumber=3uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=1uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=2uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 Directly modification of the binary so that the 2%2F be changed to 2- results in success and here is the log: PUT /clients-space/test/iip7.dmg?partNumber=1uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=2uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=4uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 POST /clients-space/test/iip7.dmg?uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 302 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 Can you think of something else?? Best regards, George OK! I will give it some time and will try again later! Thanks a lot for your help! Warmest regards, George The branch I pushed earlier was based off recent development branch. I just pushed one based off firefly (wip-10271-firefly). It will probably take a bit to build. Yehuda On Thu, Dec 11, 2014 at 12:03 PM, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote: Hi again! I have installed and enabled the development branch repositories as described here: http://ceph.com/docs/master/install/get-packages/#add-ceph-development and when I try to update the ceph-radosgw package I get the following: Installed Packages Name: ceph-radosgw Arch: x86_64 Version : 0.80.7 Release : 0.el6 Size: 3.8 M Repo: installed From repo : Ceph Summary : Rados REST gateway URL : http://ceph.com/ License : GPL-2.0 Description : radosgw is an S3 HTTP REST gateway for the RADOS object store. It is : implemented as a FastCGI module using libfcgi, and can be used in : conjunction with any FastCGI capable web server. Available Packages Name: ceph-radosgw Arch: x86_64 Epoch
Re: [ceph-users] Empty Rados log
This is very silly of me... The file wasn't writable by apache. I am writing it down for future reference. G. Hi all! I have a CEPH installation with radosgw and the radosgw.log in the /var/log/ceph directory is empty. In the ceph.conf I have log file = /var/log/ceph/radosgw.log debug ms = 1 debug rgw = 20 under the: [client.radosgw.gateway] Any ideas? Best, George ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] xfsprogs missing in rhel6 repository
Hi Guys, xfsprogs is missing in http://ceph.com/rpm-giant/rhel6/x86_64/ Unfortunately it is not avaivable in standard-rhel. Could you please add it as in firefly AND update repodata? Thanks in advance Erik -- Bayerischer Rundfunk; Rundfunkplatz 1; 80335 München Telefon: +49 89 590001; E-Mail: i...@br.de; Website: http://www.BR.de ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Block device and Trim/Discard
On Fri, Dec 12, 2014 at 2:53 PM, Wido den Hollander w...@42on.com wrote: On 12/12/2014 12:48 PM, Max Power wrote: I am new to Ceph and start discovering its features. I used ext4 partitions (also mounted with -o discard) to place several osd on them. Then I created an erasure coded pool in this cluster. On top of this there is the rados block device which holds also an ext4 filesystem (of course mounted with -o discard). How are you using Ceph, Kernel RBD, Qemu/KVM? I started to create some random 1MB files with tempfile and /dev/urandom to mess the filesystem up and use all available space (256MB on my testdrive). After this I deleted everything again. To my surprise the discard-feature did not work. Ceph reports that ~256MB are used for data (after mkfs it was around 0MB). I also tried to use 'fstrim' on the mountpoint but it reports that the discard operation is not supported. But why? It depends. Kernel RBD does not support discard/trim yet. Qemu does Just a note, discard support went into 3.18, which was released a few days ago. Thanks, Ilya ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] tgt / rbd performance
On 12/11/2014 11:39 AM, ano nym wrote: there is a ceph pool on a hp dl360g5 with 25 sas 10k (sda-sdy) on a msa70 which gives me about 600 MB/s continous write speed with rados write bench. tgt on the server with rbd backend uses this pool. mounting local(host) with iscsiadm, sdz is the virtual iscsi device. As you can see, sdz max out with 100%util at ~55MB/s when writing to it. I know that tgt-rbd is more a proof-of-concept then production-ready. Anyway, is someone using it and/or are there any hints to speed it up? Increasing the tgt nr_threads setting helps. Try 64 or 128. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Missing some pools after manual deployment
Hi there, I am using a custom Linux OS, with ceph v0.89. I have been following the monitor bootstrap instructions [1]. I have a problem in that the OS is firmly on the systemd bandwagon and lacks support to run the provided init.d script that runs the nodes. I have tried using the systemd scripts provided and can successfully start the monitor daemon with the ceph-mon command, however when the monitor starts it shows only 1 pool active: # ceph osd lspools 0 rbd, I was wondering why this might be, I expected to see three as is shown in step 16 of [1]. Given that I followed the bootstrap steps verbatim I am led to believe my problem is this: I have missed out some crucial commands, or procedure, that is carried out by the init.d script when starting the daemon. Is this the case? Or have I got the wrong idea? NB. - I have not tried to connect osd nodes to this monitor. - I have successfully set up ceph clusters on Ubuntu before, manually and using ceph-deploy. Thanks for reading, Patrick --- [1] http://docs.ceph.com/docs/v0.80.5/install/manual-deployment/#monitor-bootstrapping ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Missing some pools after manual deployment
On Fri, Dec 12, 2014 at 11:06 AM, Patrick Darley patrick.dar...@codethink.co.uk wrote: Hi there, I am using a custom Linux OS, with ceph v0.89. I have been following the monitor bootstrap instructions [1]. I have a problem in that the OS is firmly on the systemd bandwagon and lacks support to run the provided init.d script that runs the nodes. I have tried using the systemd scripts provided and can successfully start the monitor daemon with the ceph-mon command, however when the monitor starts it shows only 1 pool active: # ceph osd lspools 0 rbd, I was wondering why this might be, I expected to see three as is shown in step 16 of [1]. Given that I followed the bootstrap steps verbatim I am led to believe my problem is this: I have missed out some crucial commands, or procedure, that is carried out by the init.d script when starting the daemon. Is this the case? Or have I got the wrong idea? Ah, as of Giant (v0.88?) the rbd pool is the only one which is created by default. Everything's fine, but we should update that doc so people don't get confused! -Greg ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] xfsprogs missing in rhel6 repository
On Fri, Dec 12, 2014 at 04:57:29PM +, Lukac, Erik wrote: Hi Guys, xfsprogs is missing in http://ceph.com/rpm-giant/rhel6/x86_64/ Unfortunately it is not avaivable in standard-rhel. Could you please add it as in firefly AND update repodata? Thanks in advance Erik Um. Maybe I'm missing the point here, but if you want to run redhat and enjoy all the license goodness thereof, shouldn't you be buying their scalable filesystem add-on product so you can get the official xfsprogs? Sure it costs money - but you are running redhat precisely to do that right? [ Granted, I don't really understand redhat's pricing strategy here... ] And if you don't care about that, wouldn't it then make more sense to just run centos, where xfsprogs is just part of the standard everything? -Marcus Watts ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] pgs stuck degraded, unclean, undersized
Whereabouts to go with this? ceph -s cluster f67ef302-5c31-425d-b0fe-cdc0738f7a62 health HEALTH_WARN 256 pgs degraded; 256 pgs stuck degraded; 256 pgs stuck unclean; 256 pgs stuck undersized; 256 pgs undersized; recovery 10418/447808 objects degraded (2.326%) monmap e7: 3 mons at {0=10.10.10.240:6789/0,1=10.10.10.241:6789/0,2=10.10.10.242:6789/0}, election epoch 514, quorum 0,1,2 0,1,2 mdsmap e11: 0/0/1 up osdmap e524: 2 osds: 2 up, 2 in pgmap v231654: 768 pgs, 6 pools, 795 GB data, 213 kobjects 1632 GB used, 1899 GB / 3532 GB avail 10418/447808 objects degraded (2.326%) 256 active+undersized+degraded 1 active+clean+scrubbing+deep 511 active+clean Nothing stands out in the osd logs. Nothing seems to be happening with ceph - w Seems to have started since I added an mds server as per here: http://www.sebastien-han.fr/blog/2013/05/13/deploy-a-ceph-mds-server/ I have since removed it: http://www.sebastien-han.fr/blog/2012/07/04/remove-a-mds-server-from-a-ceph-cluster/ thanks. p.s I never see my own posts or replies to them in the list, only direct replies - any idea? they aren't turning up in spam. -- Lindsay signature.asc Description: This is a digitally signed message part. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] pgs stuck degraded, unclean, undersized
Sending a new thread as I can't see my own to reply. Solved the stuck pg's by deleting the cephfs andf the pools I created for it. Health returned to ok instantly. Side Note: I had to guess the command ceph fs rm as I could not find docs on it anywhere, and just doing ceph fs gives: Invalid command: missing required parameter ls fs ls : list filesystems Error EINVAL: invalid command -- Lindsay signature.asc Description: This is a digitally signed message part. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] AWS SDK and MultiPart Problem
In any case, I pushed earlier today another fix to the same branch that replaces the slash with a tilde. Let me know if that one works for you. Thanks, Yehuda On Fri, Dec 12, 2014 at 5:59 AM, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote: How silly of me!!! I 've just noticed that the file isn't writable by the apache! I 'll be back with the logs... G. I 'd be more than happy to provide to you all the info but for some unknown reason my radosgw.log is empty. This is the part that I have in ceph.conf [client.radosgw.gateway] host = xxx keyring = /etc/ceph/keyring.radosgw.gateway rgw socket path = /tmp/radosgw.sock rgw dns name = xxx.example.com rgw enable usage log = true rgw usage log tick interval = 30 rgw usage log flush threshold = 1024 rgw usage max shards = 32 rgw usage max user shards = 1 log file = /var/log/ceph/radosgw.log debug ms = 1 debug rgw = 20 but no matter what I put in there the log is empty $ pwd /var/log/ceph $ ls -l radosgw.log -rw-r--r-- 1 root root 0 Nov 30 03:01 radosgw.log I have already started another thread with title Empty Rados log here in ceph-users list since December 4th but haven't heard from anyone yet... If I solve this I will be able to provide you with all the data. Regards, George Ok, I've been digging a bit more. I don't have full radosgw logs for the issue, so if you could provide it (debug rgw = 20), it might help. However, as it is now, I think the issue is with the way the client library is signing the requests. Instead of using the undecoded uploadId, it uses the encoded version for the signature, which doesn't sign correctly. The same would have happened if it would have run against amazon S3 (just tested it). The two solutions that I see are to fix the client library, and/or to modify the character to one that does not require escaping. Sadly the dash character that you were using cannot be used safely in that context. Maybe tilde ('~') would could work. Yehuda On Fri, Dec 12, 2014 at 2:41 AM, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote: Dear Yehuda, I have installed the patched version as you can see: $ radosgw --version ceph version 0.80.7-1-gbd43759 (bd43759f6e76fa827e2534fa4e61547779ee10a5) $ ceph --version ceph version 0.80.7-1-gbd43759 (bd43759f6e76fa827e2534fa4e61547779ee10a5) $ sudo yum info ceph-radosgw Installed Packages Name: ceph-radosgw Arch: x86_64 Version : 0.80.7 Release : 1.gbd43759.el6 Size: 3.8 M Repo: installed From repo : ceph-source Summary : Rados REST gateway URL : http://ceph.com/ License : GPL-2.0 Description : radosgw is an S3 HTTP REST gateway for the RADOS object store. It is : implemented as a FastCGI module using libfcgi, and can be used in : conjunction with any FastCGI capable web server. Unfortunately the problem on the multipart upload with aws-sdk still remains the same! Here is a part of the apache log: PUT /clients-space/test/iip7.dmg?partNumber=3uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=1uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=2uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 Directly modification of the binary so that the 2%2F be changed to 2- results in success and here is the log: PUT /clients-space/test/iip7.dmg?partNumber=1uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=2uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=4uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 POST /clients-space/test/iip7.dmg?uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 302 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 Can you think of something else?? Best regards, George OK! I will give it some time and will try again later! Thanks a lot for your help! Warmest regards, George The branch I pushed earlier was based off recent development branch. I just pushed one based off firefly (wip-10271-firefly). It will probably take a bit to build. Yehuda On Thu, Dec 11, 2014 at 12:03 PM, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote: Hi again! I have installed and enabled the development branch repositories as described here: http://ceph.com/docs/master/install/get-packages/#add-ceph-development and when I try to update the ceph-radosgw package I get the following: Installed Packages Name: ceph-radosgw Arch: x86_64 Version : 0.80.7 Release
Re: [ceph-users] system metrics monitoring
You can also try Sensu.. On Dec 12, 2014 1:05 AM, pragya jain prag_2...@yahoo.co.in wrote: hello sir! According to TomiTakussaari/riak_zabbix https://github.com/TomiTakussaari/riak_zabbix Currently supported Zabbix keys: riak.ring_num_partitions riak.memory_total riak.memory_processes_used riak.pbc_active riak.pbc_connects riak.node_gets riak.node_puts riak.node_get_fsm_time_median riak.node_put_fsm_time_median All these metrics are monitored by collectd, OpenTSDB and Ganglia also. I need some monitoring tool that monitor metrics,like, Available Disk SpaceIOWaitRead OperationsWrite OperationsNetwork ThroughputLoad AverageDoes Zabbix provide monitoring of these metrics? Thanks Regards Pragya jain On Friday, 12 December 2014 11:05 AM, Irek Fasikhov malm...@gmail.com wrote: Hi. We use Zabbix. 2014-12-12 8:33 GMT+03:00 pragya jain prag_2...@yahoo.co.in: hello sir! I need some open source monitoring tool for examining these metrics. Please suggest some open source monitoring software. Thanks Regards Pragya Jain On Thursday, 11 December 2014 9:16 PM, Denish Patel den...@omniti.com wrote: Try http://www.circonus.com On Thu, Dec 11, 2014 at 1:22 AM, pragya jain prag_2...@yahoo.co.in wrote: please somebody reply my query. Regards Pragya Jain On Tuesday, 9 December 2014 11:53 AM, pragya jain prag_2...@yahoo.co.in wrote: hello all! As mentioned at statistics and monitoring page of Riak Systems Metrics To Graph http://docs.basho.com/riak/latest/ops/running/stats-and-monitoring/#Systems-Metrics-To-Graph MetricAvailable Disk SpaceIOWaitRead OperationsWrite OperationsNetwork ThroughputLoad Average Can somebody suggest me some monitoring tools that monitor these metrics? Regards Pragya Jain ___ riak-users mailing list riak-us...@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com -- Denish Patel, OmniTI Computer Consulting Inc. Database Architect, http://omniti.com/does/data-management http://www.pateldenish.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- С уважением, Фасихов Ирек Нургаязович Моб.: +79229045757 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Block device and Trim/Discard
On 12.12.2014 12:48, Max Power wrote: It would be great to shrink the used space. Is there a way to achieve this? Or have I done something wrong? In a professional environment you may can live with filesystems that only grow. But on my small home-cluster this really is a problem. As Wido already mentioned the kernel RBD does not support discard. When using qemu+rbd you cannot use the virto driver as this also does not support discard. My best experience is with the virtual SATA driver and the options cache=writeback and discard=on. Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Block device and Trim/Discard
On 12/12/2014 01:17 PM, Max Power wrote: Wido den Hollander w...@42on.com hat am 12. Dezember 2014 um 12:53 geschrieben: It depends. Kernel RBD does not support discard/trim yet. Qemu does under certain situations and with special configuration. Ah, Thank you. So this is my problem. I use rbd with the kernel modules. I think I should port my fileserver to qemu/kvm environment then and hope that it is safe to have a big qemu-partition with around 10 TB. Regarding discard in Kernel RBD: http://tracker.ceph.com/issues/190 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Wido den Hollander Ceph consultant and trainer 42on B.V. Phone: +31 (0)20 700 9902 Skype: contact42on ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] AWS SDK and MultiPart Problem
How silly of me!!! I 've just noticed that the file isn't writable by the apache! I 'll be back with the logs... G. I 'd be more than happy to provide to you all the info but for some unknown reason my radosgw.log is empty. This is the part that I have in ceph.conf [client.radosgw.gateway] host = xxx keyring = /etc/ceph/keyring.radosgw.gateway rgw socket path = /tmp/radosgw.sock rgw dns name = xxx.example.com rgw enable usage log = true rgw usage log tick interval = 30 rgw usage log flush threshold = 1024 rgw usage max shards = 32 rgw usage max user shards = 1 log file = /var/log/ceph/radosgw.log debug ms = 1 debug rgw = 20 but no matter what I put in there the log is empty $ pwd /var/log/ceph $ ls -l radosgw.log -rw-r--r-- 1 root root 0 Nov 30 03:01 radosgw.log I have already started another thread with title Empty Rados log here in ceph-users list since December 4th but haven't heard from anyone yet... If I solve this I will be able to provide you with all the data. Regards, George Ok, I've been digging a bit more. I don't have full radosgw logs for the issue, so if you could provide it (debug rgw = 20), it might help. However, as it is now, I think the issue is with the way the client library is signing the requests. Instead of using the undecoded uploadId, it uses the encoded version for the signature, which doesn't sign correctly. The same would have happened if it would have run against amazon S3 (just tested it). The two solutions that I see are to fix the client library, and/or to modify the character to one that does not require escaping. Sadly the dash character that you were using cannot be used safely in that context. Maybe tilde ('~') would could work. Yehuda On Fri, Dec 12, 2014 at 2:41 AM, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote: Dear Yehuda, I have installed the patched version as you can see: $ radosgw --version ceph version 0.80.7-1-gbd43759 (bd43759f6e76fa827e2534fa4e61547779ee10a5) $ ceph --version ceph version 0.80.7-1-gbd43759 (bd43759f6e76fa827e2534fa4e61547779ee10a5) $ sudo yum info ceph-radosgw Installed Packages Name: ceph-radosgw Arch: x86_64 Version : 0.80.7 Release : 1.gbd43759.el6 Size: 3.8 M Repo: installed From repo : ceph-source Summary : Rados REST gateway URL : http://ceph.com/ License : GPL-2.0 Description : radosgw is an S3 HTTP REST gateway for the RADOS object store. It is : implemented as a FastCGI module using libfcgi, and can be used in : conjunction with any FastCGI capable web server. Unfortunately the problem on the multipart upload with aws-sdk still remains the same! Here is a part of the apache log: PUT /clients-space/test/iip7.dmg?partNumber=3uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=1uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=2uploadId=2%2F9rEUmdFcuW66VJfeH3_jbqqUz0jKvrO HTTP/1.1 403 78 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 Directly modification of the binary so that the 2%2F be changed to 2- results in success and here is the log: PUT /clients-space/test/iip7.dmg?partNumber=1uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=2uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 PUT /clients-space/test/iip7.dmg?partNumber=4uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 - - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 POST /clients-space/test/iip7.dmg?uploadId=2-R6bxv4TM2Brxn-w9aHOcbb8OSJ3-Vh2 HTTP/1.1 200 302 - aws-sdk-nodejs/2.1.0 darwin/v0.10.33 Can you think of something else?? Best regards, George OK! I will give it some time and will try again later! Thanks a lot for your help! Warmest regards, George The branch I pushed earlier was based off recent development branch. I just pushed one based off firefly (wip-10271-firefly). It will probably take a bit to build. Yehuda On Thu, Dec 11, 2014 at 12:03 PM, Georgios Dimitrakakis gior...@acmac.uoc.gr wrote: Hi again! I have installed and enabled the development branch repositories as described here: http://ceph.com/docs/master/install/get-packages/#add-ceph-development and when I try to update the ceph-radosgw package I get the following: Installed Packages Name: ceph-radosgw Arch: x86_64 Version : 0.80.7 Release : 0.el6 Size: 3.8 M Repo: installed From repo : Ceph Summary : Rados REST gateway URL : http://ceph.com/ License : GPL-2.0 Description : radosgw is an S3 HTTP REST gateway for the RADOS object store. It is : implemented as a FastCGI module using libfcgi, and can be used in :