Re: [ceph-users] Fwd: Multi-site deployment RBD and Federated Gateways
Hello Logan and All - I am interested in remote replication between two ceph clusters not using federated radosgw setup. Something like ceph osd from one to ceph osd of another cluster. Any thoughts on how to accomplish this? Thanks,Lakshmi. On Wednesday, January 7, 2015 5:21 PM, Logan Barfield wrote: Hello, I'm re-sending this message since I didn't see it picked up on the list archives yesterday. My apologies if it was received previously. We are currently running a single datacenter Ceph deployment. Our setup is as follows:- 4 HDD OSD nodes (primarily used for RadosGW/Object Storage)- 2 SSD OSD nodes (used for RBD/VM block devices)- 3 Monitor daemons running on 3 of the HDD OSD nodes- The CRUSH rules are set to push all data to the HDD nodes except for the RBD pool, which uses the SSD nodes. Our goal is to have OSD nodes in 3 datacenters (US East, US West, Europe). I'm thinking that we would want the following setup:- RadosGW instance in each datacenter with geo-dns to direct clients to the closest one.- Same OSD configuration as our current location (HDD for RadosGW, SSD for RBD)- Separate RBD pool in each datacenter for VM block devices.- CRUSH rules:-> RadosGW: 3 replicas, different OSD nodes, at least 1 off-site (e.g., 2 replicas on 2 OSD nodes in one datacenter, 1 replica on 1 OSD node in a different datacenter). I don't know if RadosGW is geo-aware enough to do this efficiently-> RBD: 2 replicas across 2 OSD nodes in the same datacenter. >From the documentation it looks like the best way to accomplish this would be >to have a separate cluster in each datacenter, then use a federated RadosGW >configuration to keep geo-redundant replicas of objects. The other option >would be to have one cluster spanning all 3 locations, but since they would be >connected over VPN/WAN links that doesn't seem ideal. Concerns:- With a federated configuration it looks like only one zone will be writable, so if the master zone is on the east coast all of the west coast clients would be uploading there as well.- It doesn't appear that there is a way to only have 1 replica sent to the secondary zone, rather all data written to the master is replicated to the secondary (e.g., 3 replicas in each location). Alternatively with multiple regions both zones would be read/write, but only metadata would be synced.- From the documentation I understand that there should be different pools for each zone, and each cluster will need to have a different name. Since our current cluster is in production I don't know how safe it would be to rename/move pools, or re-name the cluster. We are using the default "ceph" cluster name right now because different names add complexity (e.g, requiring '--cluster' for all commands), and we noticed in testing that some of the init scripts don't play well with custom cluster names. It would seem to me that having a federated configuration would add a lot of complexity. It wouldn't get us exactly what we'd like for replication (one offsite copy), and doesn't allow for geo-aware writes. I've seen a few examples of CRUSH maps that span multiple datacenters. This would seem to be an easier setup, and would get us closer to what we want with replication. My only concern would be the WAN latency, setting up site-to-site VPN (which I don't think is necessary for the federated setup), and how well Ceph would handle losing a connection to one of the remote sites for a few seconds or minutes. Is there a recommended deployment for what we want to do, or any reference guides beyond the official Ceph docs? I know Ceph is being used for multi-site deployments, but other than a few blog posts demonstrating theoretical setups and vague Powerpoint slides I haven't seen any details on it. Unfortunately we are a very small company, so consulting with Inktank/RedHat isn't financially feasible right now. Any suggestions/insight would be much appreciated. Thank You, Logan BarfieldTranquil Hosting ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Radosgw with SSL enabled
Hello All - Happy 2015.
I have been successful in establishing communication using --insecure option. I
have two problems here.
1. swift calls without --insecure option continues to fail. Not sure why?
2. ceph gateway logs has the following error logs. Any thoughts on why I am
seeing this error? Please note that I do have converted keystone certs copied
in /var/ceph/nss directory.
2015-01-13 18:19:38.258956 7f54e17fa700 20 sending request to
https://192.0.2.26:5000/v2.0/tokens/revoked
2015-01-13 18:19:38.433790 7f54e17fa700 10 request returned {"signed":
"-BEGIN
CMS-\nMIIBygYJKoZIhvcNAQcCoIIBuzCCAbcCAQExDTALBglghkgBZQMEAgEwHgYJKoZI\nhvcNAQcBoBEED3sicmV2b2tlZCI6IFtdfTGCAYEwggF9AgEBMFgwUzELMAkGA1UE\nBhMCWFgxDjAMBgNVBAgMBVVuc2V0MQ4wDAYDVQQHDAVVbnNldDEOMAwGA1UECgwF\nVW5zZXQxFDASBgNVBAMMC0tleXN0b25lIENBAgECMAsGCWCGSAFlAwQCATANBgkq\nhkiG9w0BAQEFAASCAQCk8YvhUFIWL5HazFO1UexNqTiGJUmYf+nvOBASB/540qaI\n99cTg/vDI1f/bdHz9OpoNpm1ESDYuWswMBu+Z+fcN59B4Da7cK3UMZPjHLrhwYo3\nJolh9ZS1Ia2deue/F4I7nrkBsJmAcVOoAPFcu/72no1nGhAdCHApOaet2WOSUPfo\nIue4zqOYcOBHpq536adyITHiKtq4mhNHDvElZRp8OOJ0twQ+GtwIwckX7KHV2Hqk\nmEAXjGOgLks/pAmh/l8VvTCqtPS/aeLyPQW8MplUeF8a80ypEbYVuwvUWAPRaWf4\nQ4gqk47zO2AqgDUTiDmgpD7g7vmj7BNJtKs8KiNK\n-END
CMS-\n"}2015-01-13 18:19:38.435725 7f54e17fa700 10 signed=-BEGIN
CMS-MIIBygYJKoZIhvcNAQcCoIIBuzCCAbcCAQExDTALBglghkgBZQMEAgEwHgYJKoZIhvcNAQcBoBEED3sicmV2b2tlZCI6IFtdfTGCAYEwggF9AgEBMFgwUzELMAkGA1UEBhMCWFgxDjAMBgNVBAgMBVVuc2V0MQ4wDAYDVQQHDAVVbnNldDEOMAwGA1UECgwFVW5zZXQxFDASBgNVBAMMC0tleXN0b25lIENBAgECMAsGCWCGSAFlAwQCATANBgkqhkiG9w0BAQEFAASCAQCk8YvhUFIWL5HazFO1UexNqTiGJUmYf+nvOBASB/540qaI99cTg/vDI1f/bdHz9OpoNpm1ESDYuWswMBu+Z+fcN59B4Da7cK3UMZPjHLrhwYo3Jolh9ZS1Ia2deue/F4I7nrkBsJmAcVOoAPFcu/72no1nGhAdCHApOaet2WOSUPfoIue4zqOYcOBHpq536adyITHiKtq4mhNHDvElZRp8OOJ0twQ+GtwIwckX7KHV2HqkmEAXjGOgLks/pAmh/l8VvTCqtPS/aeLyPQW8MplUeF8a80ypEbYVuwvUWAPRaWf4Q4gqk47zO2AqgDUTiDmgpD7g7vmj7BNJtKs8KiNK-END
CMS-
2015-01-13 18:19:38.438107 7f54e17fa700 10
content=MIIBygYJKoZIhvcNAQcCoIIBuzCCAbcCAQExDTALBglghkgBZQMEAgEwHgYJKoZIhvcNAQcBoBEED3sicmV2b2tlZCI6IFtdfTGCAYEwggF9AgEBMFgwUzELMAkGA1UEBhMCWFgxDjAMBgNVBAgMBVVuc2V0MQ4wDAYDVQQHDAVVbnNldDEOMAwGA1UECgwFVW5zZXQxFDASBgNVBAMMC0tleXN0b25lIENBAgECMAsGCWCGSAFlAwQCATANBgkqhkiG9w0BAQEFAASCAQCk8YvhUFIWL5HazFO1UexNqTiGJUmYf+nvOBASB/540qaI99cTg/vDI1f/bdHz9OpoNpm1ESDYuWswMBu+Z+fcN59B4Da7cK3UMZPjHLrhwYo3Jolh9ZS1Ia2deue/F4I7nrkBsJmAcVOoAPFcu/72no1nGhAdCHApOaet2WOSUPfoIue4zqOYcOBHpq536adyITHiKtq4mhNHDvElZRp8OOJ0twQ+GtwIwckX7KHV2HqkmEAXjGOgLks/pAmh/l8VvTCqtPS/aeLyPQW8MplUeF8a80ypEbYVuwvUWAPRaWf4Q4gqk47zO2AqgDUTiDmgpD7g7vmj7BNJtKs8KiNK2015-01-13
18:19:38.439062 7f54e17fa700 0 ERROR: signer 0 status =
SigningCertNotFound2015-01-13 18:19:38.439492 7f54e17fa700 0 ERROR: problem
decoding2015-01-13 18:19:38.439548 7f54e17fa700 0 ceph_decode_cms returned
-222015-01-13 18:19:38.439608 7f54e17fa700 0 ERROR: keystone revocation
processing returned error r=-22
On Friday, December 5, 2014 10:41 AM, lakshmi k s wrote:
Hello -
I have rados gateway setup working with http. But when I enable SSL on gateway
node, I am having trouble making successful swift requests over https.
root@hrados:~# swift -V 1.0 -A https://hrados1.ex.com/auth/v1.0 -U
s3User:swiftUser -K 8fJfd6YW2poqhvBI+uUYJZE1uscnmrDncRXrkjHR list[Errno bad
handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify
failed')]
Output of CURL command is as follows. root@hrados:~# curl --insecure -X GET -i
-H "X-Auth-Key:8fJfd6YW2poqhvBI+uUYJZE1uscnmrDncRXrkjHR" -H
"X-Auth-User:s3User:swiftUser" https://hrados1.ex.com/auth/v1.0HTTP/1.1 204 No
ContentDate: Fri, 05 Dec 2014 17:53:58 GMTServer: Apache/2.4.10
(Debian)X-Storage-Url: https://hrados1.ex.com/swift/v1X-Storage-Token:
AUTH_rgwtk10007333557365723a737769667455736572961633914ab868f0b6428354483a6b08fc254e33b1283ed9f428c61436aa05c0f44069d8X-Auth-Token:
AUTH_rgwtk10007333557365723a737769667455736572961633914ab868f0b6428354483a6b08fc254e33b1283ed9f428c61436aa05c0f44069d8Content-Type:
application/json
Appreciate your help.Thanks,Lakshmi.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Radosgw-Agent
Thanks Yehuda. But the link seems to be pointing to Debian binaries. Can you please point me to source packages? Regards,Lakshmi. On Monday, December 15, 2014 8:16 AM, Yehuda Sadeh wrote: There's the 'radosgw-agent' package for debian, e.g., here: http://ceph.com/debian-giant/pool/main/r/radosgw-agent/radosgw-agent_1.2-1~bpo70+1_all.deb On Mon, Dec 15, 2014 at 5:12 AM, lakshmi k s wrote: > Hello - > > Can anyone help me locate the Debian-type source packages for radosgw-agent? > > Thanks, > Lakshmi. > > > On Monday, December 8, 2014 6:10 AM, lakshmi k s wrote: > > > Hello Sage - > > Just wondering if you are the module owner for radosgw-agent? If so, can you > please help me to locate the latest source bits for debian wheezy? > > Thanks, > Lakshmi. > > > On Wednesday, December 3, 2014 8:42 PM, lakshmi k s > wrote: > > > Hello - Please help me here. Where I can locate the source package? > > > On Tuesday, December 2, 2014 12:41 PM, lakshmi k s wrote: > > > Hello: > > I am trying to locate the source package used for Debian Wheezy for the > radosgw-agent 1.2-1-bpo70+1 that is available from the ceph repository. > > Our company requires us to verify package builds from source and to check > licenses from those same source packages. However I have not been able to > locate the source package for the 1.2-1~bpo70+1 version that is available as > a pre-built package for debian wheezy from the current ceph software > repository. > > Can anyone tell me where the repo is that I can put into my sources.list so > I can pull this down to do our required verification steps? > > Thank you. > Lakshmi. > > > > > > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Radosgw-Agent
Hello - Can anyone help me locate the Debian-type source packages for radosgw-agent? Thanks,Lakshmi. On Monday, December 8, 2014 6:10 AM, lakshmi k s wrote: Hello Sage - Just wondering if you are the module owner for radosgw-agent? If so, can you please help me to locate the latest source bits for debian wheezy? Thanks,Lakshmi. On Wednesday, December 3, 2014 8:42 PM, lakshmi k s wrote: Hello - Please help me here. Where I can locate the source package? On Tuesday, December 2, 2014 12:41 PM, lakshmi k s wrote: Hello: I am trying to locate the source package used for DebianWheezy for the radosgw-agent 1.2-1-bpo70+1 that is available from the cephrepository. Our company requires us to verify package builds fromsource and to check licenses from those same source packages. However I have notbeen able to locate the source package for the 1.2-1~bpo70+1 version that isavailable as a pre-built package for debian wheezy from the current cephsoftware repository. Can anyone tell me where the repo is that I can put intomy sources.list so I can pull this down to do our required verification steps? Thank you.Lakshmi. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Radosgw-Agent
Hello Sage - Just wondering if you are the module owner for radosgw-agent? If so, can you please help me to locate the latest source bits for debian wheezy? Thanks,Lakshmi. On Wednesday, December 3, 2014 8:42 PM, lakshmi k s wrote: Hello - Please help me here. Where I can locate the source package? On Tuesday, December 2, 2014 12:41 PM, lakshmi k s wrote: Hello: I am trying to locate the source package used for DebianWheezy for the radosgw-agent 1.2-1-bpo70+1 that is available from the cephrepository. Our company requires us to verify package builds fromsource and to check licenses from those same source packages. However I have notbeen able to locate the source package for the 1.2-1~bpo70+1 version that isavailable as a pre-built package for debian wheezy from the current cephsoftware repository. Can anyone tell me where the repo is that I can put intomy sources.list so I can pull this down to do our required verification steps? Thank you.Lakshmi. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Radosgw with SSL enabled
Hello -
I have rados gateway setup working with http. But when I enable SSL on gateway
node, I am having trouble making successful swift requests over https.
root@hrados:~# swift -V 1.0 -A https://hrados1.ex.com/auth/v1.0 -U
s3User:swiftUser -K 8fJfd6YW2poqhvBI+uUYJZE1uscnmrDncRXrkjHR list[Errno bad
handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify
failed')]
Output of CURL command is as follows. root@hrados:~# curl --insecure -X GET -i
-H "X-Auth-Key:8fJfd6YW2poqhvBI+uUYJZE1uscnmrDncRXrkjHR" -H
"X-Auth-User:s3User:swiftUser" https://hrados1.ex.com/auth/v1.0HTTP/1.1 204 No
ContentDate: Fri, 05 Dec 2014 17:53:58 GMTServer: Apache/2.4.10
(Debian)X-Storage-Url: https://hrados1.ex.com/swift/v1X-Storage-Token:
AUTH_rgwtk10007333557365723a737769667455736572961633914ab868f0b6428354483a6b08fc254e33b1283ed9f428c61436aa05c0f44069d8X-Auth-Token:
AUTH_rgwtk10007333557365723a737769667455736572961633914ab868f0b6428354483a6b08fc254e33b1283ed9f428c61436aa05c0f44069d8Content-Type:
application/json
Appreciate your help.Thanks,Lakshmi.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Radosgw-Agent
Hello - Please help me here. Where I can locate the source package? On Tuesday, December 2, 2014 12:41 PM, lakshmi k s wrote: Hello: I am trying to locate the source package used for DebianWheezy for the radosgw-agent 1.2-1-bpo70+1 that is available from the cephrepository. Our company requires us to verify package builds fromsource and to check licenses from those same source packages. However I have notbeen able to locate the source package for the 1.2-1~bpo70+1 version that isavailable as a pre-built package for debian wheezy from the current cephsoftware repository. Can anyone tell me where the repo is that I can put intomy sources.list so I can pull this down to do our required verification steps? Thank you.Lakshmi. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Radosgw-Agent
Hello: I am trying to locate the source package used for DebianWheezy for the radosgw-agent 1.2-1-bpo70+1 that is available from the cephrepository. Our company requires us to verify package builds fromsource and to check licenses from those same source packages. However I have notbeen able to locate the source package for the 1.2-1~bpo70+1 version that isavailable as a pre-built package for debian wheezy from the current cephsoftware repository. Can anyone tell me where the repo is that I can put intomy sources.list so I can pull this down to do our required verification steps? Thank you. Lakshmi. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Cluster with two radosgw
Yehuda - thanks much. I do have unique users for two rados gateway nodes and also defined them accordingly in ceph configuration file. From Openstack controller node, I can talk to both the nodes. Any thoughts on how to incorporate HA in controller node and test the fail-over? On Friday, November 7, 2014 9:45 AM, Yehuda Sadeh wrote: On Wed, Nov 5, 2014 at 2:08 PM, lakshmi k s wrote: > Hello - > > My ceph cluster needs to have two rados gateway nodes eventually interfacing > with Openstack haproxy. I have been successful in bringing up one of them. > What are the steps for additional rados gateway node to be included in > cluster? Any help is greatly appreciated. > > Thanks much. > Lakshmi. There isn't much difference in bringing up a second gateway. The only thing you may need is to use a different ceph user name as the radosgw user, but everything else should work seamlessly. Yehuda.___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Cluster with two radosgw
Any best practices available for Radosgw HA? Please suggest. On Wednesday, November 5, 2014 2:08 PM, lakshmi k s wrote: Hello - My ceph cluster needs to have two rados gateway nodes eventually interfacing with Openstack haproxy. I have been successful in bringing up one of them. What are the steps for additional rados gateway node to be included in cluster? Any help is greatly appreciated. Thanks much. Lakshmi.___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph Cluster with two radosgw
Hello - My ceph cluster needs to have two rados gateway nodes eventually interfacing with Openstack haproxy. I have been successful in bringing up one of them. What are the steps for additional rados gateway node to be included in cluster? Any help is greatly appreciated. Thanks much. Lakshmi.___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD (and probably other settings) not being picked up outside of the [global] section
Hello Christian - On a side note, I am facing similar issues with Keystone flags on 0.80.5/0.80.6. If they are declared under radosgw section, they are not picked up. But if they are under global section, OpenStack keystone works like a charm. I would really like to see a solution for this. Thanks, Lakshmi. On Thursday, October 16, 2014 6:54 PM, Christian Balzer wrote: Hello, Consider this rather basic configuration file: --- [global] fsid = e6687ef7-54e1-44bd-8072-f9ecab00815 mon_initial_members = ceph-01, comp-01, comp-02 mon_host = 10.0.0.21,10.0.0.5,10.0.0.6 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true mon_osd_downout_subtree_limit = host public_network = 10.0.0.0/8 osd_pool_default_pg_num = 2048 osd_pool_default_pgp_num = 2048 osd_crush_chooseleaf_type = 1 [osd] osd_mkfs_type = ext4 osd_mkfs_options_ext4 = -J size=1024 -E lazy_itable_init=0,lazy_journal_init=0 osd_op_threads = 10 osd_scrub_load_threshold = 2.5 filestore_max_sync_interval = 10 --- Let us slide the annoying fact that ceph ignores the pg and pgp settings when creating the initial pools. And that monitors are preferred based on IP address instead of the sequence they're listed in the config file. Interestingly ceph-deploy correctly picks up the mkfs_options but why it fails to choose the mkfs_type as default is beyond me. The real issue is that the other three OSD setting are NOT picked up by ceph on startup. But they sure are when moved to the global section. Anybody else seeing this (both with 0.80.1 and 0.80.6)? Regards, Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Fusion Communications http://www.gol.com/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Radosgw refusing to even attempt to use keystone auth
Sure Mark, I saw that thread last night. It will be interesting to see the resolution. Thanks, Lakshmi. On Friday, October 17, 2014 12:21 AM, Mark Kirkwood wrote: Keep an eye on the new thread "OSD (and probably other settings) not being picked up outside of the [global] section". You may be running into something similar. Regards Mark On 17/10/14 11:52, lakshmi k s wrote: > Thank you Mark. Strangely, Icehouse install that I have didn't seem to > have one. At least not in /etc/apache2/ sub-directories. Like I said > earlier, I can make the keystone openstack integration work seamlessly > if I move all the keystone related flags under global section. Not > otherwise. I am still looking into this. Appreciate all your help. > > Thanks, > Lakshmi. > > > > > On Thursday, October 16, 2014 3:17 PM, Mark Kirkwood > wrote: > > > Hi, > > While I certainly can (attached) - if your install has keystone running > it *must* have one. It will be hiding somewhere! > > Cheers > > Mark > > On 17/10/14 05:12, lakshmi k s wrote: > > Hello Mark - > > > > Can you please paste your keystone.conf? Also It seems that Icehouse > install that I have does not have keystone.conf. Do we need to create > one? Like I said, adding WSGIChunkedRequest On in Keystone.conf did not > solve my issue. > > > > >___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Radosgw refusing to even attempt to use keystone auth
Thank you Mark. Strangely, Icehouse install that I have didn't seem to have one. At least not in /etc/apache2/ sub-directories. Like I said earlier, I can make the keystone openstack integration work seamlessly if I move all the keystone related flags under global section. Not otherwise. I am still looking into this. Appreciate all your help. Thanks, Lakshmi. On Thursday, October 16, 2014 3:17 PM, Mark Kirkwood wrote: Hi, While I certainly can (attached) - if your install has keystone running it *must* have one. It will be hiding somewhere! Cheers Mark On 17/10/14 05:12, lakshmi k s wrote: > Hello Mark - > > Can you please paste your keystone.conf? Also It seems that Icehouse install > that I have does not have keystone.conf. Do we need to create one? Like I > said, adding WSGIChunkedRequest On in Keystone.conf did not solve my issue. >___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Radosgw refusing to even attempt to use keystone auth
Hello Mark - Can you please paste your keystone.conf? Also It seems that Icehouse install that I have does not have keystone.conf. Do we need to create one? Like I said, adding WSGIChunkedRequest On in Keystone.conf did not solve my issue. Thanks, Lakshmi. On Wednesday, October 15, 2014 10:17 PM, lakshmi k s wrote: Has anyone seen this issue? Appreciate your time. On Wednesday, October 15, 2014 4:50 PM, lakshmi k s wrote: I still think that there is problem with the way radosgw is setup. Two things I want to point out - 1. rgw keystone url - If this flag is under radosgw section of ceph.conf file, I do not see the packets being exchanged between keystone and gateway node when radosgw is restarted. I tried to run tcpdump on both the nodes. 2. rgw.keystone url - If this is in global section (which is wrong), I do see the packets being exchanged between the nodes when radosgw is restarted. I have tried my best to follow the instructions as per http://ceph.com/docs/master/radosgw/config/ to setup radosgw. Also with this setup, I can still create users using radosgw-admin and make swift v1.0 calls from swift-client. How should I go about resolving this issue? Please help. Thanks, Lakshmi. On Wednesday, October 15, 2014 2:58 PM, Mark Kirkwood wrote: On 16/10/14 10:37, Mark Kirkwood wrote: > On 16/10/14 09:08, lakshmi k s wrote: >> I am trying to integrate Openstack keystone with radosgw. I have >> followed the instructions as per the link - >> http://ceph.com/docs/master/radosgw/keystone/. But for some reason, >> keystone flags under [client.radosgw.gateway] section are not being >> honored. That means, presence of these flags never attempt to use >> keystone. Hence, any swift v2.0 calls results in 401-Authorization >> problem. But If I move the keystone url outside under global section, I >> see that there is initial keystone handshake between keystone and >> gateway nodes. >> >> Please note that swift v1 calls (without using keystone) work great. >> Any thoughts on how to resolve this problem? >> >> ceph.conf >> >> [global] >> fsid = f216cbe1-fa49-42ed-b28a-322aa3d48fff >> mon_initial_members = node1 >> mon_host = 192.168.122.182 >> auth_cluster_required = cephx >> auth_service_required = cephx >> auth_client_required = cephx >> filestore_xattr_use_omap = true >> >> [client.admin] >> keyring = /etc/ceph/ceph.client.admin.keyring >> >> [client.radosgw.gateway] >> host = radosgw >> keyring = /etc/ceph/ceph.client.radosgw.keyring >> rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock >> log file = /var/log/ceph/client.radosgw.gateway.log >> rgw dns name = radosgw >> >> rgw keystone url = http://192.168.122.165:5000 >> rgw keystone admin token = faedf7bc53e3371924e7b3ddb9d13ddd >> rgw keystone accepted roles = admin Member _member_ >> rgw keystone token cache size = 500 >> rgw keystone revocation interval = 500 >> rgw s3 auth use keystone = true >> nss db path = /var/ceph/nss >> >> > > I have managed to to reproduce this: > > If I copy your [client.radosgw.gateway] section and amend the obvious > differences (hostnames and ips, and socket paths), then I too see auth > failed and no sign of any attempt to use keystone auth logged. Making > the following change: > > - rgw keystone url = http://192.168.122.165:5000 > + rgw keystone url = http://192.168.122.165:35357 > > makes it work again. I'm guessing it is tied up with with the fact we > needed to add WSGI Chunked encoding... and we did that only for the > 35357 keystone virtualhost (I guess I can add it to 5000 too and see if > that fixes it). I does seem odd that there is no log entry on the rgw... > but it may be failing before the call gets logged (will look). > > So amending the keystone site config: ... WSGIChunkedRequest On ... makes the original keystone url with port 5000 work too. The logging business is a bit more tricky - I'd copied your [client.radosgw.gateway] section which lacks debug rgw = 20 line, which explains *my* lack of seeing the keystone auth log lines. When I add that line I'm seeing the debug auth info (even if I remove the WSGI chunking for 5000 and make it fail again). So Lakshmi, can you add the 'WSGIChunkedRequest On' as inidicated, and make sure you have the debug line in there and retest? Regards Mark___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Radosgw refusing to even attempt to use keystone auth
Has anyone seen this issue? Appreciate your time. On Wednesday, October 15, 2014 4:50 PM, lakshmi k s wrote: I still think that there is problem with the way radosgw is setup. Two things I want to point out - 1. rgw keystone url - If this flag is under radosgw section of ceph.conf file, I do not see the packets being exchanged between keystone and gateway node when radosgw is restarted. I tried to run tcpdump on both the nodes. 2. rgw.keystone url - If this is in global section (which is wrong), I do see the packets being exchanged between the nodes when radosgw is restarted. I have tried my best to follow the instructions as per http://ceph.com/docs/master/radosgw/config/ to setup radosgw. Also with this setup, I can still create users using radosgw-admin and make swift v1.0 calls from swift-client. How should I go about resolving this issue? Please help. Thanks, Lakshmi. On Wednesday, October 15, 2014 2:58 PM, Mark Kirkwood wrote: On 16/10/14 10:37, Mark Kirkwood wrote: > On 16/10/14 09:08, lakshmi k s wrote: >> I am trying to integrate Openstack keystone with radosgw. I have >> followed the instructions as per the link - >> http://ceph.com/docs/master/radosgw/keystone/. But for some reason, >> keystone flags under [client.radosgw.gateway] section are not being >> honored. That means, presence of these flags never attempt to use >> keystone. Hence, any swift v2.0 calls results in 401-Authorization >> problem. But If I move the keystone url outside under global section, I >> see that there is initial keystone handshake between keystone and >> gateway nodes. >> >> Please note that swift v1 calls (without using keystone) work great. >> Any thoughts on how to resolve this problem? >> >> ceph.conf >> >> [global] >> fsid = f216cbe1-fa49-42ed-b28a-322aa3d48fff >> mon_initial_members = node1 >> mon_host = 192.168.122.182 >> auth_cluster_required = cephx >> auth_service_required = cephx >> auth_client_required = cephx >> filestore_xattr_use_omap = true >> >> [client.admin] >> keyring = /etc/ceph/ceph.client.admin.keyring >> >> [client.radosgw.gateway] >> host = radosgw >> keyring = /etc/ceph/ceph.client.radosgw.keyring >> rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock >> log file = /var/log/ceph/client.radosgw.gateway.log >> rgw dns name = radosgw >> >> rgw keystone url = http://192.168.122.165:5000 >> rgw keystone admin token = faedf7bc53e3371924e7b3ddb9d13ddd >> rgw keystone accepted roles = admin Member _member_ >> rgw keystone token cache size = 500 >> rgw keystone revocation interval = 500 >> rgw s3 auth use keystone = true >> nss db path = /var/ceph/nss >> >> > > I have managed to to reproduce this: > > If I copy your [client.radosgw.gateway] section and amend the obvious > differences (hostnames and ips, and socket paths), then I too see auth > failed and no sign of any attempt to use keystone auth logged. Making > the following change: > > - rgw keystone url = http://192.168.122.165:5000 > + rgw keystone url = http://192.168.122.165:35357 > > makes it work again. I'm guessing it is tied up with with the fact we > needed to add WSGI Chunked encoding... and we did that only for the > 35357 keystone virtualhost (I guess I can add it to 5000 too and see if > that fixes it). I does seem odd that there is no log entry on the rgw... > but it may be failing before the call gets logged (will look). > > So amending the keystone site config: ... WSGIChunkedRequest On ... makes the original keystone url with port 5000 work too. The logging business is a bit more tricky - I'd copied your [client.radosgw.gateway] section which lacks debug rgw = 20 line, which explains *my* lack of seeing the keystone auth log lines. When I add that line I'm seeing the debug auth info (even if I remove the WSGI chunking for 5000 and make it fail again). So Lakshmi, can you add the 'WSGIChunkedRequest On' as inidicated, and make sure you have the debug line in there and retest? Regards Mark___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Radosgw refusing to even attempt to use keystone auth
I still think that there is problem with the way radosgw is setup. Two things I want to point out - 1. rgw keystone url - If this flag is under radosgw section of ceph.conf file, I do not see the packets being exchanged between keystone and gateway node when radosgw is restarted. I tried to run tcpdump on both the nodes. 2. rgw.keystone url - If this is in global section (which is wrong), I do see the packets being exchanged between the nodes when radosgw is restarted. I have tried my best to follow the instructions as per http://ceph.com/docs/master/radosgw/config/ to setup radosgw. Also with this setup, I can still create users using radosgw-admin and make swift v1.0 calls from swift-client. How should I go about resolving this issue? Please help. Thanks, Lakshmi. On Wednesday, October 15, 2014 2:58 PM, Mark Kirkwood wrote: On 16/10/14 10:37, Mark Kirkwood wrote: > On 16/10/14 09:08, lakshmi k s wrote: >> I am trying to integrate Openstack keystone with radosgw. I have >> followed the instructions as per the link - >> http://ceph.com/docs/master/radosgw/keystone/. But for some reason, >> keystone flags under [client.radosgw.gateway] section are not being >> honored. That means, presence of these flags never attempt to use >> keystone. Hence, any swift v2.0 calls results in 401-Authorization >> problem. But If I move the keystone url outside under global section, I >> see that there is initial keystone handshake between keystone and >> gateway nodes. >> >> Please note that swift v1 calls (without using keystone) work great. >> Any thoughts on how to resolve this problem? >> >> ceph.conf >> >> [global] >> fsid = f216cbe1-fa49-42ed-b28a-322aa3d48fff >> mon_initial_members = node1 >> mon_host = 192.168.122.182 >> auth_cluster_required = cephx >> auth_service_required = cephx >> auth_client_required = cephx >> filestore_xattr_use_omap = true >> >> [client.admin] >> keyring = /etc/ceph/ceph.client.admin.keyring >> >> [client.radosgw.gateway] >> host = radosgw >> keyring = /etc/ceph/ceph.client.radosgw.keyring >> rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock >> log file = /var/log/ceph/client.radosgw.gateway.log >> rgw dns name = radosgw >> >> rgw keystone url = http://192.168.122.165:5000 >> rgw keystone admin token = faedf7bc53e3371924e7b3ddb9d13ddd >> rgw keystone accepted roles = admin Member _member_ >> rgw keystone token cache size = 500 >> rgw keystone revocation interval = 500 >> rgw s3 auth use keystone = true >> nss db path = /var/ceph/nss >> >> > > I have managed to to reproduce this: > > If I copy your [client.radosgw.gateway] section and amend the obvious > differences (hostnames and ips, and socket paths), then I too see auth > failed and no sign of any attempt to use keystone auth logged. Making > the following change: > > - rgw keystone url = http://192.168.122.165:5000 > + rgw keystone url = http://192.168.122.165:35357 > > makes it work again. I'm guessing it is tied up with with the fact we > needed to add WSGI Chunked encoding... and we did that only for the > 35357 keystone virtualhost (I guess I can add it to 5000 too and see if > that fixes it). I does seem odd that there is no log entry on the rgw... > but it may be failing before the call gets logged (will look). > > So amending the keystone site config: ... WSGIChunkedRequest On ... makes the original keystone url with port 5000 work too. The logging business is a bit more tricky - I'd copied your [client.radosgw.gateway] section which lacks debug rgw = 20 line, which explains *my* lack of seeing the keystone auth log lines. When I add that line I'm seeing the debug auth info (even if I remove the WSGI chunking for 5000 and make it fail again). So Lakshmi, can you add the 'WSGIChunkedRequest On' as inidicated, and make sure you have the debug line in there and retest? Regards Mark___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Radosgw refusing to even attempt to use keystone auth
Hello Mark - Changing the rwg keystone url to http://192.168.122.165:35357 did not help. I continue to get 401 error. Also, I am trying to integrate with Icehouse this time. I did not see any keystone.conf in /etc/apache2/sites-available for adding WSGI chunked encoding. That said, I am having issues with initial keystone handshake itself. Thanks, Lakshmi. On Wednesday, October 15, 2014 2:37 PM, Mark Kirkwood wrote: On 16/10/14 09:08, lakshmi k s wrote: > I am trying to integrate Openstack keystone with radosgw. I have > followed the instructions as per the link - > http://ceph.com/docs/master/radosgw/keystone/. But for some reason, > keystone flags under [client.radosgw.gateway] section are not being > honored. That means, presence of these flags never attempt to use > keystone. Hence, any swift v2.0 calls results in 401-Authorization > problem. But If I move the keystone url outside under global section, I > see that there is initial keystone handshake between keystone and > gateway nodes. > > Please note that swift v1 calls (without using keystone) work great. > Any thoughts on how to resolve this problem? > > ceph.conf > > [global] > fsid = f216cbe1-fa49-42ed-b28a-322aa3d48fff > mon_initial_members = node1 > mon_host = 192.168.122.182 > auth_cluster_required = cephx > auth_service_required = cephx > auth_client_required = cephx > filestore_xattr_use_omap = true > > [client.admin] > keyring = /etc/ceph/ceph.client.admin.keyring > > [client.radosgw.gateway] > host = radosgw > keyring = /etc/ceph/ceph.client.radosgw.keyring > rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock > log file = /var/log/ceph/client.radosgw.gateway.log > rgw dns name = radosgw > > rgw keystone url = http://192.168.122.165:5000 > rgw keystone admin token = faedf7bc53e3371924e7b3ddb9d13ddd > rgw keystone accepted roles = admin Member _member_ > rgw keystone token cache size = 500 > rgw keystone revocation interval = 500 > rgw s3 auth use keystone = true > nss db path = /var/ceph/nss > > I have managed to to reproduce this: If I copy your [client.radosgw.gateway] section and amend the obvious differences (hostnames and ips, and socket paths), then I too see auth failed and no sign of any attempt to use keystone auth logged. Making the following change: - rgw keystone url = http://192.168.122.165:5000 + rgw keystone url = http://192.168.122.165:35357 makes it work again. I'm guessing it is tied up with with the fact we needed to add WSGI Chunked encoding... and we did that only for the 35357 keystone virtualhost (I guess I can add it to 5000 too and see if that fixes it). I does seem odd that there is no log entry on the rgw... but it may be failing before the call gets logged (will look). Regards Mark P.s: Added $SUBJECT header.___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] (no subject)
I am trying to integrate Openstack keystone with radosgw. I have followed the instructions as per the link - http://ceph.com/docs/master/radosgw/keystone/. But for some reason, keystone flags under [client.radosgw.gateway] section are not being honored. That means, presence of these flags never attempt to use keystone. Hence, any swift v2.0 calls results in 401-Authorization problem. But If I move the keystone url outside under global section, I see that there is initial keystone handshake between keystone and gateway nodes. Please note that swift v1 calls (without using keystone) work great. Any thoughts on how to resolve this problem? ceph.conf [global] fsid = f216cbe1-fa49-42ed-b28a-322aa3d48fff mon_initial_members = node1 mon_host = 192.168.122.182 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true [client.admin] keyring = /etc/ceph/ceph.client.admin.keyring [client.radosgw.gateway] host = radosgw keyring = /etc/ceph/ceph.client.radosgw.keyring rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock log file = /var/log/ceph/client.radosgw.gateway.log rgw dns name = radosgw rgw keystone url = http://192.168.122.165:5000 rgw keystone admin token = faedf7bc53e3371924e7b3ddb9d13ddd rgw keystone accepted roles = admin Member _member_ rgw keystone token cache size = 500 rgw keystone revocation interval = 500 rgw s3 auth use keystone = true nss db path = /var/ceph/nss Thanks much. Lakshmi.___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Openstack keystone with Radosgw
Hello Mark - I setup a new Ceph cluster like before. But this time it is talking to Icehouse. Same set of problems like before. That is keystone flags are not being honored if they are under [client.radosgw.gateway]. It seems like the issue is with my radosgw setup. Let me create a new thread for this new issue. Thanks much for all your help so far. Regards, Lakshmi. On Wednesday, October 15, 2014 6:54 AM, lakshmi k s wrote: Thanks Mark for looking into this further. As I mentioned earlier, I have following nodes in my ceph cluster - 1 admin node 3 OSD (One of them is a monitor too) 1 gateway node This should have worked technically. But I am not sure where I am going wrong. I will continue to look into this and keep you all posted. Thanks, Lakshmi. On Wednesday, October 15, 2014 2:00 AM, Mark Kirkwood wrote: Because this is an interesting problem, I added an additional host to my 4 node ceph setup that is a purely radosgw host. So I have - ceph1 (mon + osd) - ceph2-4 (osd) - ceph5 (radosgw) My ceph.conf on ceph5 included below. Obviously I changed my keystone endpoints to use this host (ceph5). After that I am unable to reproduce your problem - for a moment I thought I had, but it was just that I had forgotten to include the keystone config in there at all! So it is now working fine. My guess is that there is something subtle broken in your config that we have yet to see... (ceph5) $ cat /etc/ceph/ceph.conf [global] fsid = 2ea9a745-d84c-4fc5-95b4-2f6afa98ece1 mon_initial_members = ceph1 mon_host = 192.168.122.21 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true osd_pool_default_size = 2 osd_pg_bits = 7 osd_pgp_bits = 7 osd_journal_size = 2048 [client.radosgw.gateway] host = ceph5 keyring = /etc/ceph/ceph.rados.gateway.keyring rgw_socket_path = /var/run/ceph/$name.sock log_file = /var/log/ceph/radosgw.log rgw_data = /var/lib/ceph/radosgw/$cluster-$id rgw_dns_name = ceph5 rgw print continue = false debug rgw = 20 rgw keystone url = http://stack1:35357 rgw keystone admin token = tokentoken rgw keystone accepted roles = admin Member _member_ rgw keystone token cache size = 500 rgw keystone revocation interval = 500 rgw s3 auth use keystone = true nss db path = /var/ceph/nss/ On 15/10/14 10:25, Mark Kirkwood wrote: > Right, > > So you have 3 osds, one of whom is a mon. Your rgw is on another host > (called gateway it seems). I'm wondering if is this the issue. In my > case I'm using one of my osds as a rgw as well. This *should* not > matter... but it might be worth trying out a rgw on one of your osds > instead. I'm thinking that your gateway host is setup in some way that > is confusing the [client.radosgw.gatway] entry in ceph.conf (e.g > hostname resolution). > > Regards > > Mark > > On 15/10/14 05:40, lakshmi k s wrote: >> Hello Mark - with rgw_keystone_url under radosgw section, I do NOT see >> keystone handshake. If I move it under global section, I see initial >> keystone handshake as explained earlier. Below is the output of osd dump >> and osd tree. I have 3 nodes (node1, node2, node3) acting as OSDs. One >> of them (node1) is also a monitor node. I also have an admin node and >> gateway node in ceph cluster. Keystone server (swift client) of course >> is all together a different Openstack setup. Let me know if you need any >> more information. >> >___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Openstack keystone with Radosgw
Thanks Mark for looking into this further. As I mentioned earlier, I have following nodes in my ceph cluster - 1 admin node 3 OSD (One of them is a monitor too) 1 gateway node This should have worked technically. But I am not sure where I am going wrong. I will continue to look into this and keep you all posted. Thanks, Lakshmi. On Wednesday, October 15, 2014 2:00 AM, Mark Kirkwood wrote: Because this is an interesting problem, I added an additional host to my 4 node ceph setup that is a purely radosgw host. So I have - ceph1 (mon + osd) - ceph2-4 (osd) - ceph5 (radosgw) My ceph.conf on ceph5 included below. Obviously I changed my keystone endpoints to use this host (ceph5). After that I am unable to reproduce your problem - for a moment I thought I had, but it was just that I had forgotten to include the keystone config in there at all! So it is now working fine. My guess is that there is something subtle broken in your config that we have yet to see... (ceph5) $ cat /etc/ceph/ceph.conf [global] fsid = 2ea9a745-d84c-4fc5-95b4-2f6afa98ece1 mon_initial_members = ceph1 mon_host = 192.168.122.21 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx filestore_xattr_use_omap = true osd_pool_default_size = 2 osd_pg_bits = 7 osd_pgp_bits = 7 osd_journal_size = 2048 [client.radosgw.gateway] host = ceph5 keyring = /etc/ceph/ceph.rados.gateway.keyring rgw_socket_path = /var/run/ceph/$name.sock log_file = /var/log/ceph/radosgw.log rgw_data = /var/lib/ceph/radosgw/$cluster-$id rgw_dns_name = ceph5 rgw print continue = false debug rgw = 20 rgw keystone url = http://stack1:35357 rgw keystone admin token = tokentoken rgw keystone accepted roles = admin Member _member_ rgw keystone token cache size = 500 rgw keystone revocation interval = 500 rgw s3 auth use keystone = true nss db path = /var/ceph/nss/ On 15/10/14 10:25, Mark Kirkwood wrote: > Right, > > So you have 3 osds, one of whom is a mon. Your rgw is on another host > (called gateway it seems). I'm wondering if is this the issue. In my > case I'm using one of my osds as a rgw as well. This *should* not > matter... but it might be worth trying out a rgw on one of your osds > instead. I'm thinking that your gateway host is setup in some way that > is confusing the [client.radosgw.gatway] entry in ceph.conf (e.g > hostname resolution). > > Regards > > Mark > > On 15/10/14 05:40, lakshmi k s wrote: >> Hello Mark - with rgw_keystone_url under radosgw section, I do NOT see >> keystone handshake. If I move it under global section, I see initial >> keystone handshake as explained earlier. Below is the output of osd dump >> and osd tree. I have 3 nodes (node1, node2, node3) acting as OSDs. One >> of them (node1) is also a monitor node. I also have an admin node and >> gateway node in ceph cluster. Keystone server (swift client) of course >> is all together a different Openstack setup. Let me know if you need any >> more information. >> >___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Openstack keystone with Radosgw
Hello Mark - with rgw_keystone_url under radosgw section, I do NOT see keystone handshake. If I move it under global section, I see initial keystone handshake as explained earlier. Below is the output of osd dump and osd tree. I have 3 nodes (node1, node2, node3) acting as OSDs. One of them (node1) is also a monitor node. I also have an admin node and gateway node in ceph cluster. Keystone server (swift client) of course is all together a different Openstack setup. Let me know if you need any more information. ceph-admin@ceph-admin:~/ceph-cluster$ ceph osd dump epoch 34 fsid 199b0c6f-91c1-4ada-907c-4105c6118b40 created 2014-10-13 18:10:28.987081 modified 2014-10-13 18:55:33.028829 flags pool 0 'data' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 flags hashpspool crash_replay_interval 45 stripe_width 0 pool 1 'metadata' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 flags hashpspool stripe_width 0 pool 2 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 flags hashpspool stripe_width 0 pool 3 '.rgw.root' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 15 owner 18446744073709551615 flags hashpspool stripe_width 0 pool 4 '.rgw.control' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 17 owner 18446744073709551615 flags hashpspool stripe_width 0 pool 5 '.rgw' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 19 owner 18446744073709551615 flags hashpspool stripe_width 0 pool 6 '.rgw.gc' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 20 owner 18446744073709551615 flags hashpspool stripe_width 0 pool 7 '.users.uid' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 21 owner 18446744073709551615 flags hashpspool stripe_width 0 pool 8 '.rgw.buckets' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 23 flags hashpspool stripe_width 0 pool 9 '.rgw.buckets.index' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 100 pgp_num 100 last_change 25 flags hashpspool stripe_width 0 pool 10 '.users.swift' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 29 owner 18446744073709551615 flags hashpspool stripe_width 0 pool 11 '.users.email' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 31 owner 18446744073709551615 flags hashpspool stripe_width 0 pool 12 '.users' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 33 owner 18446744073709551615 flags hashpspool stripe_width 0 max_osd 3 osd.0 up in weight 1 up_from 4 up_thru 33 down_at 0 last_clean_interval [0,0) 192.0.2.211:6800/4163 192.0.2.211:6801/4163 192.0.2.211:6802/4163 192.0.2.211:6803/4163 exists,up 74bbdb5d-8f03-4ed5-8d33-33b710a597d1 osd.1 up in weight 1 up_from 7 up_thru 33 down_at 0 last_clean_interval [0,0) 192.0.2.212:6800/3070 192.0.2.212:6801/3070 192.0.2.212:6802/3070 192.0.2.212:6803/3070 exists,up 6ec0bea2-bba2-4d6a-b1a3-c5d7caf1c801 osd.2 up in weight 1 up_from 10 up_thru 33 down_at 0 last_clean_interval [0,0) 192.0.2.213:6800/3070 192.0.2.213:6801/3070 192.0.2.213:6802/3070 192.0.2.213:6803/3070 exists,up bb464cc6-328f-4fb9-86a7-2256c50b97a1 ceph-admin@ceph-admin:~/ceph-cluster$ ceph osd tree # idweight type name up/down reweight -1 0.05997 root default -2 0.01999 host node1 0 0.01999 osd.0 up 1 -3 0.01999 host node2 1 0.01999 osd.1 up 1 -4 0.01999 host node3 2 0.01999 osd.2 up 1 On Monday, October 13, 2014 9:52 PM, Mark Kirkwood wrote: Was that with you moving just rgw_keystone_url into [global]? If so then yeah, that won't work as it will be missing your auth token etc (so will fail to authorize always). You need to chase up why it is not seeing some/all settings in the [client.radosgw.gateway] section. I have a suspicion that you have an unusual ceph topology - so it might be beneficial to show us: $ ceph mon dump $ ceph osd tree and also mention which additional hosts are admins and which host is your radosgw. Cheers Mark On 14/10/14 15:32, lakshmi k s wrote: > I did restart the ceph cluster only to see the ceph health to be NOT OK. > I did the purge operation and re-installed ceph packages on all nodes. > This time, ceph admin node has 0.80.6 and all other cluster nodes > including Openst
Re: [ceph-users] Openstack keystone with Radosgw
I did restart the ceph cluster only to see the ceph health to be NOT OK. I did the purge operation and re-installed ceph packages on all nodes. This time, ceph admin node has 0.80.6 and all other cluster nodes including Openstack client node have 0.80.5 version. Same error logs like before - 2014-10-13 19:21:40.726717 7f88907c8700 1 == starting new request req=0x7f88c003a0e0 = 2014-10-13 19:21:40.726731 7f88907c8700 2 req 2:0.14::HEAD /swift/v1::initializing 2014-10-13 19:21:40.726755 7f88907c8700 10 ver=v1 first= req= 2014-10-13 19:21:40.726757 7f88907c8700 10 s->object= s->bucket= 2014-10-13 19:21:40.726761 7f88907c8700 2 req 2:0.45:swift:HEAD /swift/v1::getting op 2014-10-13 19:21:40.726764 7f88907c8700 2 req 2:0.48:swift:HEAD /swift/v1:stat_account:authorizing 2014-10-13 19:21:40.726768 7f88907c8700 20 token_id=02891ee2909b4f24b999038d93cbc982 2014-10-13 19:21:40.726803 7f88907c8700 20 sending request to http://192.0.2.21:35357/v2.0/tokens 2014-10-13 19:21:55.340373 7f88bbfff700 2 RGWDataChangesLog::ChangesRenewThread: start 2014-10-13 19:22:17.340566 7f88bbfff700 2 RGWDataChangesLog::ChangesRenewThread: start 2014-10-13 19:22:24.786164 7f88937ce700 0 Keystone token parse error: malformed json 2014-10-13 19:22:24.787409 7f88937ce700 10 failed to authorize request 2014-10-13 19:22:24.788450 7f88937ce700 2 req 1:75.099222:swift:HEAD /swift/v1:stat_account:http status=401 2014-10-13 19:22:24.789128 7f88937ce700 1 == req done req=0x7f88c00068e0 http_status=401 == 2014-10-13 19:22:24.789551 7f88937ce700 20 process_request() returned -1 gateway@gateway:~$ ceph auth list installed auth entries: osd.0 key: AQA2eDxU2Hi2BxAADn1H6LVbRuoL1GadYBQo3Q== caps: [mon] allow profile osd caps: [osd] allow * osd.1 key: AQBCeDxUCNw7HBAAmS80TPDupKEpbRMRTmmgdA== caps: [mon] allow profile osd caps: [osd] allow * osd.2 key: AQBMeDxUMBndOBAAnN0Ty2h3MDROlcKMYRYaWQ== caps: [mon] allow profile osd caps: [osd] allow * client.admin key: AQAFeDxUmJnTMRAADEIyXPDkOz8lHsOq9blAdA== caps: [mds] allow caps: [mon] allow * caps: [osd] allow * client.bootstrap-mds key: AQAGeDxUqARlERAAVNwTwY9tOOa0q0asJWy/AA== caps: [mon] allow profile bootstrap-mds client.bootstrap-osd key: AQAGeDxUGCFEBRAAUbV+vyvU5AqN1CHI7wfoDA== caps: [mon] allow profile bootstrap-osd client.radosgw.gateway key: AQCTejxUIHFbHRAApwnvxy4bCIOZ7esn95d5tA== caps: [mon] allow rwx caps: [osd] allow rwx Appreciate your time. Thanks, Lakshmi. On Monday, October 13, 2014 4:43 PM, Mark Kirkwood wrote: That's the same version that I'm using. Did you check the other points I mentioned: - check *all* ceph host are running the same version - restart 'em all to be sure I did think that your 'auth list' output looked strange, but I guessed that you have cut out the osd and mon info before placing it in the message...might be useful to see all of that too. Obviously something is not quite right. On 14/10/14 12:05, lakshmi k s wrote: > I have Ceph 0.85 version. I can still talk to this gateway node like > below using swift v1.0. Note that this user was created using > radosgw-admin.. > > swift -V 1.0 -A http://gateway.ex.com/auth/v1.0 -U s3User:swiftUser -K > CRV8PeotaW204nE9IyutoVTcnr+2Uw8M8DQuRP7i list > my-Test > > I am at total loss now. > > > On Monday, October 13, 2014 3:25 PM, Mark Kirkwood > wrote: > > > Well that certainly looks ok. So entries in [client.radosgw.gateway] > *should* work. If they are not then that points to something else not > setup right on the ceph or radosgw side. > > What version of ceph is this? > > I'd do the following: > - check all ceph hosts have the same ceph version running > - restart all the hosts (ahem - assuming this is not a prod setup) > > If you have not done so before, check the gateway works with all the > keystone stuff disabled (i.e create a swift user using radosgw-admin and > check you can upload a file etc as that user). *Then* enable the > keystone bits...restart the gateway and try again. > > There are a lot of fiddly bits involved in the setup of radosgw - and it > is real easy to to have one missed or not done correctly, which trips > you up later! > > Regards > > Mark > > On 14/10/14 05:06, lakshmi k s wrote: > > > > ceph auth list on gateway node has the following. I think I am using the > > correct name in ceph.conf. > > > > gateway@gateway <mailto:gateway@gateway>:~$ ceph auth list > > installed auth entries: > > client.admin > > key: AQBL3SxUiMplMxAAjrL6oT+0Q5JtdrD90toXqg== > > caps: [mds] allow > > caps: [mon] allow * > > caps: [osd]
Re: [ceph-users] Openstack keystone with Radosgw
I have Ceph 0.85 version. I can still talk to this gateway node like below using swift v1.0. Note that this user was created using radosgw-admin.. swift -V 1.0 -A http://gateway.ex.com/auth/v1.0 -U s3User:swiftUser -K CRV8PeotaW204nE9IyutoVTcnr+2Uw8M8DQuRP7i list my-Test I am at total loss now. On Monday, October 13, 2014 3:25 PM, Mark Kirkwood wrote: Well that certainly looks ok. So entries in [client.radosgw.gateway] *should* work. If they are not then that points to something else not setup right on the ceph or radosgw side. What version of ceph is this? I'd do the following: - check all ceph hosts have the same ceph version running - restart all the hosts (ahem - assuming this is not a prod setup) If you have not done so before, check the gateway works with all the keystone stuff disabled (i.e create a swift user using radosgw-admin and check you can upload a file etc as that user). *Then* enable the keystone bits...restart the gateway and try again. There are a lot of fiddly bits involved in the setup of radosgw - and it is real easy to to have one missed or not done correctly, which trips you up later! Regards Mark On 14/10/14 05:06, lakshmi k s wrote: > > ceph auth list on gateway node has the following. I think I am using the > correct name in ceph.conf. > > gateway@gateway:~$ ceph auth list > installed auth entries: > client.admin > key: AQBL3SxUiMplMxAAjrL6oT+0Q5JtdrD90toXqg== > caps: [mds] allow > caps: [mon] allow * > caps: [osd] allow * > client.radosgw.gateway > key: AQCI5C1UUH7iOhAAWazAeqVLetIDh+CptBtRrQ== > caps: [mon] allow rwx > caps: [osd] allow rwx > > > > > On Sunday, October 12, 2014 8:02 PM, Mark Kirkwood > wrote: > > > Ah, yes. So your gateway is called something other than: > > [client.radosgw.gateway] > > So take a look at what > > $ ceph auth list > > says (run from your rgw), it should pick up the correct name. Then > correct your ceph.conf, restart and see what the rgw log looks like as > you edge ever so closer to having it work :-) > > regards > > Mark > > On 13/10/14 12:27, lakshmi k s wrote: > > Yes Mark, I did restart all the services - radosgw, ceph, apache2. And > > yes, it never attempted to use keystone right from the beginning. > > Interestingly, when I moved the rgw keystone url = > > http://192.0.2.21:5000 > <http://192.0.2.21:5000/><http://192.0.2.21:5000/> under global section in > > ceph.conf file, I see 500 internal error on both the nodes and following > > logs were captured. This looks similar to yours at least during initial > > handshake. > > > > 2014-10-12 16:08:21.015597 7fca80fa9700 1 == starting new request > > req=0x7fcac002ae10 = > > 2014-10-12 16:08:21.015621 7fca80fa9700 2 req 3:0.26::GET > > /swift/v1::initializing > > 2014-10-12 16:08:21.015665 7fca80fa9700 10 ver=v1 first= req= > > 2014-10-12 16:08:21.015669 7fca80fa9700 10 s->object= > s->bucket= > > 2014-10-12 16:08:21.015676 7fca80fa9700 2 req 3:0.81:swift:GET > > /swift/v1::getting op > > 2014-10-12 16:08:21.015682 7fca80fa9700 2 req 3:0.87:swift:GET > > /swift/v1:list_buckets:authorizing > > 2014-10-12 16:08:21.015688 7fca80fa9700 20 > > token_id=7bfb869419044bec8c258e75830d55a2 > > 2014-10-12 16:08:21.015742 7fca80fa9700 20 sending request to > > http://192.0.2.21:5000/v2.0/tokens > > 2014-10-12 16:08:33.001640 7fca9d7e2700 0 Keystone token parse error: > > malformed json > > 2014-10-12 16:08:33.002756 7fca9d7e2700 10 failed to authorize request > > 2014-10-12 16:08:33.003598 7fca9d7e2700 2 req 1:75.081031:swift:GET > > /swift/v1:list_buckets:http status=401 > > 2014-10-12 16:08:33.003863 7fca9d7e2700 1 == req done > > req=0x7fcac0010670 http_status=401 == > > 2014-10-12 16:08:33.004414 7fca9d7e2700 20 process_request() returned -1 > > > >___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Openstack keystone with Radosgw
ceph auth list on gateway node has the following. I think I am using the correct name in ceph.conf. gateway@gateway:~$ ceph auth list installed auth entries: client.admin key: AQBL3SxUiMplMxAAjrL6oT+0Q5JtdrD90toXqg== caps: [mds] allow caps: [mon] allow * caps: [osd] allow * client.radosgw.gateway key: AQCI5C1UUH7iOhAAWazAeqVLetIDh+CptBtRrQ== caps: [mon] allow rwx caps: [osd] allow rwx On Sunday, October 12, 2014 8:02 PM, Mark Kirkwood wrote: Ah, yes. So your gateway is called something other than: [client.radosgw.gateway] So take a look at what $ ceph auth list says (run from your rgw), it should pick up the correct name. Then correct your ceph.conf, restart and see what the rgw log looks like as you edge ever so closer to having it work :-) regards Mark On 13/10/14 12:27, lakshmi k s wrote: > Yes Mark, I did restart all the services - radosgw, ceph, apache2. And > yes, it never attempted to use keystone right from the beginning. > Interestingly, when I moved the rgw keystone url = > http://192.0.2.21:5000 <http://192.0.2.21:5000/> under global section in > ceph.conf file, I see 500 internal error on both the nodes and following > logs were captured. This looks similar to yours at least during initial > handshake. > > 2014-10-12 16:08:21.015597 7fca80fa9700 1 == starting new request > req=0x7fcac002ae10 = > 2014-10-12 16:08:21.015621 7fca80fa9700 2 req 3:0.26::GET > /swift/v1::initializing > 2014-10-12 16:08:21.015665 7fca80fa9700 10 ver=v1 first= req= > 2014-10-12 16:08:21.015669 7fca80fa9700 10 s->object= s->bucket= > 2014-10-12 16:08:21.015676 7fca80fa9700 2 req 3:0.81:swift:GET > /swift/v1::getting op > 2014-10-12 16:08:21.015682 7fca80fa9700 2 req 3:0.87:swift:GET > /swift/v1:list_buckets:authorizing > 2014-10-12 16:08:21.015688 7fca80fa9700 20 > token_id=7bfb869419044bec8c258e75830d55a2 > 2014-10-12 16:08:21.015742 7fca80fa9700 20 sending request to > http://192.0.2.21:5000/v2.0/tokens > 2014-10-12 16:08:33.001640 7fca9d7e2700 0 Keystone token parse error: > malformed json > 2014-10-12 16:08:33.002756 7fca9d7e2700 10 failed to authorize request > 2014-10-12 16:08:33.003598 7fca9d7e2700 2 req 1:75.081031:swift:GET > /swift/v1:list_buckets:http status=401 > 2014-10-12 16:08:33.003863 7fca9d7e2700 1 == req done > req=0x7fcac0010670 http_status=401 == > 2014-10-12 16:08:33.004414 7fca9d7e2700 20 process_request() returned -1 >___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Openstack keystone with Radosgw
Hello Mark - I tried that as well, but in vain. In fact, that is how I created the endpoint to begin with. Since, that didn't work, I followed Openstack standard which was to include %tenant-id. -Lakshmi. On Friday, October 10, 2014 6:49 PM, Mark Kirkwood wrote: Hi, I think your swift endpoint: | 2ccd8523954c4491b08b648cfd42ae6c | regionOne | http://gateway.ex.com/swift/v1/AUTH_%(tenant_id)s | http://gateway.ex.com/swift/v1/AUTH_%(tenant_id)s | http://gateway.ex.com/swift/v1 | 77434bc194a3495793b5b4c943248e16 | is the issue. It should be: | 2ccd8523954c4491b08b648cfd42ae6c | regionOne | http://gateway.ex.com/swift/v1 | http://gateway.ex.com/swift/v1 | http://gateway.ex.com/swift/v1 | 77434bc194a3495793b5b4c943248e16 | i.e no AUTH_%(tenantid)s in there http://ceph.com/docs/master/radosgw/keystone/. Regards Mark On 11/10/14 14:28, lakshmi k s wrote: > With latest HA build, I found keystone_modwsgi.conf in > /etc/apache2/sites-available and added the chunking like below. We have > many controller nodes, but single virtual IP - 192.0.2.21 for which > keystone is configured. I have verified keystone setup by executing > other services like nova list, cinder list, etc. They work fine. It is > swift pointing to ceph object gateway that is not working. > > Listen 192.0.2.24:35357 > Listen 192.0.2.24:5000 > > > WSGIScriptAlias / /etc/keystone/admin > WSGIDaemonProcess keystoneadmin user=keystone group=keystone > processes=2 threads=1 home=/opt/stack/venvs/openstack > python-path=/opt/stack/venvs/openstack:/opt/stack/venvs/openstack/lib/python2.7/site-packages/ > WSGIApplicationGroup keystoneadmin > > WSGIProcessGroup keystoneadmin > > > Options FollowSymLinks > Require all granted > WSGIChunkedRequest On > > > ErrorLog /var/log/keystone/keystone_modwsgi.log > LogLevel info > CustomLog /var/log/keystone/keystone_apache_access.log combined > > > > WSGIScriptAlias / /etc/keystone/main > WSGIDaemonProcess keystonemain user=keystone group=keystone > processes=2 threads=1 home=/opt/stack/venvs/openstack > python-path=/opt/stack/venvs/openstack:/opt/stack/venvs/openstack/lib/python2.7/site-packages/ > WSGIApplicationGroup keystonemain > > WSGIProcessGroup keystonemain > > > Options FollowSymLinks > WSGIChunkedRequest On > Require all granted > > > ErrorLog /var/log/keystone/keystone_modwsgi.log > LogLevel info > CustomLog /var/log/keystone/keystone_apache_access.log combined > > > root@overcloud-ce-controller-controllermgmt0-pc23jdstfxy5:~# keystone > service-list > +--+--+---+---+ > |id| name | type | > description| > +--+--+---+---+ > | 642251f08a93444da1aa457c2a0ae9f3 | cinder | volume| Cinder > Volume Service | > | c909ea43c9244f7c8296e870986c5fc1 | glance | image | > Glance Image Service | > | bf80fcba3aec45a6988262b31b7ae12a | heat | orchestration | > Heat Service | > | 3a1cf21dd3974313ba833e807b3ff997 | keystone |identity | Keystone > Identity Service | > | 8abff3ea4bba41f4b9cc9a77a29191fe | neutron |network| > Neutron Service | > | d87e2f24576a459495f1e08439bae238 | nova |compute|Nova > Compute Service | > | 77434bc194a3495793b5b4c943248e16 | swift | object-store > | | > +--+--+---+---+ > > > root@overcloud-ce-controller-controllermgmt0-pc23jdstfxy5:~# keystone > endpoint-list > +--+---+---+---+-+--+ > |id| region | > publicurl | > internalurl| adminurl > |service_id| > +--+---+---+---+-+--+ > | 09159f243eb6457581e01af56e32bf18 | regionOne | > http://192.0.2.21:8774/v3 | > http://192.0.2.21:8774/v3 | > http://192.0.2.21:8774/v3| 9b431dae0ff642629ae8f5bfd006e578 | > | 0dda582955934dc0af898ec3db2c5fbc | regionOne | > http://192.0.2.21:87
Re: [ceph-users] Openstack keystone with Radosgw
://192.0.2.21:21131/v1
|http://192.0.2.21:21131/v1 |
296f0ce272834f70af9fc6f36924c89a |
| f898c1e25d76471c8a28147e2ddfa52e | regionOne |
http://192.0.2.21:8004/v1/%(tenant_id)s |
http://192.0.2.21:8004/v1/%(tenant_id)s |
http://192.0.2.21:8004/v1/%(tenant_id)s | bf80fcba3aec45a6988262b31b7ae12a |
| fbbc102593394c1fb7da4160dbc28f5b | regionOne |
http://192.0.2.21:9696/ | http://192.0.2.21:9696/
| http://192.0.2.21:9696/ |
8abff3ea4bba41f4b9cc9a77a29191fe |
| fc8e024afc9b43308a5cf8323be76ba5 | regionOne |
http://192.0.2.21:5000/v2.0|http://192.0.2.21:5000/v2.0
| http://192.0.2.21:35357/v2.0 |
3a1cf21dd3974313ba833e807b3ff997 |
+--+---+-
ceph.conf
[global]
fsid = b35e8496-e809-416a-bd66-aba761d78fac
mon_initial_members = node1
mon_host = 192.0.2.211
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
[client.admin]
keyring = /etc/ceph/ceph.client.admin.keyring
[client.radosgw.gateway]
rgw keystone url = http://192.0.2.21:5000
rgw keystone admin token = 337b8816f019a04396a2e00e65e6c30ea96ba59b
rgw keystone accepted roles = admim _member_ swiftoperator
rgw keystone token cache size = 500
rgw keystone revocation interval = 500
rgw s3 auth use keystone = false
nss db path = /var/lib/ceph/nss
debug rgw = 20
host = gateway
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file = /var/log/ceph/client.radosgw.gateway.log
rgw dns name = gateway
>
> On Friday, October 10, 2014 9:20 AM, lakshmi k s wrote:
>
>
> Mark, I am going no where with
this. I am going to try with latest
> OpenStack build (build internal to my company) that has HA support. I
> will keep you posted.
>
>
> On Thursday, October 9, 2014 10:46 PM, Mark Kirkwood
> wrote:
>
>
> Oh, I see. That complicates it a wee bit (looks back at your messages).
> I see you have:
>
> rgw_keystone_url = http://192.0.8.2:5000 <http://192.0.8.2:5000/>
>
> So you'll need to amend/create etc a
>
>
>
> and put it in there. I suspect you might be better off changing your rgw
> kesytone url to use port 35357 (the public one). However I think that is
> a
side issue.
>
> Also just to double check - 192.0.8.2 *is* the server you are showing us
> the sites-available from?
>
> Cheers
>
> Mark
>
> On 10/10/14 12:50, lakshmi k s wrote:
> > Yes Mark, but there is no keystone.conf in this modified Openstack code.
> > There is only horizon.conf under /etc/apache2/sites-available folder.
> > And that has virtual host 80 only. Should I simply add :35357?
> >
> > root@overcloud-controller0-fjvtpqjip2hl
> <mailto:root@overcloud-controller0-fjvtpqjip2hl>
> > <mailto:root@overcloud-controller0-fjvtpqjip2hl
> <mailto:root@overcloud-controller0-fjvtpqjip2hl>>:/etc/apache2/sites-available#
> > ls
> > 000-default.conf default-ssl.conf horizon.conf
> >
> >
> >
> >
> > On Thursday, October 9, 2014 4:45 PM, Mark Kirkwood
> > <mailto:mark.kirkw...@catalyst.net.nz>> wrote:
> >
> >
> > Hmm - It looks to me like you added
the chunked request into Horizon
> > instead of Keystone. You want virtual host *:35357
> >
> >
> > On 10/10/14 12:32, lakshmi k s wrote:
> > > Have done this too, but in vain. I made changes to Horizon.conf as
> shown
> > > below. I had only I do not see the user being validated in radosgw log
> > > at all.
> > >
> > > root@overcloud-controller0-fjvtpqjip2hl
> <mailto:root@overcloud-controller0-fjvtpqjip2hl>
> > <mailto:root@overcloud-controller0-fjvtpqjip2hl
> <mailto:root@overcloud-controller0-fjvtpqjip2hl>>:/etc/apache2/sites-available#
> > ls
> > > 000-default.conf default-ssl.conf horizon.conf
> > >
> > >
> > >
> > > WSGIScriptAlias /
> > >
> >
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/wsgi/django.wsgi
> > > WSGIDaemonProcess horizon user=horizon group=horizon processes=3
> > > threads=10
home=/opt/stack/venvs/horizon
> > >
> >
> python-path=/opt/stack/venvs/horizon:/opt/stack/venvs/horizon/lib/python2.7/site-packages/
> > >WSGIApplicationGroup %{GLOBAL}
> > >
> > > SetEnv APACHE_RUN_USER horizon
>
Re: [ceph-users] Openstack keystone with Radosgw
Mark, I am going no where with this. I am going to try with latest OpenStack
build (build internal to my company) that has HA support. I will keep you
posted.
On Thursday, October 9, 2014 10:46 PM, Mark Kirkwood
wrote:
Oh, I see. That complicates it a wee bit (looks back at your messages).
I see you have:
rgw_keystone_url = http://192.0.8.2:5000
So you'll need to amend/create etc a
and put it in there. I suspect you might be better off changing your rgw
kesytone url to use port 35357 (the public one). However I think that is
a side issue.
Also just to double check - 192.0.8.2 *is* the server you are showing us
the sites-available from?
Cheers
Mark
On 10/10/14 12:50, lakshmi k s wrote:
> Yes Mark, but there is no keystone.conf in this modified Openstack code.
> There is only horizon.conf under /etc/apache2/sites-available folder.
> And that has virtual host 80 only. Should I simply add :35357?
>
> root@overcloud-controller0-fjvtpqjip2hl
> <mailto:root@overcloud-controller0-fjvtpqjip2hl>:/etc/apache2/sites-available#
> ls
> 000-default.conf default-ssl.conf horizon.conf
>
>
>
>
> On Thursday, October 9, 2014 4:45 PM, Mark Kirkwood
> wrote:
>
>
> Hmm - It looks to me like you added the chunked request into Horizon
> instead of Keystone. You want virtual host *:35357
>
>
> On 10/10/14 12:32, lakshmi k s wrote:
> > Have done this too, but in vain. I made changes to Horizon.conf as shown
> > below. I had only I do not see the user being validated in radosgw log
> > at all.
> >
> > root@overcloud-controller0-fjvtpqjip2hl
> <mailto:root@overcloud-controller0-fjvtpqjip2hl>:/etc/apache2/sites-available#
> ls
> > 000-default.conf default-ssl.conf horizon.conf
> >
> >
> >
> > WSGIScriptAlias /
> >
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/wsgi/django.wsgi
> > WSGIDaemonProcess horizon user=horizon group=horizon processes=3
> > threads=10 home=/opt/stack/venvs/horizon
> >
> python-path=/opt/stack/venvs/horizon:/opt/stack/venvs/horizon/lib/python2.7/site-packages/
> > WSGIApplicationGroup %{GLOBAL}
> >
> > SetEnv APACHE_RUN_USER horizon
> > SetEnv APACHE_RUN_GROUP horizon
> > WSGIProcessGroup horizon
> >WSGIChunkedRequest On
> >
> > DocumentRoot
> >
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static
> > Alias /static
> >
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static
> > Alias /media
> >
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static
> >
> >
> > Options FollowSymLinks
> > AllowOverride None
> >
> >
> >>
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static>
> > Options Indexes FollowSymLinks MultiViews
> > Require all granted
> > AllowOverride None
> > Order allow,deny
> > allow from all
> >
> >
> >> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard>
> > Options Indexes FollowSymLinks MultiViews
> > Require all granted
> > AllowOverride None
> > Order allow,deny
> > allow from all
> >
> >
> > ErrorLog /var/log/httpd/horizon_error.log
> > LogLevel debug
> > CustomLog /var/log/httpd/horizon_access.log combined
> >
> >
> > WSGISocketPrefix /var/run/httpd
> >
> > --
> >
> >
> >
> >
> > On Thursday, October 9, 2014 3:51 PM, Mark Kirkwood
> > <mailto:mark.kirkw...@catalyst.net.nz>> wrote:
> >
> >
> > No, I don't have any explicit ssl enabled in the rgw site.
> >
> > Now you might be running into http://tracker.ceph.com/issues/7796
> > <http://tracker.ceph.com/issues/7796>. So
> > check if you have enabled
> >
> > WSGIChunkedRequest On
> >
> > In your keystone virtualhost setup (explained in the issue).
> >
> > Cheers
> >
> > Mark
> >
> >
> > On 10/10/14 11:03, lakshmi k s wrote:
> > > Right, I have these certs on both nodes - keystone node and rgw
> gateway
> > > node. Not sure where I am going wrong. And what about SSL? Should the
> > > following be in rgw.conf in gateway node? I am not using this as
>
Re: [ceph-users] Openstack keystone with Radosgw
Yes Mark, but there is no keystone.conf in this modified Openstack code. There
is only horizon.conf under /etc/apache2/sites-available folder. And that has
virtual host 80 only. Should I simply add :35357?
root@overcloud-controller0-fjvtpqjip2hl:/etc/apache2/sites-available# ls
000-default.conf default-ssl.conf horizon.conf
On Thursday, October 9, 2014 4:45 PM, Mark Kirkwood
wrote:
Hmm - It looks to me like you added the chunked request into Horizon
instead of Keystone. You want virtual host *:35357
On 10/10/14 12:32, lakshmi k s wrote:
> Have done this too, but in vain. I made changes to Horizon.conf as shown
> below. I had only I do not see the user being validated in radosgw log
> at all.
>
> root@overcloud-controller0-fjvtpqjip2hl:/etc/apache2/sites-available# ls
> 000-default.conf default-ssl.conf horizon.conf
>
>
>
> WSGIScriptAlias /
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/wsgi/django.wsgi
> WSGIDaemonProcess horizon user=horizon group=horizon processes=3
> threads=10 home=/opt/stack/venvs/horizon
> python-path=/opt/stack/venvs/horizon:/opt/stack/venvs/horizon/lib/python2.7/site-packages/
> WSGIApplicationGroup %{GLOBAL}
>
> SetEnv APACHE_RUN_USER horizon
> SetEnv APACHE_RUN_GROUP horizon
> WSGIProcessGroup horizon
>WSGIChunkedRequest On
>
> DocumentRoot
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static
> Alias /static
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static
> Alias /media
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static
>
>
> Options FollowSymLinks
> AllowOverride None
>
>
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static>
> Options Indexes FollowSymLinks MultiViews
> Require all granted
> AllowOverride None
> Order allow,deny
> allow from all
>
>
> /opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard>
> Options Indexes FollowSymLinks MultiViews
> Require all granted
> AllowOverride None
> Order allow,deny
> allow from all
>
>
> ErrorLog /var/log/httpd/horizon_error.log
> LogLevel debug
> CustomLog /var/log/httpd/horizon_access.log combined
>
>
> WSGISocketPrefix /var/run/httpd
>
> --
>
>
>
>
> On Thursday, October 9, 2014 3:51 PM, Mark Kirkwood
> wrote:
>
>
> No, I don't have any explicit ssl enabled in the rgw site.
>
> Now you might be running into http://tracker.ceph.com/issues/7796
> <http://tracker.ceph.com/issues/7796>. So
> check if you have enabled
>
> WSGIChunkedRequest On
>
> In your keystone virtualhost setup (explained in the issue).
>
> Cheers
>
> Mark
>
>
> On 10/10/14 11:03, lakshmi k s wrote:
> > Right, I have these certs on both nodes - keystone node and rgw gateway
> > node. Not sure where I am going wrong. And what about SSL? Should the
> > following be in rgw.conf in gateway node? I am not using this as it was
> > optional.
> >
> > SSLEngine on
> > SSLCertificateFile /etc/apache2/ssl/apache.crt
> > SSLCertificateKeyFile /etc/apache2/ssl/apache.key
> > SetEnv SERVER_PORT_SECURE 443
> >
> >
> >
> >
> >
> > On Thursday, October 9, 2014 2:48 PM, Mark Kirkwood
> > <mailto:mark.kirkw...@catalyst.net.nz>> wrote:
> >
> >
> > Almost - the converted certs need to be saved on your *rgw* host in
> > nss_db_path (default is /var/ceph/nss but wherever you have it
> > configured should be ok). Then restart the gateway.
> >
> > What is happening is the the rgw needs these certs to speak with
> > encryption to the keystone server (the latter does not need anything
> > changed, as it is already using encryption).
> >
> > Regards
> >
> > Mark
> >
> > On 10/10/14 08:31, lakshmi k s wrote:
> > > Thanks Mark. I got past this error being root. So essentially, I
> copied
> > > the certs from openstack controller node to gateway node. Did the
> > > conversion using certutil and copied the files back to controller node
> > > under /var/lib/ceph/nss directory. Is this the correct directory? Ceph
> > > doc says /var/ceph/nss though.
> > >
> > > But after this, I tried to use curl GET command, but in vain.Same old
> > > 401
Re: [ceph-users] Openstack keystone with Radosgw
Have done this too, but in vain. I made changes to Horizon.conf as shown below.
I had only I do not see the user being validated in radosgw log at all.
root@overcloud-controller0-fjvtpqjip2hl:/etc/apache2/sites-available# ls
000-default.conf default-ssl.conf horizon.conf
WSGIScriptAlias /
/opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/wsgi/django.wsgi
WSGIDaemonProcess horizon user=horizon group=horizon processes=3 threads=10
home=/opt/stack/venvs/horizon
python-path=/opt/stack/venvs/horizon:/opt/stack/venvs/horizon/lib/python2.7/site-packages/
WSGIApplicationGroup %{GLOBAL}
SetEnv APACHE_RUN_USER horizon
SetEnv APACHE_RUN_GROUP horizon
WSGIProcessGroup horizon
WSGIChunkedRequest On
DocumentRoot
/opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static
Alias /static
/opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static
Alias /media
/opt/stack/venvs/horizon/lib/python2.7/site-packages/openstack_dashboard/static
Options FollowSymLinks
AllowOverride None
Options Indexes FollowSymLinks MultiViews
Require all granted
AllowOverride None
Order allow,deny
allow from all
Options Indexes FollowSymLinks MultiViews
Require all granted
AllowOverride None
Order allow,deny
allow from all
ErrorLog /var/log/httpd/horizon_error.log
LogLevel debug
CustomLog /var/log/httpd/horizon_access.log combined
WSGISocketPrefix /var/run/httpd
--
On Thursday, October 9, 2014 3:51 PM, Mark Kirkwood
wrote:
No, I don't have any explicit ssl enabled in the rgw site.
Now you might be running into http://tracker.ceph.com/issues/7796 . So
check if you have enabled
WSGIChunkedRequest On
In your keystone virtualhost setup (explained in the issue).
Cheers
Mark
On 10/10/14 11:03, lakshmi k s wrote:
> Right, I have these certs on both nodes - keystone node and rgw gateway
> node. Not sure where I am going wrong. And what about SSL? Should the
> following be in rgw.conf in gateway node? I am not using this as it was
> optional.
>
> SSLEngine on
> SSLCertificateFile /etc/apache2/ssl/apache.crt
> SSLCertificateKeyFile /etc/apache2/ssl/apache.key
> SetEnv SERVER_PORT_SECURE 443
>
>
>
>
>
> On Thursday, October 9, 2014 2:48 PM, Mark Kirkwood
> wrote:
>
>
> Almost - the converted certs need to be saved on your *rgw* host in
> nss_db_path (default is /var/ceph/nss but wherever you have it
> configured should be ok). Then restart the gateway.
>
> What is happening is the the rgw needs these certs to speak with
> encryption to the keystone server (the latter does not need anything
> changed, as it is already using encryption).
>
> Regards
>
> Mark
>
> On 10/10/14 08:31, lakshmi k s wrote:
> > Thanks Mark. I got past this error being root. So essentially, I copied
> > the certs from openstack controller node to gateway node. Did the
> > conversion using certutil and copied the files back to controller node
> > under /var/lib/ceph/nss directory. Is this the correct directory? Ceph
> > doc says /var/ceph/nss though.
> >
> > But after this, I tried to use curl GET command, but in vain.Same old
> > 401 - Authorization failure.
> >
> > curl -i -X GET
> > http://gateway.ex.com/swift/v1/AUTH_bad9e2232b304f89acb03436635b80cc
> <http://gateway.ex.com/swift/v1/AUTH_bad9e2232b304f89acb03436635b80cc>-H
> > "X-Auth-
> > Token: a510edb22f074946940cd4c07aafcd9d"
> >
> > HTTP/1.1 401 Unauthorized
> > Date: Thu, 09 Oct 2014 19:17:31 GMT
> > Server: Apache/2.4.7 (Ubuntu)
> > Accept-Ranges: bytes
> > Content-Length: 12
> > Content-Type: text/plain; charset=utf-8
> > AccessDeniedroot
> >
> > Not much difference in radosgw logs too. Note that the token used above
> > is same one in ceph.conf file too. Please help.
> >
> > [client.radosgw.gateway]
> > rgw keystone url = http://192.0.8.2:5000 <http://192.0.8.2:5000/>
> > rgw keystone admin token = a510edb22f074946940cd4c07aafcd9d
> > rgw keystone accepted roles = admim Member _member_ swiftoperator
> > rgw keystone token cache size = 500
> > rgw keystone revocation interval = 500
> > rgw s3 auth use keystone = false
> > nss db path = /var/lib/ceph/nss
> > debug rgw = 20
> > host = gateway
> > keyring = /etc/ceph/ceph.client.radosgw.keyring
> > rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
> > log file = /var/log/ceph/client.radosgw.gateway.log
> > rgw
Re: [ceph-users] Openstack keystone with Radosgw
Right, I have these certs on both nodes - keystone node and rgw gateway node. Not sure where I am going wrong. And what about SSL? Should the following be in rgw.conf in gateway node? I am not using this as it was optional. SSLEngine on SSLCertificateFile /etc/apache2/ssl/apache.crt SSLCertificateKeyFile /etc/apache2/ssl/apache.key SetEnv SERVER_PORT_SECURE 443 On Thursday, October 9, 2014 2:48 PM, Mark Kirkwood wrote: Almost - the converted certs need to be saved on your *rgw* host in nss_db_path (default is /var/ceph/nss but wherever you have it configured should be ok). Then restart the gateway. What is happening is the the rgw needs these certs to speak with encryption to the keystone server (the latter does not need anything changed, as it is already using encryption). Regards Mark On 10/10/14 08:31, lakshmi k s wrote: > Thanks Mark. I got past this error being root. So essentially, I copied > the certs from openstack controller node to gateway node. Did the > conversion using certutil and copied the files back to controller node > under /var/lib/ceph/nss directory. Is this the correct directory? Ceph > doc says /var/ceph/nss though. > > But after this, I tried to use curl GET command, but in vain.Same old > 401 - Authorization failure. > > curl -i -X GET > http://gateway.ex.com/swift/v1/AUTH_bad9e2232b304f89acb03436635b80cc -H > "X-Auth- > Token: a510edb22f074946940cd4c07aafcd9d" > > HTTP/1.1 401 Unauthorized > Date: Thu, 09 Oct 2014 19:17:31 GMT > Server: Apache/2.4.7 (Ubuntu) > Accept-Ranges: bytes > Content-Length: 12 > Content-Type: text/plain; charset=utf-8 > AccessDeniedroot > > Not much difference in radosgw logs too. Note that the token used above > is same one in ceph.conf file too. Please help. > > [client.radosgw.gateway] > rgw keystone url = http://192.0.8.2:5000 > rgw keystone admin token = a510edb22f074946940cd4c07aafcd9d > rgw keystone accepted roles = admim Member _member_ swiftoperator > rgw keystone token cache size = 500 > rgw keystone revocation interval = 500 > rgw s3 auth use keystone = false > nss db path = /var/lib/ceph/nss > debug rgw = 20 > host = gateway > keyring = /etc/ceph/ceph.client.radosgw.keyring > rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock > log file = /var/log/ceph/client.radosgw.gateway.log > rgw dns name = gateway > > > > > > On Thursday, October 9, 2014 1:15 AM, Mark Kirkwood > wrote: > > > I ran into this - needed to actually be root via sudo -i or similar, > *then* it worked. Unhelpful error message is I think referring to no > intialized db. > > On 09/10/14 16:36, lakshmi k s wrote: > > Good workaround. But it did not work. Not sure what this error is all > > about now. > > > > gateway@gateway <mailto:gateway@gateway>:~$ openssl x509 -in > /home/gateway/ca.pem -pubkey | > > certutil -d /var/lib/ceph/nss -A -n ca -t "TCu,Cu,Tuw" > > certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > > certificate/key database is in an old, unsupported format. > > > > > > > > On Wednesday, October 8, 2014 7:55 PM, Mark Kirkwood > > <mailto:mark.kirkw...@catalyst.net.nz>> wrote: > > > > > > As a workaround check if your rgw host has openssl and certutil > > installed, if so you can copy the relevant unconverted certs over to it > > and convert 'em there. > > > > On 09/10/14 15:07, lakshmi k s wrote: > > > Tried aptitude as well, but no luck. > > > > > > Ceph users, have you tried to install libnss3-tools or certutil > tool on > > > debian/ubuntu? If so, how did you go about this problem. > > > > > > > > > On Wednesday, October 8, 2014 7:01 PM, Mark Kirkwood > > > mailto:mark.kirkw...@catalyst.net.nz> > > <mailto:mark.kirkw...@catalyst.net.nz > <mailto:mark.kirkw...@catalyst.net.nz>>> wrote: > > > > > > > > > > Ok, so that is the thing to get sorted. I'd suggest posting the > error(s) > > > you are getting perhaps here (someone else might know), but definitely > > > to one of the Debian specific lists. > > > > > > In the meantime perhaps try installing the packages with aptitude > rather > > > than apt-get - if there is some fancy footwork required it is fairly > > > smart about what needs to be done. > > > > > > Cheers > > > > > > Mark > > > > > > On 09/10/14 14:38, lakshmi k s wrote: > > > > Thanks Mark. I have been trying to install this on controller > > node. But > > > > for some reason, I am unable to install certutil or > libnss3-tools on > > > > debian. I am not sure how to proceed. > > > > > > > > > > > > > > > > > > > > > >___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Openstack keystone with Radosgw
Thanks Mark. I got past this error being root. So essentially, I copied the certs from openstack controller node to gateway node. Did the conversion using certutil and copied the files back to controller node under /var/lib/ceph/nss directory. Is this the correct directory? Ceph doc says /var/ceph/nss though. But after this, I tried to use curl GET command, but in vain.Same old 401 - Authorization failure. curl -i -X GET http://gateway.ex.com/swift/v1/AUTH_bad9e2232b304f89acb03436635b80cc -H "X-Auth- Token: a510edb22f074946940cd4c07aafcd9d" HTTP/1.1 401 Unauthorized Date: Thu, 09 Oct 2014 19:17:31 GMT Server: Apache/2.4.7 (Ubuntu) Accept-Ranges: bytes Content-Length: 12 Content-Type: text/plain; charset=utf-8 AccessDeniedroot Not much difference in radosgw logs too. Note that the token used above is same one in ceph.conf file too. Please help. [client.radosgw.gateway] rgw keystone url = http://192.0.8.2:5000 rgw keystone admin token = a510edb22f074946940cd4c07aafcd9d rgw keystone accepted roles = admim Member _member_ swiftoperator rgw keystone token cache size = 500 rgw keystone revocation interval = 500 rgw s3 auth use keystone = false nss db path = /var/lib/ceph/nss debug rgw = 20 host = gateway keyring = /etc/ceph/ceph.client.radosgw.keyring rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock log file = /var/log/ceph/client.radosgw.gateway.log rgw dns name = gateway On Thursday, October 9, 2014 1:15 AM, Mark Kirkwood wrote: I ran into this - needed to actually be root via sudo -i or similar, *then* it worked. Unhelpful error message is I think referring to no intialized db. On 09/10/14 16:36, lakshmi k s wrote: > Good workaround. But it did not work. Not sure what this error is all > about now. > > gateway@gateway:~$ openssl x509 -in /home/gateway/ca.pem -pubkey | > certutil -d /var/lib/ceph/nss -A -n ca -t "TCu,Cu,Tuw" > certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The > certificate/key database is in an old, unsupported format. > > > > On Wednesday, October 8, 2014 7:55 PM, Mark Kirkwood > wrote: > > > As a workaround check if your rgw host has openssl and certutil > installed, if so you can copy the relevant unconverted certs over to it > and convert 'em there. > > On 09/10/14 15:07, lakshmi k s wrote: > > Tried aptitude as well, but no luck. > > > > Ceph users, have you tried to install libnss3-tools or certutil tool on > > debian/ubuntu? If so, how did you go about this problem. > > > > > > On Wednesday, October 8, 2014 7:01 PM, Mark Kirkwood > > <mailto:mark.kirkw...@catalyst.net.nz>> wrote: > > > > > > Ok, so that is the thing to get sorted. I'd suggest posting the error(s) > > you are getting perhaps here (someone else might know), but definitely > > to one of the Debian specific lists. > > > > In the meantime perhaps try installing the packages with aptitude rather > > than apt-get - if there is some fancy footwork required it is fairly > > smart about what needs to be done. > > > > Cheers > > > > Mark > > > > On 09/10/14 14:38, lakshmi k s wrote: > > > Thanks Mark. I have been trying to install this on controller > node. But > > > for some reason, I am unable to install certutil or libnss3-tools on > > > debian. I am not sure how to proceed. > > > > > > > > > > > >___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Openstack keystone with Radosgw
Good workaround. But it did not work. Not sure what this error is all about now. gateway@gateway:~$ openssl x509 -in /home/gateway/ca.pem -pubkey | certutil -d /var/lib/ceph/nss -A -n ca -t "TCu,Cu,Tuw" certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format. On Wednesday, October 8, 2014 7:55 PM, Mark Kirkwood wrote: As a workaround check if your rgw host has openssl and certutil installed, if so you can copy the relevant unconverted certs over to it and convert 'em there. On 09/10/14 15:07, lakshmi k s wrote: > Tried aptitude as well, but no luck. > > Ceph users, have you tried to install libnss3-tools or certutil tool on > debian/ubuntu? If so, how did you go about this problem. > > > On Wednesday, October 8, 2014 7:01 PM, Mark Kirkwood > wrote: > > > Ok, so that is the thing to get sorted. I'd suggest posting the error(s) > you are getting perhaps here (someone else might know), but definitely > to one of the Debian specific lists. > > In the meantime perhaps try installing the packages with aptitude rather > than apt-get - if there is some fancy footwork required it is fairly > smart about what needs to be done. > > Cheers > > Mark > > On 09/10/14 14:38, lakshmi k s wrote: > > Thanks Mark. I have been trying to install this on controller node. But > > for some reason, I am unable to install certutil or libnss3-tools on > > debian. I am not sure how to proceed. > > > > >___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Openstack keystone with Radosgw
Tried aptitude as well, but no luck. Ceph users, have you tried to install libnss3-tools or certutil tool on debian/ubuntu? If so, how did you go about this problem. On Wednesday, October 8, 2014 7:01 PM, Mark Kirkwood wrote: Ok, so that is the thing to get sorted. I'd suggest posting the error(s) you are getting perhaps here (someone else might know), but definitely to one of the Debian specific lists. In the meantime perhaps try installing the packages with aptitude rather than apt-get - if there is some fancy footwork required it is fairly smart about what needs to be done. Cheers Mark On 09/10/14 14:38, lakshmi k s wrote: > Thanks Mark. I have been trying to install this on controller node. But > for some reason, I am unable to install certutil or libnss3-tools on > debian. I am not sure how to proceed. >___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Openstack keystone with Radosgw
Thanks Mark. I have been trying to install this on controller node. But for some reason, I am unable to install certutil or libnss3-tools on debian. I am not sure how to proceed. On Wednesday, October 8, 2014 6:26 PM, Mark Kirkwood wrote: If you are using ceph + radosgw packages they should be built with the nss option (--with-nss), so nothing to do there. For the server running keystone you need to do: (root) $ mkdir /var/ceph/nss (root) $ openssl x509 -in /etc/keystone/ssl/certs/ca.pem -pubkey | \ certutil -d /var/ceph/nss -A -n ca -t "TCu,Cu,Tuw" (root) $ openssl x509 -in /etc/keystone/ssl/certs/signing_cert.pem -pub (root) rsync -av /var/ceph/nss/* rgw-host:/var/ceph/nss as indicated in the ceph docs. I found I needed to actually be root for this to work (i.e sudo did not work), but apart from that no problem. You need to install whatever packages give you the openssl and certutil binaries. Cheers Mark On 09/10/14 05:21, lakshmi k s wrote: > Hello Mark, > > Thanks for your reply. Where should I be installing NSS package? On > Gateway or Openstack Controller node? On both, I could not execute the > following command as it resulted in bunch of errors. > > openssl x509 > -in /etc/keystone/ssl/certs/ca.pem -pubkey | certutil -d /var/ceph/nss -A > -n ca -t "TCu,Cu,Tuw" > > Also, you mentioned about SSL. What should I be doing for this? Should > rgw.conf in /etc/apache2/sites-enabled on gateway node be configured for SSL > like this below. I do not have this right now. > > SSLEngine on > SSLCertificateFile /etc/apache2/ssl/apache.crt > SSLCertificateKeyFile /etc/apache2/ssl/apache.key > SetEnv SERVER_PORT_SECURE 443 > >___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Openstack keystone with Radosgw
Hello Mark,
Thanks for your reply. Where should I be installing NSS package? On Gateway or
Openstack Controller node? On both, I could not execute the following command
as it resulted in bunch of errors.
openssl x509 -in /etc/keystone/ssl/certs/ca.pem -pubkey | certutil -d
/var/ceph/nss -A -n ca -t "TCu,Cu,Tuw"
Also, you mentioned about SSL. What should I be doing for this? Should rgw.conf
in /etc/apache2/sites-enabled on gateway node be configured for SSL like this
below. I do not have this right now.
SSLEngine on
SSLCertificateFile /etc/apache2/ssl/apache.crt
SSLCertificateKeyFile /etc/apache2/ssl/apache.key
SetEnv SERVER_PORT_SECURE 443
Appreciate your help.
Lakshmi.
On Tuesday, October 7, 2014 10:23 PM, Mark Kirkwood
wrote:
On 08/10/14 11:02, lakshmi k s wrote:
> I am trying to integrate OpenStack Keystone with Ceph Object Store using
> the link - http://ceph.com/docs/master/radosgw/keystone.
> <http://ceph.com/docs/master/radosgw/keystone> Swift V1.0 (without
> keystone) works quite fine. But for some reason, Swift v2.0 keystone
> calls to Ceph Object Store always results in 401 - Unauthorized message.
> I have tried to get a new token by contacting keystone and used that
> token for making Swift calls. But no luck. Please note that all other
> services like nova list, cinder list work which means Keystone is setup
> correctly. But Swift service fails. Only step I did not execute is to
> install nss db as I ran into package dependency issues. But I have
> commented that flag in ceph.conf . My ceph.conf looks like this below.
> [global]
> fsid = b35e8496-e809-416a-bd66-aba761d78fac
> mon_initial_members = node1
> mon_host = 192.0.2.211
> auth_cluster_required = cephx
> auth_service_required = cephx
> auth_client_required = cephx
> filestore_xattr_use_omap = true
> [client.admin]
> keyring = /etc/ceph/ceph.client.admin.keyring
> [client.radosgw.gateway]
> rgw keystone url = http://192.0.8.2:5000
> rgw keystone admin token = 9c2ef11a69044defb9dbfa0f8ab73d86
> rgw keystone accepted roles = admin, Member, swiftoperator
> rgw keystone token cache size = 100
> rgw keystone revocation interval = 600
> rgw s3 auth use keystone = false
> #nss db path = /var/ceph/nss
> host = gateway
> keyring = /etc/ceph/ceph.client.radosgw.keyring
> rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
> log file = /var/log/ceph/client.radosgw.gateway.log
> rgw dns name = gateway
>
>
> *Output of Swift list*
> root@overcloud-controller0-fjvtpqjip2hl:~# swift --debug -V 2.0 -A
> http://192.0.8.2:5000/v2.0 -U ceph:cephUser -K "ceph123" list
>
> DEBUG:keystoneclient.session:REQ: curl -i -X POST
> http://192.0.8.2:5000/v2.0/tokens -H "Content-Type: application/json" -H
> "Accept: application/json" -H "User-Agent: python-keystoneclient" -d
> '{"auth": {"tenantName": "ceph", "passwordCredentials": {"username":
> "cephUser", "password": "ceph123"}}}'
> INFO:requests.packages.urllib3.connectionpool:Starting new HTTP
> connection (1): 192.0.8.2
> DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens
> HTTP/1.1" 200 3910
> DEBUG:keystoneclient.session:RESP: [200] {'date': 'Tue, 07 Oct 2014
> 20:05:20 GMT', 'content-type': 'application/json', 'content-length':
> '3910', 'vary': 'X-Auth-Token'}
> RESP BODY: {"access": {"token": {"issued_at":
> "2014-10-07T20:05:20.480562", "expires": "2014-10-08T00:05:20Z", "id":
> "45e14981c41f4c8c8055849b39bd4c23", "tenant": {"description": "",
> "enabled": true, "id": "bad9e2232b304f89acb03436635b80cc", "name":
> "ceph"}}, "serviceCatalog": [{"endpoints": [{"adminURL":
> "http://192.0.8.2:8774/v2/bad9e2232b304f89acb03436635b80cc";, "region":
> "regionOne", "internalURL":
> "http://192.0.8.2:8774/v2/bad9e2232b304f89acb03436635b80cc";, "id":
> "40e53124619d479ab0c34a99c7619bcc", "publicURL":
> "http://192.0.8.2:8774/v2/bad9e2232b304f89acb03436635b80cc"}],
> "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints":
> [{"adminURL": "http://192.0.8.2:9696/";, "region": "regionOne",
> "internalURL": "http://192.0.8.2:9696/";, "id":
> "4e5fb12504024554a762b46391b46309", "publicURL":
> "h
[ceph-users] Openstack keystone with Radosgw
I am trying to integrate OpenStack Keystone with Ceph Object
Store using the link - http://ceph.com/docs/master/radosgw/keystone. Swift
V1.0 (without keystone) works
quite fine. But for some reason, Swift v2.0 keystone calls to Ceph Object Store
always
results in 401 - Unauthorized message. I have tried to get a new token by
contacting keystone
and used that token for making Swift calls. But no luck. Please note that all
other services like nova list, cinder list work which means Keystone is setup
correctly. But Swift service fails. Only step I did not execute is to install
nss db as I ran into package dependency issues. But I have commented that flag
in ceph.conf . My ceph.conf looks like this below.
[global]
fsid = b35e8496-e809-416a-bd66-aba761d78fac
mon_initial_members = node1
mon_host = 192.0.2.211
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
[client.admin]
keyring = /etc/ceph/ceph.client.admin.keyring
[client.radosgw.gateway]
rgw keystone url = http://192.0.8.2:5000
rgw keystone admin token = 9c2ef11a69044defb9dbfa0f8ab73d86
rgw keystone accepted roles = admin, Member, swiftoperator
rgw keystone token cache size = 100
rgw keystone revocation interval = 600
rgw s3 auth use keystone = false
#nss db path = /var/ceph/nss
host = gateway
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path =
/var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file = /var/log/ceph/client.radosgw.gateway.log
rgw dns name = gateway
Output of Swift list
root@overcloud-controller0-fjvtpqjip2hl:~#
swift --debug -V 2.0 -A http://192.0.8.2:5000/v2.0 -U ceph:cephUser -K
"ceph123" list
DEBUG:keystoneclient.session:REQ:
curl -i -X POST http://192.0.8.2:5000/v2.0/tokens -H "Content-Type:
application/json" -H "Accept: application/json" -H
"User-Agent: python-keystoneclient" -d '{"auth":
{"tenantName": "ceph", "passwordCredentials":
{"username": "cephUser", "password":
"ceph123"}}}'
INFO:requests.packages.urllib3.connectionpool:Starting
new HTTP connection (1): 192.0.8.2
DEBUG:requests.packages.urllib3.connectionpool:"POST
/v2.0/tokens HTTP/1.1" 200 3910
DEBUG:keystoneclient.session:RESP:
[200] {'date': 'Tue, 07 Oct 2014 20:05:20 GMT', 'content-type':
'application/json', 'content-length': '3910', 'vary': 'X-Auth-Token'}
RESP
BODY: {"access": {"token": {"issued_at": "2014-10-07T20:05:20.480562",
"expires": "2014-10-08T00:05:20Z", "id":
"45e14981c41f4c8c8055849b39bd4c23", "tenant":
{"description": "", "enabled": true,
"id": "bad9e2232b304f89acb03436635b80cc", "name":
"ceph"}}, "serviceCatalog": [{"endpoints":
[{"adminURL":
"http://192.0.8.2:8774/v2/bad9e2232b304f89acb03436635b80cc";,
"region": "regionOne", "internalURL":
"http://192.0.8.2:8774/v2/bad9e2232b304f89acb03436635b80cc";,
"id": "40e53124619d479ab0c34a99c7619bcc",
"publicURL": "http://192.0.8.2:8774/v2/bad9e2232b304f89acb03436635b80cc"}],
"endpoints_links": [], "type": "compute",
"name": "nova"}, {"endpoints":
[{"adminURL": "http://192.0.8.2:9696/";, "region":
"regionOne", "internalURL":
"http://192.0.8.2:9696/";, "id":
"4e5fb12504024554a762b46391b46309", "publicURL":
"http://192.0.8.2:9696/"}], "endpoints_links": [],
"type": "network", "name": "neutron"},
{"endpoints": [{"adminURL":
"http://192.0.8.2:8774/v3";, "region":
"regionOne", "internalURL":
"http://192.0.8.2:8774/v3";, "id":
"4e9f7514c3d94bd4b505207cfa52c306", "publicURL":
"http://192.0.8.2:8774/v3"}], "endpoints_links": [],
"type": "computev3", "name": "nova"},
{"endpoints": [{"adminURL": "http://192.0.8.2:9292/";,
"region": "regionOne", "internalURL":
"http://192.0.8.2:9292/";, "id":
"3305668e44fc43f4bb57b45aa599d454", "publicURL":
"http://192.0.8.2:9292/"}], "endpoints_links": [],
"type": "image", "name": "glance"},
{"endpoints": [{"adminURL": "http://192.0.8.2:21131/v1";,
"region": "regionOne", "internalURL":
"http://192.0.8.2:21131/v1";, "id": "7b4ac2efaeba4074988e397bee403caa",
"publicURL": "http://192.0.8.2:21131/v1"}],
"endpoints_links": [], "type": "hp-catalog",
"name": "sherpa"}, {"endpoints":
[{"adminURL": "http://192.0.8.2:8777/";, "region":
"regionOne", "internalURL":
"http://192.0.8.2:8777/";, "id": "2f1de9c2e81049e99cd4da266931780b",
"publicURL": "http://192.0.8.2:8777/"}],
"endpoints_links": [], "type": "metering",
"name": "ceilometer"}, {"endpoints":
[{"adminURL":
"http://192.0.8.2:8776/v1/bad9e2232b304f89acb03436635b80cc";,
"region": "regionOne", "internalURL":
"http://192.0.8.2:8776/v1/bad9e2232b304f89acb03436635b80cc";,
"id": "0bbc1c8d91574c2083b6b28b237c7004",
"publicURL":
"http://192.0.8.2:8776/v1/bad9e2232b304f89acb03436635b80cc"}],
"endpoints_links": [], "type": "volume",
"name": "cinder"}, {"endpoints":
[{"adminURL": "http://192.0.8.2:8773/services/Admin";,
"region": "regionOne", "internalURL":
"http://192.0.8.2:8773/services/Cloud";, "id":
"b15e7b43c7a44831a036f6f01479a6b1", "publicURL":
"http://192.0.8.2:8773/services/Cloud"}],
"endpoints_links": [], "type": "ec2",
"name": "ec2"}, {"endpoints":
Re: [ceph-users] 403-Forbidden error using radosgw
Below is the output of radosgw admin user info. Am I missing something here.
Appreciate your help.
ceph-gateway@ceph-gateway:~$ radosgw-admin user info --uid=ganapati
{ "user_id": "ganapati",
"display_name": "I",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [
{ "id": "ganapati:swift",
"permissions": "full-control"}],
"keys": [
{ "user": "ganapati",
"access_key": "123",
"secret_key": "456"},
{ "user": "ganapati:swift",
"access_key": "Q39BTCD9D0MKN546RNDO",
"secret_key": ""}],
"swift_keys": [
{ "user": "ganapati:swift",
"secret_key": "GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ"}],
"caps": [
{ "type": "metadata",
"perm": "*"},
{ "type": "usage",
"perm": "*"},
{ "type": "users",
"perm": "*"},
{ "type": "zone",
"perm": "*"}],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": { "enabled": false,
"max_size_kb": -1,
"max_objects": -1},
"user_quota": { "enabled": false,
"max_size_kb": -1,
"max_objects": -1},
"temp_url_keys": []}
On Wednesday, July 16, 2014 8:23 AM, Vincenzo Pii wrote:
Maybe some of the user data is not correct...
If you try
radosgw-admin user info --uid=ganapati
is the subuser there?
The key that you must use should be under "swift_keys".
Otherwise, be sure that the user is created with
radosgw-admin key create --subuser=username:subusername --key-type=swift
--gen-secret
2014-07-16 16:31 GMT+02:00 lakshmi k s :
Thanks for the response. Curl yields the following -
>
>
>ceph-gateway@ceph-gateway:~$ curl -v -i http://ceph-gateway/auth -X GET -H
>"X-Auth-User:ganapati:swift" -H
>"X-Auth-Key:GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ"
>
>Hostname was NOT found in DNS cache
>Trying 127.0.1.1...
>Connected to ceph-gateway (127.0.1.1) port 80 (#0)
>GET /auth HTTP/1.1
>User-Agent: curl/7.35.0
>Host: ceph-gateway
>Accept: */*
>X-Auth-User:ganapati:swift
>X-Auth-Key:GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ
>
>
>HTTP/1.1 403 Forbidden
>HTTP/1.1 403 Forbidden
>Date: Wed, 16 Jul 2014 14:24:11 GMT
>Date: Wed, 16 Jul 2014 14:24:11 GMT
>Server Apache/2.4.7 (Ubuntu) is not blacklisted
>Server: Apache/2.4.7 (Ubuntu)
>Server: Apache/2.4.7 (Ubuntu)
>Accept-Ranges: bytes
>Accept-Ranges: bytes
>Content-Length: 23
>Content-Length: 23
>Content-Type: application/json
>Content-Type: application/json
>
>
><
>* Connection #0 to host ceph-gateway left intact
>{"Code":"AccessDenied"}ceph-gateway@ceph-gateway:~$
>
>
>
>
>
>
>
>On Wednesday, July 16, 2014 7:06 AM, Vincenzo Pii wrote:
>
>
>
>You may try to debug your issue by using curl requests.
>
>
>If you use your Swift credentials, a request of this format should give you a
>20X return code (probably 204):
>
>
>curl -v -i http:///auth -X GET -H "X-Auth-User: testuser:swiftuser"
>-H "X-Auth-Key: ksYDp8dul80Ta1PeDkFFyLem1FlrtvnyzYiaqvh8"
>
>
>
>If this works, you at least know that your auth mechanism is working.
>
>
>2014-07-16 8:33 GMT+02:00 Wido den Hollander :
>
>On 07/16/2014 07:58 AM, lakshmi k s wrote:
>>> Hello Ceph Users -
>>>
>>> My Ceph setup consists of 1 admin node, 3 OSDs, I radosgw and 1 client.
>>> One of OSD node also hosts monitor node. Ceph Health is OK and I have
>>> verified the radosgw runtime. I have created S3 and Swift users using
>>> radosgw-admin. But when I try to make any S3 or Swift calls, everything
>>> falls apart. For example -
>>> Python script -
>>> import boto
>>> import boto.s3.connection
>>> access_key = '123'
>>> secret_key = '456'
>>
>>Are you sure the access and secret key are correct? See my lines a bit
>>below.
>>
>>> conn = boto.connect_s3(
>>> aws_access_key_id = access_key,
>>> aws_secret_access_key = secret_key,
>>> host = 'ceph-gateway.e
Re: [ceph-users] 403-Forbidden error using radosgw
Thanks for the response. Curl yields the following -
ceph-gateway@ceph-gateway:~$ curl -v -i http://ceph-gateway/auth -X GET -H
"X-Auth-User:ganapati:swift" -H
"X-Auth-Key:GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ"
Hostname was NOT found in DNS cache
Trying 127.0.1.1...
Connected to ceph-gateway (127.0.1.1) port 80 (#0)
GET /auth HTTP/1.1
User-Agent: curl/7.35.0
Host: ceph-gateway
Accept: */*
X-Auth-User:ganapati:swift
X-Auth-Key:GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ
HTTP/1.1 403 Forbidden
HTTP/1.1 403 Forbidden
Date: Wed, 16 Jul 2014 14:24:11 GMT
Date: Wed, 16 Jul 2014 14:24:11 GMT
Server Apache/2.4.7 (Ubuntu) is not blacklisted
Server: Apache/2.4.7 (Ubuntu)
Server: Apache/2.4.7 (Ubuntu)
Accept-Ranges: bytes
Accept-Ranges: bytes
Content-Length: 23
Content-Length: 23
Content-Type: application/json
Content-Type: application/json
<
* Connection #0 to host ceph-gateway left intact
{"Code":"AccessDenied"}ceph-gateway@ceph-gateway:~$
On Wednesday, July 16, 2014 7:06 AM, Vincenzo Pii wrote:
You may try to debug your issue by using curl requests.
If you use your Swift credentials, a request of this format should give you a
20X return code (probably 204):
curl -v -i http:///auth -X GET -H "X-Auth-User: testuser:swiftuser"
-H "X-Auth-Key: ksYDp8dul80Ta1PeDkFFyLem1FlrtvnyzYiaqvh8"
If this works, you at least know that your auth mechanism is working.
2014-07-16 8:33 GMT+02:00 Wido den Hollander :
On 07/16/2014 07:58 AM, lakshmi k s wrote:
>> Hello Ceph Users -
>>
>> My Ceph setup consists of 1 admin node, 3 OSDs, I radosgw and 1 client.
>> One of OSD node also hosts monitor node. Ceph Health is OK and I have
>> verified the radosgw runtime. I have created S3 and Swift users using
>> radosgw-admin. But when I try to make any S3 or Swift calls, everything
>> falls apart. For example -
>> Python script -
>> import boto
>> import boto.s3.connection
>> access_key = '123'
>> secret_key = '456'
>
>Are you sure the access and secret key are correct? See my lines a bit
>below.
>
>> conn = boto.connect_s3(
>> aws_access_key_id = access_key,
>> aws_secret_access_key = secret_key,
>> host = 'ceph-gateway.ex.com',
>> is_secure=False,
>> calling_format = boto.s3.connection.OrdinaryCallingFormat(),
>> )
>> for bucket in conn.get_all_buckets():
>> print "{name}\t{created}".format(
>> name = bucket.name,
>> created = bucket.creation_date,
>> )
>> Client error-
>> Traceback (most recent call last):
>> File "dconnect.py", line 18, in
>> for bucket in conn.get_all_buckets():
>> File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line
>> 387, in get_all_buckets
>> response.status, response.reason, body)
>> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
>> > encoding="UTF-8"?>AccessDenied
>> Radosgw log
>> 2014-07-15 22:48:15.769125 7fbb85fdb7001 == starting new request
>> req=0x7fbbe910b290 =
>> 2014-07-15 22:48:15.769443 7fbb85fdb7002 req 17:0.000334::GET
>> http://ceph-gateway.ex.com/::initializing
>> 2014-07-15 22:48:15.769998 7fbb85fdb700 10 s->object= s->bucket=
>> 2014-07-15 22:48:15.770199 7fbb85fdb7002 req 17:0.001084:s3:GET
>> http://ceph-gateway.ex.com/::getting op
>> 2014-07-15 22:48:15.770345 7fbb85fdb7002 req 17:0.001231:s3:GET
>> http://ceph-gateway.ex.com/:list_buckets:authorizing
>> 2014-07-15 22:48:15.770846 7fbb85fdb700 20 get_obj_state:
>> rctx=0x7fbbc800f750 obj=.users:I420IKX56ZP09BTN4CML state=0x7fbbc8007c08
>> s->prefetch_data=0
>> 2014-07-15 22:48:15.771314 7fbb85fdb700 10 cache get:
>> name=.users+I420IKX56ZP09BTN4CML : hit
>> 2014-07-15 22:48:15.771442 7fbb85fdb700 20 get_obj_state: s->obj_tag was
>> set empty
>> 2014-07-15 22:48:15.771537 7fbb85fdb700 10 cache get:
>> name=.users+I420IKX56ZP09BTN4CML : hit
>> 2014-07-15 22:48:15.773278 7fbb85fdb700 20 get_obj_state:
>> rctx=0x7fbbc800f750 obj=.users.uid:lakshmi state=0x7fbbc8008208
>> s->prefetch_data=0
>> 2014-07-15 22:48:15.773288 7fbb85fdb700 10 cache get:
>> name=.users.uid+lakshmi : hit
>> 2014-07-15 22:48:15.773293 7fbb85fdb700 20 get_obj_state: s->obj_tag was
>> set empty
>> 2014-07-15 22:48:15.773297 7fbb85fdb700 10 cache get:
>> name=.users.uid+lakshmi : hit
>> 2014-07-15 22:48:15.774247 7fbb85fdb700 10 get_canon_resource():
>> dest=http://ceph-gateway.ex.com/
>> 2014-07-15 22:48:15.774326 7fbb85fdb700 10 auth_hdr:
>> GET
>> Wed, 16 Jul 2014 05:48:48 GMT
>> http://ceph-gateway.ex.com/
>> 2014-07-15 22:48:
Re: [ceph-users] 403-Forbidden error using radosgw
Resending my earlier message.
On Tuesday, July 15, 2014 10:58 PM, lakshmi k s wrote:
Hello Ceph Users -
My Ceph setup consists of 1 admin
node, 3 OSDs, I radosgw and 1 client. One of OSD node also hosts monitor node.
Ceph Health is OK and I have verified the radosgw runtime. I have created S3
and Swift users using radosgw-admin. But when I try to make any S3 or Swift
calls, everything falls apart. For example -
Python script -
import boto
import boto.s3.connection
access_key = '123'
secret_key = '456'
conn = boto.connect_s3(
aws_access_key_id = access_key,
aws_secret_access_key = secret_key,
host = 'ceph-gateway.ex.com',
is_secure=False,
calling_format = boto.s3.connection.OrdinaryCallingFormat(),
)
for bucket in conn.get_all_buckets():
print "{name}\t{created}".format(
name = bucket.name,
created = bucket.creation_date,
)
Client error-
Traceback (most recent call last):
File "dconnect.py",
line 18, in
for bucket in
conn.get_all_buckets():
File
"/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 387,
in get_all_buckets
response.status,
response.reason, body)
boto.exception.S3ResponseError:
S3ResponseError: 403 Forbidden
AccessDenied
Radosgw log
2014-07-15 22:48:15.769125
7fbb85fdb700 1 == starting new
request req=0x7fbbe910b290 =
2014-07-15 22:48:15.769443
7fbb85fdb700 2 req 17:0.000334::GET
http://ceph-gateway.ex.com/::initializing
2014-07-15 22:48:15.769998
7fbb85fdb700 10 s->object= s->bucket=
2014-07-15 22:48:15.770199
7fbb85fdb700 2 req 17:0.001084:s3:GET
http://ceph-gateway.ex.com/::getting op
2014-07-15 22:48:15.770345
7fbb85fdb700 2 req 17:0.001231:s3:GET
http://ceph-gateway.ex.com/:list_buckets:authorizing
2014-07-15 22:48:15.770846
7fbb85fdb700 20 get_obj_state: rctx=0x7fbbc800f750
obj=.users:I420IKX56ZP09BTN4CML state=0x7fbbc8007c08 s->prefetch_data=0
2014-07-15 22:48:15.771314
7fbb85fdb700 10 cache get: name=.users+I420IKX56ZP09BTN4CML : hit
2014-07-15 22:48:15.771442
7fbb85fdb700 20 get_obj_state: s->obj_tag was set empty
2014-07-15 22:48:15.771537
7fbb85fdb700 10 cache get: name=.users+I420IKX56ZP09BTN4CML : hit
2014-07-15 22:48:15.773278
7fbb85fdb700 20 get_obj_state: rctx=0x7fbbc800f750 obj=.users.uid:lakshmi
state=0x7fbbc8008208 s->prefetch_data=0
2014-07-15 22:48:15.773288
7fbb85fdb700 10 cache get: name=.users.uid+lakshmi : hit
2014-07-15 22:48:15.773293
7fbb85fdb700 20 get_obj_state: s->obj_tag was set empty
2014-07-15 22:48:15.773297
7fbb85fdb700 10 cache get: name=.users.uid+lakshmi : hit
2014-07-15 22:48:15.774247
7fbb85fdb700 10 get_canon_resource(): dest=http://ceph-gateway.ex.com/
2014-07-15 22:48:15.774326
7fbb85fdb700 10 auth_hdr:
GET
Wed, 16 Jul 2014 05:48:48 GMT
http://ceph-gateway.ex.com/
2014-07-15 22:48:15.775425
7fbb85fdb700 15 calculated digest=k80Z0p3KlwX4TtrZa0Ws0IWCpVU=
2014-07-15 22:48:15.775498
7fbb85fdb700 15 auth_sign=aAd2u8uD1x/FwLAojm+vceWaITY=
2014-07-15 22:48:15.775536
7fbb85fdb700 15 compare=-10
2014-07-15 22:48:15.775603
7fbb85fdb700 10 failed to authorize request
2014-07-15 22:48:15.776202
7fbb85fdb700 2 req 17:0.007071:s3:GET
http://ceph-gateway.ex.com/:list_buckets:http status=403
2014-07-15 22:48:15.776325
7fbb85fdb700 1 == req done
req=0x7fbbe910b290 http_status=403 ==
2014-07-15 22:48:15.776435
7fbb85fdb700 20 process_request() returned -1
Using Swift-Client -
swift --debug -V 1.0 -A http://ceph-gateway.ex.com/auth/1.0 -U ganapati:swift -K
"GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ" list
INFO:urllib3.connectionpool:Starting
new HTTP connection (1): ceph-gateway.ex.com
DEBUG:urllib3.connectionpool:Setting
read timeout to
DEBUG:urllib3.connectionpool:"GET
/auth/1.0 HTTP/1.1" 403 23
('lks: response %s', )
INFO:swiftclient:REQ: curl -i
http://ceph-gateway.ex.com/auth/1.0 -X GET
INFO:swiftclient:RESP STATUS: 403
Forbidden
INFO:swiftclient:RESP HEADERS:
[('date', 'Wed, 16 Jul 2014 05:45:22 GMT'), ('accept-ranges', 'bytes'),
('content-type', 'application/json'), ('content-length', '23'), ('server',
'Apache/2.4.7 (Ubuntu)')]
INFO:swiftclient:RESP BODY:
{"Code":"AccessDenied"}
ERROR:swiftclient:Auth GET failed:
http://ceph-gateway.ex.com/auth/1.0 403 Forbidden
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/swiftclient/client.py",
line 1187, in _retry
self.url, self.token = self.get_auth()
File "/usr/lib/python2.7/dist-packages/swiftclient/client.py",
line 1161, in get_auth
insecure=self.insecure)
File "/usr/lib/python2.7/dist-packages/swiftclient/client.py",
line 324, in
