Re: [ceph-users] S3 Bucket usage up 150% diference between rgw-admin and external metering tools.
On Tue, Jan 21, 2020 at 05:57:51PM +, Robin H. Johnson wrote: > On Mon, Jan 20, 2020 at 12:57:51PM +, EDH - Manuel Rios wrote: > > Hi Cephs > > > > Several nodes of our Ceph 14.2.5 are fully dedicated to host cold storage / > > backups information. > > > > Today checking the data usage with a customer found that rgw-admin is > > reporting: > ... > > That's near 5TB used space in CEPH, and the external tools are reporting > > just 1.42TB. > - What are the external tools? > - How many objects do the external tools report as existing? > - Do the external tools include incomplete multipart uploads in their > size data? > - If bucket versioning is enabled, do the tools include all versions in the > size data? > - Are there leftover multipart pieces without a multipart head? (this > is a Ceph bug that I think is fixed in your release, but old pieces > might still exist). This has been filed as https://tracker.ceph.com/issues/43756 It's definitely lots of leftover MPU bits, as well as inconsistency between MPU heads/parts in the index vs RADOS objects existing: esp. cases with leftover MPU part in RADOS+index but the rest of the MPU is gone. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] S3 Bucket usage up 150% diference between rgw-admin and external metering tools.
On Mon, Jan 20, 2020 at 12:57:51PM +, EDH - Manuel Rios wrote: > Hi Cephs > > Several nodes of our Ceph 14.2.5 are fully dedicated to host cold storage / > backups information. > > Today checking the data usage with a customer found that rgw-admin is > reporting: ... > That's near 5TB used space in CEPH, and the external tools are reporting just > 1.42TB. - What are the external tools? - How many objects do the external tools report as existing? - Do the external tools include incomplete multipart uploads in their size data? - If bucket versioning is enabled, do the tools include all versions in the size data? - Are there leftover multipart pieces without a multipart head? (this is a Ceph bug that I think is fixed in your release, but old pieces might still exist). -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rgw S3 lifecycle cannot keep up
On Wed, Oct 02, 2019 at 01:48:40PM +0200, Christian Pedersen wrote: > Hi Martin, > > Even before adding cold storage on HDD, I had the cluster with SSD only. That > also could not keep up with deleting the files. > I am no where near I/O exhaustion on the SSDs or even the HDDs. Please see my presentation from Cephalic on 2019 about RGW S3 where I touch on slowness in Lifecycle processing and deletion. The efficiency of the code is very low: it requires a full scan of the bucket index every single day. Depending on the traversal order (unordered listing helps), this might mean it takes a very long time to find the items that can be deleted, and even when it gets to them, it's bound by the deletion time, which is also slow (that the head of the objects is a synchronous deletion in many cases, while the tails are async garbage-collected). Fixing this isn't trivial: either you have to scan the entire bucket, or you have to maintain a secondary index in insertion-order for EACH prefix in a lifecycle policy. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Does anybody know whether S3 encryption of Ceph is ready for production?
On Wed, May 15, 2019 at 10:59:38AM +, Guoyong wrote: > Does anybody know whether S3 encryption of Ceph is ready for production? SSE-C I can say I have used & offered in production; I cannot speak for the SSE-S3 & SSE-KMS. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RFC: relicence Ceph LGPL-2.1 code as LGPL-2.1 or LGPL-3.0
On Fri, May 10, 2019 at 02:27:11PM +, Sage Weil wrote: > If you are a Ceph developer who has contributed code to Ceph and object to > this change of license, please let us know, either by replying to this > message or by commenting on that pull request. Am I correct in reading the diff that only a very small number of files did not already have the 'or later' clause of *GPL in effect? As a slight tangent, can we get SPDX tags on files rather than this hard-to-parse text? -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Are there any statistics available on how most production ceph clusters are being used?
On Sun, Apr 21, 2019 at 03:11:44PM +0200, Marc Roos wrote: > Double thanks for the on-topic reply. The other two repsonses, were > making me doubt if my chinese (which I didn't study) is better than my > english. They were almost on topic, but not that useful. Please don't imply language failings on this list. English may be the lingua franca, but it is by far not the first language for most list members. Not being useful to you didn't mean they weren't useful overall. > >> I am a bit curious on how production ceph clusters are being used. I > am > >> reading here that the block storage is used a lot with openstack and > > >> proxmox, and via iscsi with vmare. > >Have you looked at the Ceph User Surveys/Census? > >https://ceph.com/ceph-blog/ceph-user-survey-2018-results/ > >https://ceph.com/geen-categorie/results-from-the-ceph-census/ > > Sort of what I was looking for, so 42% use rgw, of which 74% s3. > I guess this main archive usage, is mostly done by providers Not just archive, but also API-driven for web services, usually hidden behind hostnames/CDNs. Image/video upload sites are a big part of this, esp. things like Instagram clones in emerging markets. > >As the quantity of data by a single user increases, the odds that GUI > >tools are used for it decreases, as it's MUCH more likely to be driven > >by automation & tooling around the API. > Hmm, interesting. I am having more soho clients. And was thinking of > getting them such gui client. That's great, but orthogonal to the overall issue. Some of the cloud providers DO offer setup docs for GUI clients as well, off the top of my head I know Dreamhost & DigitalOcean's ones, because I contributed to their docs: https://help.dreamhost.com/hc/en-us/sections/11559232-DreamObjects-clients https://www.digitalocean.com/docs/spaces/resources/ > I think if you take the perspective of some end user that associates s3, > with something like an audi and nothing else. It is quite necessary > to have a client that is easy and secure to use, where you just enter > preferably only two things, your access key and your secret. There's a bare minimum of three things you'd need in a generic client: - endpoint(s) - access key - secret The Endpoint could be partially pre-provisioned (think like you'd give your clients an INI file that pointed them to your private Ceph RGW deployment). If it's a deployment with multiple regions, endpoints & region-specifics become more important (e.g. AWS S3 has differing signature requirements in different regions) > The advantage of having a more rgw specific gui client, is that you > - do not have the default amazon 'advertisements' (think of storage > classes etc.) > - less configuration options, everything ceph does not support we do not > need to configure. > - no ftp, no what ever else, just this s3 > - you do not have configuration options that ceph doesn't offer > (eg. this life cycle, bucket access logging?) - Storage Classes: supported - Bucket Lifecycle: supported - Bucket Access Logging: not quite supported, PR exists, some debate about better designs. https://github.com/ceph/ceph/pull/14841 > I can imagine if you have quite a few clients, you could get quite > some questions to answer, about things not working. > - you have better support for specific things like multi tenant account, > etc. Tenacy in RGW if effectively parallel S3 scopes; with different endpoints. > - for once the https urls are correctly advertised What issue do you have with HTTPS URLs? The main gotcha that most people hit is that S3's ssl hostname validation rule is NOT the same as the general SSL hostname validation rule, and trips up browser access. Specifically in a wildcard SSL cert, '*.myrgwendpoint.com', the general rule is that '*' should only match one DNS fragment [e.g. no '.'], while S3's validation says it can match one or more DNS fragments. The AWS S3 docs are even horrible about this, with the text: "To work around this, use HTTP or write your own certificate verification logic." https://github.com/awsdocs/amazon-s3-developer-guide/blame/f498926b68f4f1b11c7f708ac0fbd52ee2a0aa19/doc_source/BucketRestrictions.md#L35 > Whether one likes it or not ceph is afaik not fully s3 compatible No, Ceph isn't fully AWS-S3 compatible, and I did specifically include in my talk at Cephalocon last year that we should explicitly be returning 501 NotImplemented in more cases. AWS-S3 in itself is a moving target, and some of the operations ARE best offloaded to something other than Ceph. Even if Ceph/RGW does support a given set of operations, does the deployment want to consider those operations supported? This thinking lead to the torrent ops being behind a configuration option in Ceph, and other ops can be & are blocked by providers in the reverse proxy. There ARE RGW-specific features that would be valuable to have in more clients: - RGW Admin operations [the list of them is much longer than the docs suggest] -
Re: [ceph-users] Are there any statistics available on how most production ceph clusters are being used?
On Fri, Apr 19, 2019 at 12:10:02PM +0200, Marc Roos wrote: > I am a bit curious on how production ceph clusters are being used. I am > reading here that the block storage is used a lot with openstack and > proxmox, and via iscsi with vmare. Have you looked at the Ceph User Surveys/Census? https://ceph.com/ceph-blog/ceph-user-survey-2018-results/ https://ceph.com/geen-categorie/results-from-the-ceph-census/ > But I since nobody here is interested in a better rgw client for end > users. I am wondering if the rgw is even being used like this, and what > most production environments look like. Your end-user client thread was specifically asking targeting GUI clients on OSX & Windows. I feel that the GUI client usage of S3 protocol has a much higher visibility to data size ratio than automation/tooling usage. As the quantity of data by a single user increases, the odds that GUI tools are used for it decreases, as it's MUCH more likely to be driven by automation & tooling around the API. My earliest Ceph production deployment was mostly RGW (~16TB raw), with a little bit of RBD/iSCSI usage (~1TB of floating disk between VMs). Very little of the RGW usage was GUI driven (there certainly was some, because it made business sense to offer it rather than FTP sites; but it tiny compared to the automation flows). My second production deployment I worked was Dreamhost's DreamObjects, which was over 3PB then: and MOST of the usage was still not GUI-driven. I'm working at DigitalOcean's Spaces offering now; again, mostly non-GUI access. For the second part of your original-query, I feel that any new clients SHOULD not be RGW-specific; they should be able to work on a wide range of services that expose the S3 API, and have a good test-suite around that (s3-tests, but for testing the client implementation; even Boto is not bug-free). -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] radosgw cloud sync aws s3 auth failed
On Mon, Apr 08, 2019 at 06:38:59PM +0800, 黄明友 wrote: > > hi,all > >I had test the cloud sync module in radosgw. ceph verion is >13.2.5 , git commit id is >cbff874f9007f1869bfd3821b7e33b2a6ffd4988; Reading src/rgw/rgw_rest_client.cc shows that it only generates v2 signatures for the sync module :-( AWS China regions are some of the v4-only regions. I don't know of any current work to tackle this, but there is v4 signature generation code already in the codebase, would just need to be wired up in src/rgw/rgw_rest_client.cc. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Multicast communication compuverde
On Wed, Feb 06, 2019 at 11:49:28AM +0200, Maged Mokhtar wrote: > It could be used for sending cluster maps or other configuration in a > push model, i believe corosync uses this by default. For use in sending > actual data during write ops, a primary osd can send to its replicas, > they do not have to process all traffic but can listen on specific group > address associated with that pg, which could be an increment from a base > multicast address defined. Some additional erasure codes and > acknowledgment messages need to be added to account for errors/dropped > packets. > i doubt it will give a appreciable boost given most pools use 3 > replicas in total, additionally there could be issues to get multicast > working correctly like setup igmp, so all in all in it could be a > hassle. A separate concern there is that there are too many combinations of OSDs vs multicast limitations in switchgear. As a quick math testcase: Having 3 replicas with 512 OSDs, split over 32 hosts for is ~30k unique host combinations. At at IPv4 protocol layer, this does fit into the 232/8 network for SSM scope or 239/8 LSA scope; in each of those 16.7M multicast addresses. On the switchgear side, even the big Cisco gear, the limits are even lower: 32K. | Output interface lists are stored in the multicast expansion table | (MET). The MET has room for up to 32,000 output interface lists. The | MET resources are shared by both Layer 3 multicast routes and by Layer 2 | multicast entries. The actual number of output interface lists available | in hardware depends on the specific configuration. If the total number | of multicast routes exceed 32,000, multicast packets might not be | switched by the Integrated Switching Engine. They would be forwarded by | the CPU subsystem at much slower speeds. older switchgear was even lower :-(. This would also be a switch from TCP to UDP, and redesign of other pieces, including CephX security. I'm not convinced of the overall gain at this scale for actual data. For heartbeat and other cluster-wide stuff, yes, I do agree that multicast might have benefits. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph in OSPF environment
On Sun, Jan 20, 2019 at 09:05:10PM +, Max Krasilnikov wrote: > > Just checking, since it isn't mentioned here: Did you explicitly add > > public_network+cluster_network as empty variables? > > > > Trace the code in the sourcefile I mentioned, specific to your Ceph > > version, as it has changed slightly over the years. > > My config is looks like that for one host: > > [osd] > # keep empty; empty != unset > public network = > cluster network = > public_network_interface = vlo > cluster_network_interface = vlo > cluster_addr = 10.10.200.5 > public_addr = 10.10.200.5 If you tell the daemon to dump the config, does it still show these set as you have in the config? 'ceph daemon osd.0 config show' -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph in OSPF environment
On Sun, Jan 20, 2019 at 08:54:57PM +, Max Krasilnikov wrote: > День добрий! > > Fri, Jan 18, 2019 at 11:02:51PM +, robbat2 wrote: > > > On Fri, Jan 18, 2019 at 12:21:07PM +, Max Krasilnikov wrote: > > > Dear colleagues, > > > > > > we build L3 topology for use with CEPH, which is based on OSPF routing > > > between Loopbacks, in order to get reliable and ECMPed topology, like > > > this: > > ... > > > CEPH configured in the way > > You have a minor misconfiguration, but I've had trouble with the address > > picking logic before, on a L3 routed ECMP BGP topography on IPv6 (using > > the Cumulus magic link-local IPv6 BGP) > > > > > > > > [global] > > > public_network = 10.10.200.0/24 > > Keep this, but see below. > > > > > [osd.0] > > > public bind addr = 10.10.200.5 > > public_bind_addr is only used by mons. > > > > > cluster bind addr = 10.10.200.5 > > There is no such option as 'cluster_bind_addr'; it's just 'cluster_addr' > > > > Set the following in the OSD block: > > | public_network = # keep empty; empty != unset > > | cluster_network = # keep empty; empty != unset > > | cluster_addr = 10.10.200.5 > > | public_addr = 10.10.200.5 > > > > Alternatively, see the code src/common/pick_address.cc to see about > > using cluster_network_interface and public_network_interface. > > Unfortunatelly, all osds continue to bind to interface addresses instead of > vlo > bridge address even after setting cluster_addr, public_addr, > cluster_network_interface and public_network_interface :( Just checking, since it isn't mentioned here: Did you explicitly add public_network+cluster_network as empty variables? Trace the code in the sourcefile I mentioned, specific to your Ceph version, as it has changed slightly over the years. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph in OSPF environment
On Fri, Jan 18, 2019 at 12:21:07PM +, Max Krasilnikov wrote: > Dear colleagues, > > we build L3 topology for use with CEPH, which is based on OSPF routing > between Loopbacks, in order to get reliable and ECMPed topology, like this: ... > CEPH configured in the way You have a minor misconfiguration, but I've had trouble with the address picking logic before, on a L3 routed ECMP BGP topography on IPv6 (using the Cumulus magic link-local IPv6 BGP) > > [global] > public_network = 10.10.200.0/24 Keep this, but see below. > [osd.0] > public bind addr = 10.10.200.5 public_bind_addr is only used by mons. > cluster bind addr = 10.10.200.5 There is no such option as 'cluster_bind_addr'; it's just 'cluster_addr' Set the following in the OSD block: | public_network = # keep empty; empty != unset | cluster_network = # keep empty; empty != unset | cluster_addr = 10.10.200.5 | public_addr = 10.10.200.5 Alternatively, see the code src/common/pick_address.cc to see about using cluster_network_interface and public_network_interface. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-users Digest, Vol 70, Issue 23
On Sun, Nov 25, 2018 at 07:43:30AM +0700, Lazuardi Nasution wrote: > Hi Robin, > > Do you mean that Cumulus quagga fork is FRRouting (https://frrouting.org/)? > As long as I know Cumulus using it now. I started this before Cumulus was fully shipping FRRouting; and used their binaries. Earlier versions of this: https://docs.cumulusnetworks.com/display/ROH/Installing+the+Cumulus+Quagga+Package+on+a+Host+Server Should be entirely possible w/ FRRouting now. VRFs as well for load balancers is a huge help. > What dummy interfaces do you mean? modprobe dummy > Why did you use it instead of loopback address? Some applications (e.g. keepalived & BIRD) have hardcoded special behavior for the 'lo' interface, with no easy way to work around that behavior. Using Dummy prevents said special behavior. This isn't directly relevant to Ceph itself, but matters on load balancers RGW workloads for example. > Anyway, how can you isolate between some kind of traffic to be not routable? > On L2 implementation, I separate two Ceph traffic (public and cluster) > with other traffic by using VLANs. In my past deployment, the cluster network was using IPv6 ULA addressing to prevent leaks. QoS via DSCP as well to prioritize cluster traffic over public traffic (Ceph already marks the heartbeat packets suitably). -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Full L3 Ceph
On Fri, Nov 23, 2018 at 04:03:25AM +0700, Lazuardi Nasution wrote: > I'm looking example Ceph configuration and topology on full layer 3 > networking deployment. Maybe all daemons can use loopback alias address in > this case. But how to set cluster network and public network configuration, > using supernet? I think using loopback alias address can prevent the > daemons down due to physical interfaces disconnection and can load balance > traffic between physical interfaces without interfaces bonding, but with > ECMP. I can say I've done something similar**, but I don't have access to that environment or most*** of the configuration anymore. One of the parts I do recall, was explicitly setting cluster_network and public_network to empty strings, AND using public_addr+cluster_addr instead, with routable addressing on dummy interfaces (NOT loopback). **:For values of similar: - 99.9% IPv6 environment - BGP everywhere - The only IPv4 was on the outside of HAProxy for legacy IPv4 clients. - Quanta switchgear running Cumulus Linux, 10Gbit ports - Hosts running Cumulus quagga fork (REQUIRED) - Host to 2xToR using IPv6 link-local addressing only https://blog.ipspace.net/2015/02/bgp-configuration-made-simple-with.html - Reliable ~19Gbit aggregate (2x10GBit) - watch out for NIC overheating: no warning, just thermal throttle down to ~2.5Gbit/port. ***:Some parts of the configuration ARE public: https://github.com/dreamhost/ceph-chef/tree/dokken -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] "rgw relaxed s3 bucket names" and underscores
On Tue, Oct 02, 2018 at 12:37:02PM -0400, Ryan Leimenstoll wrote: > I was hoping to get some clarification on what "rgw relaxed s3 bucket > names = false” is intended to filter. Yes, it SHOULD have caught this case, but does not. Are you sure it rejects the uppercase? My test also showed that it did NOT reject the uppercase as intended. This code did used to work, I contributed to the logic and discussion for earlier versions. A related part I wanted was allowing access to existing buckets w/ relaxed names, but disallowing creating of relaxed names. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] radosgw rest API to retrive rgw log entries
On Fri, Sep 21, 2018 at 04:17:35PM -0400, Jin Mao wrote: > I am looking for an API equivalent of 'radosgw-admin log list' and > 'radosgw-admin log show'. Existing /usage API only reports bucket level > numbers like 'radosgw-admin usage show' does. Does anyone know if this is > possible from rest API? /admin/log is the endpoint you want. params: REQUIRED: type=(metadata|bucket-index|data) The API is a little inconsistent. metadata & data default to an global info operation, and need an 'id' argument for listing (also if both 'info' & 'id' are passed, you get ShardInfo). bucket-index defaults to listing, but responds to the 'info' argument with info response. All types support the status argument as well. The complete list of /admin/ resources as of Luminous: /admin/usage /admin/user /admin/bucket /admin/metadata /admin/log /admin/opstat /admin/replica_log /admin/config /admin/realm -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph User Survey 2018
On Mon, Apr 30, 2018 at 11:39:11PM -0300, Leonardo Vaz wrote: > Hey Cephers! > > We just announced the 2018 edition of Ceph user Survey: > > https://www.surveymonkey.com/r/ceph2018 > > It will be accepting answers until May 15th and the results will be > published on the project website. > > Please consider to contribute, your feedback is very important to us! Some of the wording needs improvement. First you ask how many clusters, then you ask questions about a cluster singular (capacity, servers, osds). Should the latter be total over all clusters, or of some specific cluster? -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Purged a pool, buckets remain
On Tue, Apr 10, 2018 at 10:06:57PM -0500, Robert Stanford wrote: > I used this command to purge my rgw data: > > rados purge default.rgw.buckets.data --yes-i-really-really-mean-it > > Now, when I list the buckets with s3cmd, I still see the buckets (s3cmd ls > shows a listing of them.) When I try to delete one (s3cmd rb) I get this: ... > I thought maybe the names were sticking around in > default.rgw.buckets.index, so I purged that too. But no luck, the phantom > buckets are still there. The list of buckets is in the OMAP of the users. But as the others said, this was not a good way to go about trying to delete the data. The only case I can see is if you were playing around and wanted to completely stop using RGW in an existing cluster, and do CephFS or RBD instead. If you did want want to completely get rid of RGW data, you should wipe out ALL of the RGW pools, not just the data pool. "radosgw-admin zone get" will show them to you. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] change radosgw object owner
On Tue, Mar 06, 2018 at 02:40:11PM -0500, Ryan Leimenstoll wrote:
> Hi all,
>
> We are trying to move a bucket in radosgw from one user to another in an
> effort both change ownership and attribute the storage usage of the data to
> the receiving user’s quota.
>
> I have unlinked the bucket and linked it to the new user using:
>
> radosgw-admin bucket unlink —bucket=$MYBUCKET —uid=$USER
> radosgw-admin bucket link —bucket=$MYBUCKET —bucket-id=$BUCKET_ID
> —uid=$NEWUSER
>
> However, perhaps as expected, the owner of all the objects in the
> bucket remain as $USER. I don’t believe changing the owner is a
> supported operation from the S3 protocol, however it would be very
> helpful to have the ability to do this on the radosgw backend. This is
> especially useful for large buckets/datasets where copying the objects
> out and into radosgw could be time consuming.
At the raw radosgw-admin level, you should be able to do it with
bi-list/bi-get/bi-put. The downside here is that I don't think the BI ops are
exposed in the HTTP Admin API, so it's going to be really expensive to chown
lots of objects.
Using a quick example:
# radosgw-admin \
--uid UID-CENSORED \
--bucket BUCKET-CENSORED \
bi get \
--object=OBJECTNAME-CENSORED
{
"type": "plain",
"idx": "OBJECTNAME-CENSORED",
"entry": {
"name": "OBJECTNAME-CENSORED",
"instance": "",
"ver": {
"pool": 5,
"epoch": 266028
},
"locator": "",
"exists": "true",
"meta": {
"category": 1,
"size": 1066,
"mtime": "2016-11-17 17:01:29.668746Z",
"etag": "e7a75c39df3d123c716d5351059ad2d9",
"owner": "UID-CENSORED",
"owner_display_name": "UID-CENSORED",
"content_type": "image/png",
"accounted_size": 1066,
"user_data": ""
},
"tag": "default.293024600.1188196",
"flags": 0,
"pending_map": [],
"versioned_epoch": 0
}
}
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
signature.asc
Description: Digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Developer Monthly - March 2018
On Wed, Feb 28, 2018 at 10:51:29PM +, Sage Weil wrote: > On Wed, 28 Feb 2018, Dan Mick wrote: > > Would anyone else appreciate a Google Calendar invitation for the CDMs? > > Seems like a natural. > > Funny you should mention it! I was just talking to Leo this morning about > creating a public Ceph Events calendar that has all of the public events > (CDM, tech talks, weekly perf call, etc.). > > (Also, we're setting up a Ceph Meetings calendar for meetings that aren't > completely public that can be shared with active developers for standing > meetings that are currently invite-only meetings. e.g., standups, > advisory board, etc.) Yes please on the calendars! -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] identifying public buckets
On Wed, Feb 21, 2018 at 10:19:58AM +, Dave Holland wrote:
> Hi,
>
> We would like to scan our users' buckets to identify those which are
> publicly-accessible, to avoid potential embarrassment (or worse), e.g.
> http://www.bbc.co.uk/news/technology-42839462
>
> I didn't find a way to use radosgw-admin to report ACL information for a
> given bucket. And using the API to query a bucket's information would
> require a valid access key for that bucket. What am I missing, please?
You can do it via the S3 API. The below in Luminous, but should work fine in
Jewel (might have to force AWS-CLI to use a v2 signature).
You need to create a RGW user with the system flag set (it might be
possible with the newer admin flag as well).
As a concrete example, using Amazon's awscli, here:
# set the system bit on a user, if you don't already have a user with
# this power.
$ radosgw-admin user modify --uid $UID --system
# use the access+secret key from the above user.
$ AWS_ACCESS_KEY_ID='...' AWS_SECRET_ACCESS_KEY='...' \
aws \
--endpoint-url=https://$ENDPOINT \
s3api get-bucket-acl \
--bucket $BUCKETNAME
Example output (censored):
{
"Owner": {
"DisplayName": "ANOTHER-USER-THAT-WAS-NOT-SYSTEM",
"ID": "ANOTHER-USER-THAT-WAS-NOT-SYSTEM"
},
"Grants": [
{
"Grantee": {
"Type": "CanonicalUser",
"DisplayName": "ANOTHER-USER-THAT-WAS-NOT-SYSTEM",
"ID": "ANOTHER-USER-THAT-WAS-NOT-SYSTEM"
},
"Permission": "FULL_CONTROL"
}
]
}
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
signature.asc
Description: Digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rgw bucket inaccessible - appears to be using incorrect index pool?
On Mon, Feb 19, 2018 at 07:57:18PM -0600, Graham Allan wrote: > Sorry to send another long followup, but actually... I'm not sure how to > change the placement_rule for a bucket - or at least what I tried does > not seem to work. Using a different (more disposable) bucket, my attempt > went like this:: [snip] > first created a new placement rule "old-placement" in both the zonegroup > and zone commit new period - this looks ok. ... > I edit "placement_rule": to change "" -> "old-placement" and write it > back using: > > > radosgw-admin metadata put bucket.instance:boto-demo-100:default.2170793.10 > > < boto-demo-100.json > > Now when I run "radosgw-admin bucket list --bucket=boto-demo-100" I am > getting an empty list, though I'm pretty sure the bucket contains some > objects. > > When I read the bucket instance metadata back again, it still reads > "placement_rule": "" so I wonder if the bucket_info change is really > taking effect. So it never showed the new placement_rule if you did a get after the put? > A quick debug session seems to show it still querying the wrong pool > (100) for the index, so it seems that my attempt to update the > bucket_info is either failing or incorrect! Did you run a local build w/ the linked patch? I think that would have more effect than -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rgw bucket inaccessible - appears to be using incorrect index pool?
On Fri, Feb 16, 2018 at 07:06:21PM -0600, Graham Allan wrote: [snip great debugging] This seems similar to two open issues, could be either of them depending on how old that bucket is. http://tracker.ceph.com/issues/22756 http://tracker.ceph.com/issues/22928 - I have a mitigation posted to 22756. - There's a PR posted for 22928, but it'll probably only be in v12.2.4. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Broken Buckets after Jewel->Luminous Upgrade
On Tue, Jan 30, 2018 at 10:32:04AM +0100, Ingo Reimann wrote: > What could be the problem,and how may I solve that? For anybody else tracking this, the logs & debugging info are filed at http://tracker.ceph.com/issues/22928 -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Broken Buckets after Jewel->Luminous Upgrade
On Wed, Jan 31, 2018 at 07:39:02AM +0100, Ingo Reimann wrote: > Hi Robin, > > thanks for your reply. > > Concerning "https://tracker.ceph.com/issues/22756 - buckets showing as > empty": Our cluster is rather old - argonaut, but the affected bucket and > user are created under jewel. > > If you need more data, I may post it. Yes, please, I'm even more interested now, as I only have customer reports from old buckets! I'm wondering if it's slightly related to issue 22714 for some other object that starts with an underscore. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Broken Buckets after Jewel->Luminous Upgrade
On Tue, Jan 30, 2018 at 10:32:04AM +0100, Ingo Reimann wrote: > The problem: > Some Buckets are not accessible from the luminous gateway. The metadata > for that buckets seemed ok, but listing was not possible. A local s3cmd > got "404 NoSuchKey". I exported and imported the metadata for one instance > and ran radosgw-admin --check. Now the bucket is listable but empty under > luminous and broken under jewel. The corresponding directory object still > contains the file in its omap. > > I am afraid to corrupt my cluster so I stopped the upgrade for the other > gateways. I have a couple of bugs open for possible the same issue: https://tracker.ceph.com/issues/22756 - buckets showing as empty http://tracker.ceph.com/issues/22714 - old AccessKeys not working One more to come after more diagnosis my side, where some old files don't work properly anymore (dropping off at a multiple of 512K) -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] RGW CreateBucket: AWS vs RGW, 200/409 responses
On Mon, Dec 25, 2017 at 11:52:36AM +0800, QR wrote: > Is anyone know the reason that ERR_BUCKET_EXISTS is modified to zero? > Thanks. This comes down to arguing about AWS S3 CreateBucket behavior if the bucket already existed and was owned by you (plus which region it is in vs where the request was made to). AWS S3 us-east-1: returns 200 OK AWS S3 other regions: return 409 Conflict: BucketAlreadyOwnedByYou RGW implements the 200-OK behavior, which is why the return is modified to zero (ret==0 -> HTTP 200 OK). I created issue 17398 a year ago to discuss it: http://tracker.ceph.com/issues/17398 I see recently that issue 22279 was created for the same thing, with an attached PR, but the same discussion. http://tracker.ceph.com/issues/22279 https://github.com/ceph/ceph/pull/19249 Yehuda's opinion is that the 200-OK behavior is better, because CreateBucket should be an idempotent operation. My opinion is that the 409-BucketAlreadyOwnedByYou behavior is better, because the CreateBucket annoyingly is actually two operations: (CreateBucket, SetBucketACL). If the bucket does already exist, but the ACLs you pass in bucket creation are NOT the same as the existing bucket ACLs, what should happen? - Skip creation, skip ACLs? 200-OK response is dangerously misleading here. - Skip creation, replace ACLs? This is going to overwrite the existing ACLs, rather than modify them. - Do nothing and issue a warning about ACL mismatch? Annoying in that we're diverging more from AWS behavior. AWS S3 went to 409-BucketAlreadyOwnedByYou for a reason. They've never stated exactly why, but I believe their choice makes sense. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RGW Logging pool
On Fri, Dec 15, 2017 at 05:21:37PM +, David Turner wrote: > We're trying to build an auditing system for when a user key pair performs > an operation on a bucket (put, delete, creating a bucket, etc) and so far > were only able to find this information in the level 10 debug logging in > the rgw systems logs. > > We noticed that our rgw log pool has been growing somewhat indefinitely and > we had to move it off of the nvme's and put it to HDD's due to it's growing > size. What is in that pool and how can it be accessed? I haven't found > the right terms to search for to find anything about what's in this pool on > the ML or on Google. > > What I would like to do is export the log to ElasticSearch, cleanup the log > on occasion, and hopefully find the information we're looking for to > fulfill our user auditing without having our RGW daemons running on debug > level 10 (which is a lot of logging!). I have a terrible solution in HAProxy's Lua that recognizes most S3 operations and spits out UDP/logs based on that. It's not ideal, has LOTS of drawbacks (mostly in duplication of code, incl S3 signature stuff). I'd be very interested in writing useful log data out either in a difference channel or as part of the HTTP response (key, bucket, object, operation, actual bytes moved [esp for in-place S3 COPY]) -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to remove a faulty bucket?
On Mon, Dec 11, 2017 at 09:29:11AM +, Martin Emrich wrote: > Hi! > > Am 09.12.17, 00:19 schrieb "Robin H. Johnson" <robb...@gentoo.org>: > > If you use 'radosgw-admin bi list', you can get a listing of the raw > bucket > index. I'll bet that the objects aren't being shown at the S3 layer > because something is wrong with them. But since they are in the bi-list, > you'll get 409 BucketNotEmpty. > > Yes indeed. Running "radosgw-admin bi list" results in an incomplete 300MB > JSON file, before it freezes. That's a very good starting point to debug. The bucket index is stored inside the OMAP area of a raw RADOS object. (in a filestore OSD it's in the LevelDB), I wonder if you have corruption or something else awry. How many objects were in this bucket? The number from 'bucket stats' is a good starting point. Newer versions of Jewel do report OMAP inconsistency after deep-scrub, so that would be a help in your case too. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How to remove a faulty bucket?
If you use 'radosgw-admin bi list', you can get a listing of the raw bucket index. I'll bet that the objects aren't being shown at the S3 layer because something is wrong with them. But since they are in the bi-list, you'll get 409 BucketNotEmpty. At this point, I've found two different approaches, depending how much you want to do in rgw vs the S3 APIs. A) S3 APIs: upload new zero-byte files that match all names from the bucket index. Then delete them. B) 'radosgw-admin object unlink'. This got messy with big multipart items. Other things that can stop deletion of buckets that look empty: - open/incomplete multipart uploads: run Abort Multipart Upload on each upload. - bucket subresources (cors, website) [iirc this was a bug that got fixed]. I have not personally played with editing the bi entries in cases like this. There are more drastic ways to delete the entry points into a bucket as well (but it would otherwise leave the mess around). -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Problem making RadosGW dual stack
On Mon, Oct 30, 2017 at 04:42:00PM +, alastair.dewhu...@stfc.ac.uk wrote: > Hello .. > We have tested that individually both the IPv4 and IPv6 works (the > service starts and transfers work), so we believe the problem is with > how ceph parses the port setting. We did consider the possibility of > running two radosgw instances on the same machine, but thought it more > sensible to ask if anyone else has had the same problem and if they > managed to fix it? This looks like but 20942: http://tracker.ceph.com/issues/20942 I worked around the port part by only binding to a single port and doing the rest of the work in the load balancer. Still not possible to use commas in some other args. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Problems with CORS
On Sun, Oct 22, 2017 at 01:31:03PM +, Rudenko Aleksandr wrote:
> In past we rewrite http response header by Apache rules for our
> web-interface and pass CORS check. But now it’s impossible to solve on
> balancer level.
You CAN modify the CORS responses at the load-balancer level.
Find below the snippets needed to do it in HAProxy w/ Jewel-Civetweb;
specifically, this completely overrides the CORS if the Origin matches some
strings.
We use this to override the CORS for access via our customer interface panel,
so regardless of what CORS they set on the bucket, the panel always works.
frontend ...
# Store variable for using later in the response.
http-request set-var(txn.origin) req.hdr(Origin)
acl override_cors var(txn.origin) -m end -i SOMEDOMAIN
acl override_cors var(txn.origin) -m sub -i SOMEDOMAIN
# Export fact as a boolean
http-request set-var(txn.override_cors) bool(true) if override_cors
http-request set-var(txn.override_cors) bool(false) unless override_cors
backend ...
# We inject Origin headers for ..., so we must declare to the client
# that the might be different in other requests.
http-response add-header Vary Origin if { var(txn.origin) -m len gt 1
# If the origin is the Panel, then override the CORS headers
acl override_cors var(txn.override_cors),bool
# 1. if OPTIONS: Override any 403 error to say it's ok instead
# 403 means the OPTIONS request was being denied by the RGW as it didn't
match.
acl res_status_403 status eq 403
http-response set-status 200 if res_status_403 override_cors METH_OPTIONS
# 2. if OPTIONS: Valid CORS has length 0
http-response set-header Content-Length 0 if override_cors METH_OPTIONS
# 3. Override CORS headers
http-response set-header Access-Control-Allow-Headers origin,\
content-length,\ content-type,\ content-md5 if override_cors
http-response set-header Access-Control-Allow-Origin %[var(txn.origin)] if
override_cors
http-response set-header Access-Control-Allow-Methods GET,\ HEAD,\ POST,\
OPTIONS,\ PUT,\ DELETE if override_cors
http-response set-header Access-Control-Max-Age 600 if override_cors
>
> What is right way?
>
> ---
> Best regards,
>
> Aleksandr Rudenko
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
signature.asc
Description: Digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Significant uptick in inconsistent pgs in Jewel 10.2.9
On Thu, Sep 07, 2017 at 08:24:04PM +, Robin H. Johnson wrote:
> pg 5.3d40 is active+clean+inconsistent, acting [1322,990,655]
> pg 5.f1c0 is active+clean+inconsistent, acting [631,1327,91]
Here is the output of 'rados list-inconsistent-obj' for the PGs:
$ sudo rados list-inconsistent-obj 5.f1c0 |json_pp -json_opt canonical,pretty
{
"epoch" : 1221254,
"inconsistents" : [
{
"errors" : [
"omap_digest_mismatch"
],
"object" : {
"locator" : "",
"name" : ".dir.default.292886573.13181.12",
"nspace" : "",
"snap" : "head",
"version" : 483490
},
"selected_object_info" :
"5:038f1cff:::.dir.default.292886573.13181.12:head(1221843'483490
client.417313345.0:19515832 dirty|omap|data_digest s 0 uv 483490 dd
alloc_hint [0 0])",
"shards" : [
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x928b0c0b",
"osd" : 91,
"size" : 0
},
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x928b0c0b",
"osd" : 631,
"size" : 0
},
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x6556c868",
"osd" : 1327,
"size" : 0
}
],
"union_shard_errors" : []
}
]
}
$ sudo rados list-inconsistent-obj 5.3d40 |json_pp -json_opt canonical,pretty
{
"epoch" : 1210895,
"inconsistents" : [
{
"errors" : [
"omap_digest_mismatch"
],
"object" : {
"locator" : "",
"name" : ".dir.default.64449186.344176",
"nspace" : "",
"snap" : "head",
"version" : 1177199
},
"selected_object_info" :
"5:02bc4def:::.dir.default.64449186.344176:head(1177700'1180639
osd.1322.0:537914 dirty|omap|data_digest|omap_digest s 0 uv 1177199 dd
od 337cf025 alloc_hint [0 0])",
"shards" : [
{
"data_digest" : "0x",
"errors" : [
"omap_digest_mismatch_oi"
],
"omap_digest" : "0x3242b04e",
"osd" : 655,
"size" : 0
},
{
"data_digest" : "0x",
"errors" : [],
"omap_digest" : "0x337cf025",
"osd" : 990,
"size" : 0
},
{
"data_digest" : "0x",
"errors" : [
"omap_digest_mismatch_oi"
],
"omap_digest" : "0xc90d06a8",
"osd" : 1322,
"size" : 0
}
],
"union_shard_errors" : [
"omap_digest_mismatch_oi"
]
}
]
}
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
signature.asc
Description: Digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Significant uptick in inconsistent pgs in Jewel 10.2.9
Hi, Our clusters were upgraded to v10.2.9, from ~v10.2.7 (actually a local git snapshot that was not quite 10.2.7), and since then, we're seeing a LOT more scrub errors than previously. The digest logging on the scrub errors, in some cases, is also now maddeningly short: it doesn't contain ANY information on what the mismatch was, and many of the errors seem to also be 3-way mismatches in the digest :-(. I'm wondering if other people have seen something similar rises in scrub errors in the upgrade, and/or the lack of digest output. I did hear one anecdotal report that 10.2.9 seemed much more likely to fail out marginal disks. The only two changesets I can spot in Jewel that I think might be related are these: 1. http://tracker.ceph.com/issues/20089 https://github.com/ceph/ceph/pull/15416 2. http://tracker.ceph.com/issues/19404 https://github.com/ceph/ceph/pull/14204 Two example PGs that are inconsistent (chosen because they didn't convey any private information so I didn't have to redact anything except IP): $ sudo ceph health detail |grep -e 5.3d40 -e 5.f1c0 pg 5.3d40 is active+clean+inconsistent, acting [1322,990,655] pg 5.f1c0 is active+clean+inconsistent, acting [631,1327,91] $ fgrep 5.3d40 /var/log/ceph/ceph.log 2017-09-07 19:50:16.231523 osd.1322 [REDACTED::8861]:6808/3479303 1736 : cluster [INF] osd.1322 pg 5.3d40 Deep scrub errors, upgrading scrub to deep-scrub 2017-09-07 19:50:16.231862 osd.1322 [REDACTED::8861]:6808/3479303 1737 : cluster [INF] 5.3d40 deep-scrub starts 2017-09-07 19:54:38.631232 osd.1322 [REDACTED::8861]:6808/3479303 1738 : cluster [ERR] 5.3d40 shard 655: soid 5:02bc4def:::.dir.default.64449186.344176:head omap_digest 0x3242b04e != omap_digest 0x337cf025 from auth oi 5:02bc4def:::.dir.default.64449186.344176:head(1177700'1180639 osd.1322.0:537914 dirty|omap|data_digest|omap_digest s 0 uv 1177199 dd od 337cf025 alloc_hint [0 0]) 2017-09-07 19:54:38.631332 osd.1322 [REDACTED::8861]:6808/3479303 1739 : cluster [ERR] 5.3d40 shard 1322: soid 5:02bc4def:::.dir.default.64449186.344176:head omap_digest 0xc90d06a8 != omap_digest 0x3242b04e from shard 655, omap_digest 0xc90d06a8 != omap_digest 0x337cf025 from auth oi 5:02bc4def:::.dir.default.64449186.344176:head(1177700'1180639 osd.1322.0:537914 dirty|omap|data_digest|omap_digest s 0 uv 1177199 dd od 337cf025 alloc_hint [0 0]) 2017-09-07 20:03:54.721681 osd.1322 [REDACTED::8861]:6808/3479303 1740 : cluster [ERR] 5.3d40 deep-scrub 0 missing, 1 inconsistent objects 2017-09-07 20:03:54.721687 osd.1322 [REDACTED::8861]:6808/3479303 1741 : cluster [ERR] 5.3d40 deep-scrub 3 errors $ fgrep 5.f1c0 /var/log/ceph/ceph.log 2017-09-07 11:11:36.773986 osd.631 [REDACTED::8877]:6813/4036028 4234 : cluster [INF] osd.631 pg 5.f1c0 Deep scrub errors, upgrading scrub to deep-scrub 2017-09-07 11:11:36.774127 osd.631 [REDACTED::8877]:6813/4036028 4235 : cluster [INF] 5.f1c0 deep-scrub starts 2017-09-07 11:25:26.231502 osd.631 [REDACTED::8877]:6813/4036028 4236 : cluster [ERR] 5.f1c0 deep-scrub 0 missing, 1 inconsistent objects 2017-09-07 11:25:26.231508 osd.631 [REDACTED::8877]:6813/4036028 4237 : cluster [ERR] 5.f1c0 deep-scrub 1 errors -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RadosGW ADMIN API
On Wed, Sep 06, 2017 at 02:08:14PM +, Engelmann Florian wrote:
> we are running a luminous cluster and three radosgw to serve a s3 compatible
> objectstore. As we are (currently) not using Openstack we have to use the
> RadosGW Admin API to get our billing data. I tried to access the API with
> pathon like:
>
> [...]
> import rgwadmin
> [...]
> Users = radosgw.get_users()
> [...]
>
> But I get a 403 "AccessDenied" using python 2.7.13.
>
> What's the easiest method to access the Admin API from a remote host?
You can have a look at why it's generating the 403, if you increase the
debug level of rgw & civetweb.
The user associated with the access key & secret key tuple you're using
DOES need to have user capabilities for reading users.
$ sudo radosgw-admin metadata get user:MYADMINUSER-REDACTED
{
"key": "user:MYADMINUSER-REDACTED",
...
"data": {
"user_id": "MYADMINUSER-REDACTED",
"display_name": "MYADMINUSER-REDACTED",
...,
"caps": [
{
"type": "buckets",
"perm": "read"
},
{
"type": "usage",
"perm": "read"
},
{
"type": "users",
"perm": "*"
}
],
"system": "true",
...,
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
signature.asc
Description: Digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] jewel - radosgw-admin bucket limit check broken?
I just hit this too, and found it was fixed in master, so generated a backport issue & PR: http://tracker.ceph.com/issues/20966 https://github.com/ceph/ceph/pull/16952 -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] autoconfigured haproxy service?
(Trim lots of good related content). The upcoming HAProxy 1.8 has landed further patches for improving hot restarts/reloads of HAProxy, which previously lead to a brief gap period when new connections were not serviced. Lots of other approaches had been seen, including delaying TCP SYN momentarily [1]. It also notably fixes how you needed a separate wrapper for using haproxy w/ systemd. In the DreamHost RGW usage, I've been building additional functionality into HAProxy via Lua, but I'm not convinced it's the best place for some of it: - rate-limiting by access key (with explicitly whitelisted & blacklisted keys), over multiple frontends [2] - Planned: different rates for different operations/customer classes. - Planned: issue redirects for buckets being moved (because getting federated metadata between existing clusters is painful) [3]. This differs from the CERN backend-selection-by-bucket, because the bucket is going to be moving regions! RGW doesn't seem like the right place for some of this functionality either, because storing rate data in multiple places means lots of cross-talk. [1] https://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html [2] http://blog.armbruster-it.de/2015/08/neo4j-and-haproxy-some-best-practices-and-tricks/ [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/Redirects.html > > I don't know if the bucket mapping concept is generally applicable. > > Maybe this haproxy-agent should focus on configuring a single backend > > populated with the radosgw's, and leave more complex configurations up > > to their admins? > Yeah. (The dynamic remapping is interesting, though! That could > potentially be controlled by rgw as well to automatically isolate busy > buckets or objects.) Yes, this would fit well to populate a single HAProxy backend w/ the available RGWs, and then that can be wrapped with whatever frontend configuration (esp. SSL) that the admin wants. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] radosgw refuses upload when Content-Type missing from POST policy
On Wed, May 31, 2017 at 05:02:14PM +0100, Dave Holland wrote: > I put a radosgw debug=20 log of the successful OPTIONS call and failing > POST call here: > https://docs.google.com/document/d/1i3exJSil1xj14ZrDOF_oM9eZC238gnNVAsnaZ-Pkvzo/edit?usp=sharing > > Happy to provide other debug info if necessary. Can you share the testcase and a capture on the user-side of it running? (tcpdump showing the HTTP headers & complete payload for request+response would be SUPERB). BTW, the POST policy is entirely unrelated to CORS policy, but if the request is cross-domain, the rest of the CORS policy will need to permit the POST to make the browser happy. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large OSD omap directories (LevelDBs)
On Fri, May 19, 2017 at 01:55:50PM +, george.vasilaka...@stfc.ac.uk wrote: > Anyone seen this before who can point me in the right direction to start > digging? Your RGW buckets, how many objects in them, and do they have the index sharded? I know we have some very large & old buckets (10M+ RGW objects in a single bucket), with correspondingly large OMAPs wherever that bucket index is living (sufficently large that trying to list the entire thing online is fruitless). ceph's pgmap status says we have 2G RADOS objects however, and you're only at 61M RADOS objects. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Limit bandwidth on RadosGW?
On Thu, May 04, 2017 at 04:35:21PM +0800, hrchu wrote: > Thanks for reply. > > tc can only do limit on interfaces or given IPs, but what I am talking > about is "per connection", e.g., each put object could be 5MB/s, get > object could be 1MB/s. To achieve your required level of control, you need haproxy, or other HTTP-aware reverse proxy, as to have a different limit based on the operation (and possibly the access key). -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph Cluster Failures
On Thu, Mar 16, 2017 at 02:22:08AM +, Rich Rocque wrote:
> Has anyone else run into this or have any suggestions on how to remedy it?
We need a LOT more info.
> After a couple months of almost no issues, our Ceph cluster has
> started to have frequent failures. Just this week it's failed about
> three times.
>
> The issue appears to be than an MDS or Monitor will fail and then all
> clients hang. After that, all clients need to be forcibly restarted.
- Can you define monitor 'failing' in this case?
- What do the logs contain?
- Is it running out of memory?
- Can you turn up the debug level?
- Has your cluster experienced continual growth and now might be
undersized in some regard?
> The architecture for our setup is:
Are these virtual machines? The overall specs seem rather like VM
instances rather than hardware.
> 3 ea MON, MDS instances (co-located) on 2cpu, 4GB RAM servers
What sort of SSD are the monitor datastores on? ('mon data' in the
config)
> 12 ea OSDs (ssd), on 1cpu, 1GB RAM servers
12 SSDs to a single server, with 1cpu/1GB RAM? That's absurdly low-spec.
How many OSD servers, what SSDs?
What is the network setup & connectivity between them (hopefully
10Gbit).
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] object store backup tool recommendations
On Fri, Mar 03, 2017 at 10:55:06AM +1100, Blair Bethwaite wrote: > Does anyone have any recommendations for good tools to perform > file-system/tree backups and restores to/from a RGW object store (Swift or > S3 APIs)? Happy to hear about both FOSS and commercial options please. This isn't Ceph specific, but is something that has come up for me, and I did a lot of research into it for the Gentoo distribution to use on it's infrastructure. The wiki page with all of our needs & contenders is here: https://wiki.gentoo.org/wiki/Project:Infrastructure/Backups_v3 TL;DR: restic is probably the closest fit to your needs, but do evaluate it carefully. > I'm interested in: > 1) tools known to work or not work at all for a basic file-based data backup > > Plus these extras: > 2) preserves/restores correct file metadata (e.g. owner, group, acls etc) > 3) preserves/restores xattrs Restic has acl/xattr in master, but not yet in version 0.4.0. > 4) backs up empty directories and files Yes. > 5) supports some sort of snapshot/versioning/differential functionality, > i.e., will keep a copy or diff or last N versions of a file or whole backup > set, e.g., so that one can restore yesterday's file/s or last week's but > not have to keep two full copies to achieve it Yes. > 6) is readily able to restore individual files Yes. > 7) can encrypt/decrypt client side Yes, but beware the key model, it's fully symmetric, any client with the key can touch the entire repo. > 8) anything else I should be considering restic does not do any compression, it's planned still. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Random Health_warn
On Thu, Feb 23, 2017 at 10:40:31PM +, Scottix wrote: > Ya the ceph-mon.$ID.log > > I was running ceph -w when one of them occurred too and it never output > anything. > > Here is a snippet for the the 5:11AM occurrence. Yep, I don't see anything in there that should have triggered HEALTH_WARN. All I can suggest is dumping the JSON health blob when it occurs again, and seeing if anything stands out in it. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Random Health_warn
On Thu, Feb 23, 2017 at 09:49:21PM +, Scottix wrote: > ceph version 10.2.5 (c461ee19ecbc0c5c330aca20f7392c9a00730367) > > We are seeing a weird behavior or not sure how to diagnose what could be > going on. We started monitoring the overall_status from the json query and > every once in a while we would get a HEALTH_WARN for a minute or two. > > Monitoring logs. > 02/23/2017 07:25:54 AM HEALTH_OK > 02/23/2017 07:24:54 AM HEALTH_WARN > 02/23/2017 07:23:55 AM HEALTH_OK > 02/23/2017 07:22:54 AM HEALTH_OK > ... > 02/23/2017 05:13:55 AM HEALTH_OK > 02/23/2017 05:12:54 AM HEALTH_WARN > 02/23/2017 05:11:54 AM HEALTH_WARN > 02/23/2017 05:10:54 AM HEALTH_OK > 02/23/2017 05:09:54 AM HEALTH_OK > > When I check the mon leader logs there is no indication of an error or > issues that could be occuring. Is there a way to find what is causing the > HEALTH_WARN? By leader logs, do you mean the cluster log (mon_cluster_log_file), or the mon log (log_file)? Eg /var/log/ceph/ceph.log vs /var/log/ceph/ceph-mon.$ID.log. Could you post the log entries for a time period between two HEALTH_OK states with a HEALTH_WARN in the middle? The reason for WARN _should_ be included on the logged status line. Alternatively, you should be able to just log the output of 'ceph -w' for a while, and find the WARN status as well. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rgw static website docs 404
On Fri, Jan 20, 2017 at 11:37:47AM +0100, Wido den Hollander wrote: > Maybe the dev didn't want to write docs, he/she forgot or just didn't get to > it yet. > > It would be very much appreciated if you would send a PR with the updated > documentation :) As the dev, I did write docs, and have posted them here before. RedHat wrote their docs based on my work. It was very unclear on how best to fit my docs into the existing documentation, because a LOT of it isn't about how to set up Ceph, but rather how to set up the OTHER pieces you need, as well as the tradeoffs that you must make if you don't run the recommended setup of two separate RGW instances, each on their own unique public IP. https://gist.github.com/robbat2/ec0a66eed28e5f0e1ef7018e9c77910c -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Rsync to object store
On Wed, Dec 28, 2016 at 09:31:57PM +0100, Marc Roos wrote: > Is it possible to rsync to the ceph object store with something like > this tool of amazon? > https://aws.amazon.com/customerapps/1771 That's a service built on top of AWS EC2 that just happens to back storage into AWS S3. There's no fundamental reason it couldn't support Ceph RGW S3, but you'd need to contact the service provider and work out the details with them (like running their service close to your RGW instances). -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] rgw / s3website, MethodNotAllowed on Jewel 10.2.3
On Wed, Oct 26, 2016 at 11:43:15AM +0200, Trygve Vea wrote: > Hi! > > I'm trying to get s3website working on one of our Rados Gateway > installations, and I'm having some problems finding out what needs to > be done for this to work. It looks like this is a halfway secret > feature, as I can only find it briefly mentioned in the release notes > for v10.0.4 - and nowhere in the documentation - so I've tried to wrap > my head around this by looking through the source code without much > luck. Here's the documention I wrote for people to use StaticSites: https://gist.github.com/robbat2/ec0a66eed28e5f0e1ef7018e9c77910c -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] S3 API - Canonical user ID
On Mon, Jul 18, 2016 at 10:48:16AM +0300, Victor Efimov wrote: > xmlns="http://s3.amazonaws.com/doc/2006-03-01/;>someownerSOMEOWNER > > note the "someowner" is used as id. > Problem that S3-compatible library that I use crashes on this, it > expects 64 character hex string. > > According to S3 docs owner id indeed is 64 char hex string: > http://docs.aws.amazon.com/general/latest/gr/acct-identifiers.html > === > The canonical user ID is a long string, such as > 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be. > === It's an example where it happens to be a 64-hex string, not a specification that says it MUST be a 64-hex string. The specification implied by this page is simply it's a long string (and opaque). > How to qualify this? > 3) Bug in library which should not validate this field that way? This is definitely a library bug. It should not be making assumptions about the nature or formatting of the ID string. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 403 AccessDenied with presigned url in Jewel AWS4.
On Fri, Jun 03, 2016 at 11:34:35AM +0700, Khang Nguyễn Nhật wrote: > s3 = boto3.client(service_name='s3', region_name='', use_ssl=False, > endpoint_url='http://192.168.1.10:', aws_access_key_id=access_key, > aws_secret_access_key= secret_key, > config=Config(signature_version='s3v4', region_name='')) The region part doesn't seem right. Try setting it to 'ap' or 'ap-southeast'. Failing that, turn up the RGW loglevel to 20, and run a request, then look at the logs of how it created the signature, and manually compare them to what your client should have built (with boto in verbose debugging). -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Deploying ceph by hand: a few omissions
On Sun, May 01, 2016 at 08:46:36PM +1000, Stuart Longland wrote: > Hi all, > > This evening I was in the process of deploying a ceph cluster by hand. > I did it by hand because to my knowledge, ceph-deploy doesn't support > Gentoo, and my cluster here runs that. You'll want the ceph-disk & ceph-detect-init pieces here: https://github.com/ceph/ceph/pull/8317 ceph-deploy on Gentoo should only a little bit of work after this. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Deprecating ext4 support
On Mon, Apr 11, 2016 at 06:49:09PM -0400, Shinobu Kinjo wrote: > Just to clarify to prevent any confusion. > > Honestly I've never used ext4 as underlying filesystem for the Ceph cluster, > but according to wiki [1], ext4 is recommended -; > > [1] https://en.wikipedia.org/wiki/Ceph_%28software%29 Clearly somebody made a copy error from the actual documentation. Here's the docs on master and the recent LTS releases. http://docs.ceph.com/docs/firefly/rados/configuration/filesystem-recommendations/ http://docs.ceph.com/docs/hammer/rados/configuration/filesystem-recommendations/ http://docs.ceph.com/docs/master2/rados/configuration/filesystem-recommendations/ The documentation has NEVER recommended ext4. Here's a slice of all history for that file: http://dev.gentoo.org/~robbat2/ceph-history-of-filesystem-recommendations.patch Generated with $ git log -C -C -M -p ceph/master -- \ doc/rados/configuration/filesystem-recommendations.rst \ doc/config-cluster/file-system-recommendations.rst \ doc/config-cluster/file_system_recommendations.rst The very first version, back in 2012, said: > ``ext4`` is a poor file system choice if you intend to deploy the > RADOS Gateway or use snapshots on versions earlier than 0.45. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Single key delete performance against increasing bucket size
On Wed, Mar 16, 2016 at 06:36:33AM +, Pavan Rallabhandi wrote: > I find this to be discussed here before, but couldn¹t find any solution > hence the mail. In RGW, for a bucket holding objects in the range of ~ > millions, one can find it to take for ever to delete the bucket(via > radosgw-admin). I understand the gc(and its parameters) that would reclaim > the space eventually, but am looking more at the bucket deletion options > that can possibly speed up the operation. This ties well into a mail I had sitting in my drafts, but never got around to sending. Whilst doing some rough benchmarking on bucket index sharding, I ran into some terrible performance for key deletion on non-existent keys. Shards did NOT alleviate this performance issue, but did help elsewhere. Numbers given below are for unsharded buckets; relatively empty buckets perform worse when shards before performance picks up again. Test methodology: - Fire single DELETE key ops to the RGW; not using multi-object delete. - I measured the time taken for each delete, and report it here for the 99% percentile (1% of operations took longer than this). - I took at least 1K samples for #keys up to and including 10k keys per bucket. For 50k keys/bucket I capped it to the first 100 samples instead of waiting 10 hours for the run to complete. - The DELETE operations were run single-threaded, with no concurrency. Test environments: Clusters are were both running Hammer 0.94.5 on Ubuntu precise; the hardware is a long way from being new; there are no SSDs, the journal is the first partition on each OSD's disk. The test source host was unloaded, and approx 1ms of latency away from the RGWs. Cluster 1 (Congress, ~1350 OSDs; production cluster; haproxy of 10 RGWs) #keys-in-bucket time per single key delete 0 6.899ms 10 7.507ms 10013.573ms 1000 327.936ms 14825.597ms 5 33802.497ms 10 did-not-finish Cluster 2 (Benjamin, ~50 OSDs; test cluster, practically idle; haproxy of 2 RGWs) #keys-in-bucket time per single key delete 0 4.825ms 10 6.749ms 100 6.146ms 10006.816ms 11233.727ms 5 64262.764ms 10 did-not-finish The cases marked with did-not-finish are where the RGW seems to time out the operation even with the client having an unlimited timeout. It did occur also connected directly to CivetWeb and not HAProxy. I'm not sure why the 100-keys case on the second cluster seems to have been faster than the 10-key case, but I'm willing to put it down to statistical noise. The huge increase at the end, and the operation not returning over 100k items is concerning. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Problem: silently corrupted RadosGW objects caused by slow requests
On Thu, Mar 03, 2016 at 01:55:13PM +0100, Ritter Sławomir wrote: > Hi, > > I think this is really serious problem - again: > > - we silently lost S3/RGW objects in clusters > > Moreover, it our situation looks very similiar to described in > uncorrected bug #13764 (Hammer) and in corrected #8269 (Dumpling). FYI fix in #8269 _is_ present in Hammer: commit bd8e026f88b rgw: don't allow multiple writers to same multiobject part -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] s3 bucket creation time
On Mon, Feb 29, 2016 at 04:58:07PM +, Luis Periquito wrote: > Hi all, > > I have a biggish ceph environment and currently creating a bucket in > radosgw can take as long as 20s. > > What affects the time a bucket takes to be created? How can I improve that > time? > > I've tried to create in several "bucket-location" with different > backing pools (some of them empty) and the time was the same. How many shards do you have configured for the bucket index? I was recently benchmarking different bucket index shard values, and also saw a notable increase relative to the number of shards. Plus a concerning increase directly correlated to number of keys in the bucket, but I need more data before I post to the lists about it. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph S3 Tests
On Tue, Feb 16, 2016 at 04:16:49PM -0600, Justin Restivo wrote: > I verified that this issue is on Amazons side -- I watched it populate to > 101 and failed to let me produce buckets past that. I just submitted a new > ticket as I should have had a bucket limit of 500. Thank you for your > response! If the fixes are working properly, it shouldn't ever get to even 100 buckets. Ideally the bucket cleanup should run after EVERY function. If you look at the website patch, there's some new decorator code I wrote to make the website tests easier, and we can port those to the rest of the checks. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph S3 Tests
On Tue, Feb 16, 2016 at 10:08:38AM -0600, Justin Restivo wrote: > Hi all, > > I am attempting to run the Ceph S3 tests and am really struggling. Any help > at all would be appreciated. > > I have my credentials pointing at my AWS environment, which has a 500 > bucket limit. When I run the tests, I get tons of ERRORS, SKIPS, & FAILS. I > surely can't be the only one to have experienced this! What am I missing? > > S3ResponseError: S3ResponseError: 400 Bad Request > TooManyBuckets How recent in your copy of s3-tests? There was a bug in the testsuite cleanup that I fixed a few months ago, wherein it wasn't cleaning up all the buckets after each test, only the first one. Which it could hit the AWS bucket limit within the run. Commit de65c582 was merged Dec 18 (958a7185). I haven't run any passes against AWS in the last month, but prior to that, I was running the tests a lot when I developed the website code (pending merge still, s3-tests PR#92). -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How-to doc: hosting a static website on radosgw
(I'm on the list, no need to respond directly to either my addresses, robb...@gentoo.org or robin.john...@dreamhost.com). On Tue, Jan 26, 2016 at 02:46:00PM -0800, Yehuda Sadeh-Weinraub wrote: > > The moment this lands in a release, we'll be more than happy to ditch > > the HAProxy request/response mangling bits. But that WIP branch hasn't > > seen commits in 4 months, so we took it as an exercise in coming up I don't have commit on the Ceph organization, so you can track my other WIP pieces in that gap: https://github.com/dreamhost/ceph/branches/all?utf8=%E2%9C%93=website > Here's a more up-to-date branch: > https://github.com/ceph/ceph/tree/wip-rgw-static-website-yehuda > > We're currently testing it, and the plan is to get it in before jewel. > One caveat though, the error page handling still has some issues so at > the moment so the feature will be disabled by default for now. Plus the latest merge to master seems to have broken a few more pieces :-(. The s3-tests for website is available here: https://github.com/dreamhost/s3-tests/tree/wip-static-website -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How-to doc: hosting a static website on radosgw
On Tue, Jan 26, 2016 at 11:51:51PM +0100, Florian Haas wrote: > Hey, slick. Thanks! Out of curiosity, does the wip branch correctly > handle Accept-Encoding: gzip? No, Accept-Encoding is NOT presently implemented in RGW; regardless of static-website. It's pretty low priority for the use-cases I need, as we always run haproxy in front of multiple RGW instances for load balancing. Maybe Yehuda knows of somebody else working to implement it. > By the way, Yehuda, there's one other S3 API feature that we needed in > a different context, which is largely undocumented but works really > nicely, and that's setting CORS rules on buckets. So thanks for that, > too. :) I wrote some user-oriented documentation on CORS here: http://wiki.dreamhost.com/DreamObjects_CORS -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] How-to doc: hosting a static website on radosgw
On Wed, Jan 27, 2016 at 12:08:36AM +0100, Florian Haas wrote: > Agreed, but you don't necessarily need haproxy to do load balancing > (round-robin DNS CNAME with short TTLs is another option), and Wido > started the discussion around an option to ditch HAProxy for radosgw > altogether. ;) There's a long way to go before that's possible: - per-bucket SSL certs, lots and lots of them - overriding HSTS/CORS on a per-origin basis (we use it to inject CORS rules on a specific Origin and give users a nice panel regardless of what rules they set) [* this might make a good RGW feature] - being more responsive to load / pushing certain buckets to a specific RGW, even when using non-subdomain 'Ordinary' calling formats. [hard to do in DNS load balancing] > Neat, that's helpful. Minor suggestion: in your section on client > support, you might want to add that s3cmd supports setting CORS rules > since version 1.6.0 (s3cmd setcors rules.xml s3://bucketname). Thanks, added to that documentation. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph stable releases team: call for participation
On Sat, Oct 03, 2015 at 11:07:22AM +0200, Loic Dachary wrote: > Hi Ceph, > > TL;DR: If you have one day a week to work on the next Ceph stable releases > [1] your help would be most welcome. I'd like to throw my name in. As of August, I work on Ceph development for Dreamhost. Most of my work focuses on RGW, but I also care about getting my RGW fixes out to the world. Presently, that means I have to backport to Firefly & Hammer for production. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph, SSD, and NVMe
On Thu, Oct 01, 2015 at 10:01:03PM -0400, J David wrote: > So, do medium-sized IT organizations (i.e. those without the resources > to have a Ceph developer on staff) run Hammer-based deployments in > production successfully? I'm not sure if I count, given that I'm now working at DreamHost as the in-house Ceph/RGW developer, but as it gave me my background on Ceph At one of my prior positions, I did the prototype & production deployment of our (small) Ceph cluster. Usage predominantly via RGW/S3, but a few RBD volumes exported via iSCSI because it was convenient. As a very small non-profit, we had extremely little budget, and the hardware reflects that. The hardware also ran VMs, which shared the SSDs, but were otherwise not using Ceph except for a very small number of RBD volumes. Per-node Hardware for the production cluster was: Supermicro 2U twin (X9DRT-HF+ boards) Specs for each side of the twin: Dual Xeon E5-2650 256GB RAM (started at 64GB, grew over time for the VMs) 4x 4TB SAS 2x 512GB Samsung 840 PRO (later upgraded with 10Gbit SFP interconnect) Initial build date August 2013. The development cluster was built about 8 months earlier from scraps & spares. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Important security noticed regarding release signing key
On Thu, Sep 17, 2015 at 09:29:35AM -0700, Sage Weil wrote: > Last week, Red Hat investigated an intrusion on the sites of both the Ceph > community project (ceph.com) and Inktank (download.inktank.com), which > were hosted on a computer system outside of Red Hat infrastructure. > > Ceph.com provided Ceph community versions downloads signed with a Ceph > signing key (id 7EBFDD5D17ED316D). Download.inktank.comprovided releases > of the Red Hat Ceph product for Ubuntu and CentOS operating systems signed > with an Inktank signing key (id 5438C7019DCEEEAD). While the investigation > into the intrusion is ongoing, our initial focus was on the integrity of > the software and distribution channel for both sites. Please revoke the old keys, so that if they were taken by the attacker, they cannot be used (you can't un-revoke a key generally). -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Important security noticed regarding release signing key
On Thu, Sep 17, 2015 at 11:19:28AM -0700, Sage Weil wrote: > > Please revoke the old keys, so that if they were taken by the attacker, > > they cannot be used (you can't un-revoke a key generally). > Done: > http://pgp.mit.edu/pks/lookup?search=ceph=index Thank you! -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph/Radosgw v0.94 Content-Type versus Content-type
On Wed, Sep 09, 2015 at 05:28:26PM +, Chang, Fangzhe (Fangzhe) wrote: > I noticed that S3 Java SDK for getContentType() no longer works in > Ceph/Radosgw v0.94 (Hammer). It seems that S3 SDK expects the metadata > “Content-Type” whereas ceph responds with “Content-type”. > Does anyone know how to make a request for having this issue fixed? I put a fix in place for it already, it just needs backport merging to Hammer https://github.com/ceph/ceph/pull/58012 http://tracker.ceph.com/issues/12939 The S3 JDK should also NOT be case-sensitive, the HTTP spec declares that all field names should be treated case-insensitive. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RadosGW - multiple dns names
On Wed, Aug 26, 2015 at 11:52:02AM +0100, Luis Periquito wrote: On Mon, Feb 23, 2015 at 10:18 PM, Yehuda Sadeh-Weinraub yeh...@redhat.com wrote: -- *From: *Shinji Nakamoto shinji.nakam...@mgo.com *To: *ceph-us...@ceph.com *Sent: *Friday, February 20, 2015 3:58:39 PM *Subject: *[ceph-users] RadosGW - multiple dns names We have multiple interfaces on our Rados gateway node, each of which is assigned to one of our many VLANs with a unique IP address. Is it possible to set multiple DNS names for a single Rados GW, so it can handle the request to each of the VLAN specific IP address DNS names? Not yet, however, the upcoming hammer release will support that (hostnames will be configured as part of the region). I tested this using Hammer ( 0.94.2) and it doesn't seem to work. I'm just adding multiple rgw dns name lines to the configuration. Did it make Hammer, or am I doing it the wrong way? I couldn't find any docs either way... http://ceph.com/docs/master/radosgw/config-ref/#get-a-region Look at the hostname entry, which has a description of: hostnames: A list of all the hostnames in the region. For example, you may use multiple domain names to refer to the same region. Optional. The rgw dns name setting will automatically be included in this list. You should restart the radosgw daemon(s) after changing this setting. Then you can stop using 'rgw dns name'. What the functionality does NOT do, is allow you to require a specific hostname arrives on a specific interface. All hostnames are valid for all interfaces/IPs. If you want to restrict it, I'd suggest doing the validation in haproxy, in front of civetweb. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RadosGW - Multiple instances on same host
Override 'rgw frontends' for each instance as well. Eg: rgw frontends = civetweb port=7480 rgw frontends = civetweb port=7481 The default value is: rgw frontends = fastcgi, civetweb port=7480 On Wed, Jun 24, 2015 at 07:52:36PM -0300, Italo Santos wrote: Hello Somnath, I’ve create the files, one for each instance but when I restart the radosgw I see the message bellow on radosgw logs: 2015-06-24 18:46:47.567711 7f30190e3780 0 civetweb: 0xaed360: set_ports_option: cannot bind to 7480: 98 (Address already in use) Could you show how did you do that with civetweb? Regards. Italo Santos http://italosantos.com.br/ On Wednesday, June 24, 2015 at 18:26, Somnath Roy wrote: Hi Italo, What we did was with civetweb and each RGW instances will be running with its own civetweb instances I guess. Now, it seems you are trying to run multiple fastcgi external servers with single apache instance. This should work but I never tried and I am not sure how apache load balanced between those. I think along with the different fcgi file, you need to create separate rgw conf file within /etc/apache2/sites-available . Thanks Regards Somnath From: Italo Santos [mailto:okd...@gmail.com] Sent: Wednesday, June 24, 2015 1:46 PM To: Somnath Roy Cc: ceph-users@lists.ceph.com (mailto:ceph-users@lists.ceph.com) Subject: Re: [ceph-users] RadosGW - Multiple instances on same host Thanks Somnath, In this case I’ll need create a ceph.conf like bellow Also, I’ll need to create two different .fcgi files on /var/www/ one for each instance, as same with the apache config, right? [client.radosgw.instance-01] host = hostname keyring = /etc/ceph/ceph.client.radosgw.instance-01.keyring rgw socket path = /var/run/ceph/ceph.radosgw.instance-01.fastcgi.sock log file = /var/log/radosgw/client.radosgw.log rgw dns name = instance-01.mydomain rgw enable apis = s3, admin rgw enable ops log = true [client.radosgw.instance-02] host = hostname keyring = /etc/ceph/ceph.client.radosgw.instance-02.keyring rgw socket path = /var/run/ceph/ceph.radosgw.instance-02.fastcgi.sock log file = /var/log/radosgw/client.radosgw.log rgw dns name = instance-02.mydomain rgw enable apis = s3, admin rgw enable ops log = true At. Italo Santos http://italosantos.com.br/ On Wednesday, June 24, 2015 at 17:33, Somnath Roy wrote: Yes, it is possible. Each RGW instance is very cpu intensive, make sure you don’t run out of cpu though. Also, each one will be having its own webserver in front so you need a LB in front. Thanks Regards Somnath From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Italo Santos Sent: Wednesday, June 24, 2015 1:05 PM To: ceph-users@lists.ceph.com (mailto:ceph-users@lists.ceph.com) Subject: [ceph-users] RadosGW - Multiple instances on same host Hello everyone, I’d like to know if someone knows if its possible run multiples radosgw instances on the same host? What I’m trying to do is have two diferente radosgw instances responding by different rgw dns... Regards. Italo Santos http://italosantos.com.br/ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Merging two active ceph clusters: suggestions needed
On Wed, Sep 24, 2014 at 11:31:29AM -0700, Yehuda Sadeh wrote: On Wed, Sep 24, 2014 at 11:17 AM, Craig Lewis cle...@centraldesktop.com wrote: Yehuda, are there any potential problems there? I'm wondering if duplicate bucket names that don't have the same contents might cause problems? Would the second cluster be read-only while replication is running? I might have missed part of the original requirements. This sync assumes that B starts as a clean slate. No writes are allowed to it while data is being copied into it. Once ready, all writes to A should be quiesced. Once sync completes, they would then need to reconfigure their system to make B the primary zone. If my B side was empty, I would simply add all the OSDs in as a single cluster. It's the there are S3 buckets RBD images on both sides; none of the bucket names or RBD images conflict, so I was hoping there was a way to merge them. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Merging two active ceph clusters: suggestions needed
On Tue, Sep 23, 2014 at 03:12:53PM -0600, John Nielsen wrote: Keep Cluster A intact and migrate it to your new hardware. You can do this with no downtime, assuming you have enough IOPS to support data migration and normal usage simultaneously. Bring up the new OSDs and let everything rebalance, then remove the old OSDs one at a time. Replace the MONs one at a time. Since you will have the same data on the same cluster (but different hardware), you don't need to worry about mtimes or handling RBD or S3 data at all. The B side already has data however, and that's one of the merge problems (see below re S3). Make sure you have top-level ceph credentials on the new cluster that will work for current users of Cluster B. Use a librbd-aware tool to migrate the RBD volumes from Cluster B onto the new Cluster A. qemu-img comes to mind. This would require downtime for each volume, but not necessarily all at the same time. Thanks, qemu-img didn't come to mind as an RBD migration tool. Migrate your S3 user accounts from Cluster B to the new Cluster A (should be easily scriptable with e.g. JSON output from radosgw-admin). It's fixed now, but didn't used to be possible to create all the various keys. Check for and resolve S3 bucket name conflicts between Cluster A and ClusterB. None. Migrate your S3 data from Cluster B to the new Cluster A using an S3-level tool. s3cmd comes to mind. s3cmd does not preserve mtimes, ACLs or CORS data; that's the largest part of the concern. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Merging two active ceph clusters: suggestions needed
On Sun, Sep 21, 2014 at 02:33:09PM +0900, Christian Balzer wrote: For a variety of reasons, none good anymore, we have two separate Ceph clusters. I would like to merge them onto the newer hardware, with as little downtime and data loss as possible; then discard the old hardware. Cluster A (2 hosts): - 3TB of S3 content, 100k files, file mtimes important - 500GB of RBD volumes, exported via iscsi Cluster B (4 hosts): - 50GiB of S3 content - 7TB of RBD volumes, exported via iscsi Short of finding somewhere to dump all of the data from one side, and re-importing it after merging with that cluster as empty; are there any other alternatives available to me? Having recently seen a similar question and the answer by the Ceph developers, no. As in there is no way (and no plans) for merging clusters. There are export functions for RBD volumes, not sure about S3 and the mtimes as I don't use that functionality. Can somebody else make comments about migrating S3 buckets with preserved mtime data (and all of the ACLs CORS) then? -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Merging two active ceph clusters: suggestions needed
For a variety of reasons, none good anymore, we have two separate Ceph clusters. I would like to merge them onto the newer hardware, with as little downtime and data loss as possible; then discard the old hardware. Cluster A (2 hosts): - 3TB of S3 content, 100k files, file mtimes important - 500GB of RBD volumes, exported via iscsi Cluster B (4 hosts): - 50GiB of S3 content - 7TB of RBD volumes, exported via iscsi Short of finding somewhere to dump all of the data from one side, and re-importing it after merging with that cluster as empty; are there any other alternatives available to me? -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
