Re: [openstack-dev] [ceilometer] [swift] Improving ceilometer.objectstore.swift_middleware
Hi, Is there any way to proceed ahead the following topic? Best Regards, Hisashi Osanai On Friday, August 01, 2014 7:32 PM, Hisashi Osanai wrote: I would like to follow this discussion so I picked up points. - There are two way to collect info from swift, one is pollster and the other is notification. And we discussed about how to solve the performance degradation of swift_middleware here. pollster: - storage.objects - storage.objects.size - storage.objects.containers - storage.containers.objects - storage.containers.objects.size notification: - storage.objects.incoming.bytes - storage.objects.outgoing.bytes - storage.api.request - storage.objects.imcoming.bytes, storage.objects.outgoing.bytes and storage.api.request are handled with swift_middleware because ceilometer need to have the info with per-user and per-tenant basis. - swift has statsd but there is no per-user and per-tenant related info because to realize this swift has to have keystone-isms into core swift code. - improves swift_middleware with stopping the 1:1 mapping b/w API calls and notifications - swift may consume 10s of thousands of event per second and this case is fairly unique so far. I would like to think this performance problem with the following point of view. - need to handle 10s of thousands of event per second - possibility to lost events (i.e. swift proxy goes down when events queued in a swift process) With the notification style there are restriction for above points. Therefore I change the style to get storage.objects.imcoming.bytes, storage.objects.outgoing.bytes and storage.api.request from notification to pollster. Here I met a problem that pointed out by Mr. Merritt, swift has dependency with keystone. But I prefer to solve this problem rather than a problem for notification style. What do you think? My rough idea to solve the dependency problem is - enable statsd (or similar function) in swift - put a middleware in swift proxy - this middleware does not have any communication with ceilometer but put a mark to followed middleware or swift proxy - store metrics with a tenant and a user by statsd if there is the mark store metrics by statsd if there is no mark - Ceilometer (central agent) call APIs to get the metrics Is there any way to solve the dependency problem? Best Regards, Hisashi Osanai ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] [swift] Improving ceilometer.objectstore.swift_middleware
I would like to follow this discussion so I picked up points. - There are two way to collect info from swift, one is pollster and the other is notification. And we discussed about how to solve the performance degradation of swift_middleware here. pollster: - storage.objects - storage.objects.size - storage.objects.containers - storage.containers.objects - storage.containers.objects.size notification: - storage.objects.incoming.bytes - storage.objects.outgoing.bytes - storage.api.request - storage.objects.imcoming.bytes, storage.objects.outgoing.bytes and storage.api.request are handled with swift_middleware because ceilometer need to have the info with per-user and per-tenant basis. - swift has statsd but there is no per-user and per-tenant related info because to realize this swift has to have keystone-isms into core swift code. - improves swift_middleware with stopping the 1:1 mapping b/w API calls and notifications - swift may consume 10s of thousands of event per second and this case is fairly unique so far. I would like to think this performance problem with the following point of view. - need to handle 10s of thousands of event per second - possibility to lost events (i.e. swift proxy goes down when events queued in a swift process) With the notification style there are restriction for above points. Therefore I change the style to get storage.objects.imcoming.bytes, storage.objects.outgoing.bytes and storage.api.request from notification to pollster. Here I met a problem that pointed out by Mr. Merritt, swift has dependency with keystone. But I prefer to solve this problem rather than a problem for notification style. What do you think? My rough idea to solve the dependency problem is - enable statsd (or similar function) in swift - put a middleware in swift proxy - this middleware does not have any communication with ceilometer but put a mark to followed middleware or swift proxy - store metrics with a tenant and a user by statsd if there is the mark store metrics by statsd if there is no mark - Ceilometer (central agent) call APIs to get the metrics Is there any way to solve the dependency problem? Best Regards, Hisashi Osanai ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] [swift] Improving ceilometer.objectstore.swift_middleware
Swift is already emitting those numbers[1] in statsd format; could ceilometer consume those metrics and convert them to whatever notification format it uses? The problem with that approach, IIUC, is that the statsd metrics provide insufficient context. Ceilometer wants to meter usage on a per-user per-tenant basis, so captures[1] the http-x-user-id and http-x-tenant-id headers from the incoming request for this purpose. Similarly, the resource-id is fabricated from the swift account. I don't think this supporting contextual info would be available from raw statsd metrics, or? Cheers, Eoghan [1] https://github.com/openstack/ceilometer/blob/master/ceilometer/objectstore/swift_middleware.py#L168 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] [swift] Improving ceilometer.objectstore.swift_middleware
On Wed, Jul 30 2014, Chris Dent wrote: What are other options? Of those above which are best or most realistic? I'm just thinking out loud and did not push that through, but I wonder if we should not try to use the oslo.messaging notifier middleware for that. It would be more standard (as it's the one usable on all HTTP pipelines) and rely on notification and generates events, as anyway, HTTP requests are events. Then it'd be up to Ceilometer to handle those notifications like it does for the rest of OpenStack. -- Julien Danjou /* Free Software hacker http://julien.danjou.info */ signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] [swift] Improving ceilometer.objectstore.swift_middleware
As a curiosity, are there any ballpark numbers around the volume of notifications ceilometers can handle? Thinking about large scale swift deployments I expect to see thousands or even 10s of thousands of events per second. And that's with today's technology. Looking longer term I wouldn't be surprised to see 100s or even millions per second. And that's just swift. I'd think other services not yet invented will have their own firehoses of data to contribute. -mark -Original Message- From: Julien Danjou [mailto:jul...@danjou.info] Sent: Thursday, July 31, 2014 5:24 AM To: Chris Dent Cc: OpenStack-dev@lists.openstack.org Subject: Re: [openstack-dev] [ceilometer] [swift] Improving ceilometer.objectstore.swift_middleware On Wed, Jul 30 2014, Chris Dent wrote: What are other options? Of those above which are best or most realistic? I'm just thinking out loud and did not push that through, but I wonder if we should not try to use the oslo.messaging notifier middleware for that. It would be more standard (as it's the one usable on all HTTP pipelines) and rely on notification and generates events, as anyway, HTTP requests are events. Then it'd be up to Ceilometer to handle those notifications like it does for the rest of OpenStack. -- Julien Danjou /* Free Software hacker http://julien.danjou.info */ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] [swift] Improving ceilometer.objectstore.swift_middleware
On Thu, 31 Jul 2014, Julien Danjou wrote: I'm just thinking out loud and did not push that through, but I wonder if we should not try to use the oslo.messaging notifier middleware for that. It would be more standard (as it's the one usable on all HTTP pipelines) and rely on notification and generates events, as anyway, HTTP requests are events. Then it'd be up to Ceilometer to handle those notifications like it does for the rest of OpenStack. I assume you mean this stuff: https://github.com/openstack/oslo-incubator/tree/master/openstack/common/middleware If this is an option, and having swift own this is not, that's certainly more in line with what I would prefer: a common-code based solution with no swift dependencies. That would get both of the issues from the aforementioned bugs: * the presumed more performant use of notifications * get rid of the swift dependencies Will link to this thread from the bugs for visibility. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] [swift] Improving ceilometer.objectstore.swift_middleware
As a curiosity, are there any ballpark numbers around the volume of notifications ceilometers can handle? Thinking about large scale swift deployments I expect to see thousands or even 10s of thousands of events per second. And that's with today's technology. Looking longer term I wouldn't be surprised to see 100s or even millions per second. And that's just swift. I'd think other services not yet invented will have their own firehoses of data to contribute. As things currently stand, 10s of thousands of events per second would not be consumable by ceilometer, nor I'd suspect by the messaging bus (open to correction on the latter point). WRT other services not yet invented, I think the approach used in the swift case is fairly unique in that individual API calls are being metered, as opposed to the existence and state of resources. In the ceilometer swift middleware case, I feel we need to re-visit the 1:1 mapping between API calls and notifications, and instead batch up the API activity for individual tenants (and/or users) over a longer time horizon before emitting a summary notification. The existing ceilometer aggregation transformer in the local pipeline may provide the pattern here, but probably requires some specialization to accommodate multiple users PUT'ing to or GET'ing from the same (tenant-scoped) object. Cheers, Eoghan -mark -Original Message- From: Julien Danjou [mailto:jul...@danjou.info] Sent: Thursday, July 31, 2014 5:24 AM To: Chris Dent Cc: OpenStack-dev@lists.openstack.org Subject: Re: [openstack-dev] [ceilometer] [swift] Improving ceilometer.objectstore.swift_middleware On Wed, Jul 30 2014, Chris Dent wrote: What are other options? Of those above which are best or most realistic? I'm just thinking out loud and did not push that through, but I wonder if we should not try to use the oslo.messaging notifier middleware for that. It would be more standard (as it's the one usable on all HTTP pipelines) and rely on notification and generates events, as anyway, HTTP requests are events. Then it'd be up to Ceilometer to handle those notifications like it does for the rest of OpenStack. -- Julien Danjou /* Free Software hacker http://julien.danjou.info */ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] [swift] Improving ceilometer.objectstore.swift_middleware
On 7/31/14, 1:06 AM, Eoghan Glynn wrote: Swift is already emitting those numbers[1] in statsd format; could ceilometer consume those metrics and convert them to whatever notification format it uses? The problem with that approach, IIUC, is that the statsd metrics provide insufficient context. Ceilometer wants to meter usage on a per-user per-tenant basis, so captures[1] the http-x-user-id and http-x-tenant-id headers from the incoming request for this purpose. Similarly, the resource-id is fabricated from the swift account. I don't think this supporting contextual info would be available from raw statsd metrics, or? Good point. Adding per-user and per-tenant fields to the statsd metrics is the wrong way to go on a couple of levels. First, it would leak Keystone-isms into the core Swift code, which is at odds with Swift having pluggable auth systems. Second, it would immediately wreck anyone who's got the statds metrics flowing into Graphite, as suddenly there'd be lots of new metrics for every single tenant/user pair, which would rapidly fill up the Graphite system's disks until it fell over. I think your suggestion elsewhere in the thread of combining multiple API calls into a single notification is a better way to go. That'll certainly result in less client-visible slowdown from sending notifications, particularly if the notifications are sent in a separate greenthread. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [ceilometer] [swift] Improving ceilometer.objectstore.swift_middleware
ceilometer/objectstore/swift_middleware.py[1] counts the size of web
request and reponse bodies through the swift proxy server and publishes
metrics of the size of the request and response and that a request
happened at all.
There are (at least) two bug reports associated with this bit of code:
* avoid requirement on tarball for unit tests
https://bugs.launchpad.net/ceilometer/+bug/1285388
* significant performance degradation when ceilometer middleware for
swift proxy uses
https://bugs.launchpad.net/ceilometer/+bug/1337761
On the first bug the goal is to remove the dependency on swift from
ceilometer. This is halfway done but there are barriers[2] with
regard to the apparently unique way that swift does logging and the
fact that InputProxy and split_path live in swift rather than some
communal location. The barriers may be surmountable but if other
things in the same context are changing, it might not be necessary.
On the second bug, while the majority of the performance cost is in
the call to rpc_server.cast(), achieving maximum performance would
probably come from doing the counts and notifications _not_ in
middlewhere. The final application in the WSGI stack will know the
size of requests and responses without needing to sometime recalculate.
May as well use that.
These two situations overlap in a few ways that suggest we could make
some improvements. I'm after input from both the swift crew and the
ceilometer crew to see if we can reach something that is good for the
long term rather than short term fixes to these bugs.
Some options appear to be:
* Move the middleware to swift or move the functionality to swift.
In the process make the functionality drop generic notifications for
storage.objects.incoming.bytes and storage.objects.outgoing.bytes
that anyone can consume, including ceilometer.
This could potentially address both bugs.
* Move or copy swift.common.utils.{InputProxy,split_path} to somewhere
in oslo, but keep the middleware in ceilometer.
This would require somebody sharing the info on how to properly
participate in swift's logging setup without incorporating swift.
This would fix the first bug without saying anything about the
second.
* Carry on importing the swift tarball or otherwise depending on
swift.
Fixes neither bug, maintains status quo.
What are other options? Of those above which are best or most
realistic?
Personally I'm a fan of the first option: move the functionality into
swift and take it out of middleware. This gets the maximum win for
performance and future flexibility (of consumers).
[1]
https://github.com/openstack/ceilometer/blob/master/ceilometer/objectstore/swift_middleware.py
[2] https://review.openstack.org/#/c/110302/
--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] [swift] Improving ceilometer.objectstore.swift_middleware
On 7/30/14, 8:06 AM, Chris Dent wrote:
ceilometer/objectstore/swift_middleware.py[1] counts the size of web
request and reponse bodies through the swift proxy server and publishes
metrics of the size of the request and response and that a request
happened at all.
There are (at least) two bug reports associated with this bit of code:
* avoid requirement on tarball for unit tests
https://bugs.launchpad.net/ceilometer/+bug/1285388
* significant performance degradation when ceilometer middleware for
swift proxy uses
https://bugs.launchpad.net/ceilometer/+bug/1337761
[snip]
Some options appear to be:
* Move the middleware to swift or move the functionality to swift.
In the process make the functionality drop generic notifications for
storage.objects.incoming.bytes and storage.objects.outgoing.bytes
that anyone can consume, including ceilometer.
This could potentially address both bugs.
* Move or copy swift.common.utils.{InputProxy,split_path} to somewhere
in oslo, but keep the middleware in ceilometer.
This would require somebody sharing the info on how to properly
participate in swift's logging setup without incorporating swift.
This would fix the first bug without saying anything about the
second.
* Carry on importing the swift tarball or otherwise depending on
swift.
Fixes neither bug, maintains status quo.
What are other options? Of those above which are best or most
realistic?
Swift is already emitting those numbers[1] in statsd format; could
ceilometer consume those metrics and convert them to whatever
notification format it uses?
When configured to log to statsd, the Swift proxy will emit metrics of
the form proxy-server.type.verb.status.xfer; for example, a
successful object download would have a metric name of
proxy-server.object.GET.200.xfer and a value of the number of bytes
downloaded. Similarly, PUTs would look like
proxy-server.object.PUT.2xx.xfer.
If ceilometer were to consume these metrics in a process outside the
Swift proxy server, this would solve both problems. The performance fix
comes by being outside the Swift proxy, and consuming statsd metrics can
be done without pulling in Swift code[2].
[1]
http://docs.openstack.org/developer/swift/admin_guide.html#reporting-metrics-to-statsd
[2] e.g. pystatsd
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
