Re: [openstack-dev] [all] how to send messages (and events) to our users
From: Angus Salkeld asalk...@mirantis.commailto:asalk...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Thursday, April 9, 2015 6:43 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [all] how to send messages (and events) to our users On Fri, Apr 10, 2015 at 2:24 AM, Sandy Walsh sandy.wa...@rackspace.commailto:sandy.wa...@rackspace.com wrote: From: Angus Salkeld asalk...@mirantis.commailto:asalk...@mirantis.com Sent: Wednesday, April 8, 2015 8:24 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all] how to send messages (and events) to our users I also want to point out that what I'd actually rather see is that all of the services provide functionality like this. Users would be served by having an event stream from Nova telling them when their instances are active, deleted, stopped, started, error, etc. Also, I really liked Sandy's suggestion to use the notifications on the backend, and then funnel them into something that the user can consume. The project they have, yagi, for putting them into atom feeds is pretty interesting. If we could give people a simple API that says subscribe to Nova/Cinder/Heat/etc. notifications for instance X, and put them in an atom feed, that seems like something that would make sense as an under-the-cloud service that would be relatively low cost and would ultimately reduce load on API servers. an under-the-clould service ? - That is not what I am after here. Yeah, we're using this as an under cloud service. Our notifications are only consumed internally, so it's not a multi-tenant/SaaS solution. What I am really after is a general OpenStack solution for how end users can consume service notifications (and replace heat event-list). Right now there is ceilometer event-list, but as some Ceilometer devs have said, they don't want to store every notification that comes. So is the yagi + atom hopper solution something we can point end-users to? Is it per-tenant etc... However, there is a team within Rax working on this SaaS offering: Peter Kazmir and Joe Savak. I'll let them respond with their lessons on AtomHopper, etc. Great, thanks Sandy. It would be good to see what this is (is it just Zaqar? or something totally different). AtomHopper is an atom-pub server (it's open source: http://atomhopper.org/ ), we use it with Repose (http://openrepose.org/) to support Keystone auth, etc. We use Yagi to read notifications from the rabbit queues and publish notifications in Atom format to a feed. We've been using this setup internally for a couple of years now. As Sandy mentioned, Peter and Joe are working on the project to provide these Atom feeds to end users. Sandy, do you have a write up somewhere on how to set this up so I can experiment a bit? Yagi: https://github.com/rackerlabs/yagi AtomHopper: http://atomhopper.org/ (java warning) The StackTach.v3 sandbox is DevStack-for-Notifications. It simulates notifications (no openstack deploy needed) and it has Yagi set up to consume them. There's also Vagrant scripts to get you going. http://www.stacktach.com/install.html https://github.com/stackforge/stacktach-sandbox and some, slightly older, screencasts on the Sandbox here: http://www.stacktach.com/screencasts.html We're in the #stacktach channel, by all means ping us if you run into problems. Or if a Hangout works better for you, just scream :) Thanks for the help! -Angus __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
On 08/04/15 20:23 -0700, Joshua Harlow wrote: Hope this helps: 'Let's do away with' == 'remove it/no longer use it' vs 'let's use RPC-over-AMQP' which means continue using it/use it more. a-ha... That explains it. My, obviously, non-native english translation failed to parse that. Thanks for explaining :) Feel free to ignore my last 2 emails :P Cheers, Flavio Flavio Percoco wrote: On 08/04/15 15:35 -0700, Min Pae wrote: Uh sorry to nitpick, I think he said “let’s do away with” not “let’s use” RPC-over-AMQP How is that different? I honestly don't see the difference but I'm surre I'm missing something in my translation. On Wed, Apr 8, 2015 at 10:56 AM, Flavio Percoco fla...@redhat.com wrote: On 08/04/15 16:38 +, Sandy Walsh wrote: From: Clint Byrum cl...@fewbar.com Sent: Wednesday, April 8, 2015 1:15 PM There's this: https://wiki.openstack.org/wiki/Cue Hmm, that looks interesting. Will read. I also want to point out that what I'd actually rather see is that all of the services provide functionality like this. Users would be served by having an event stream from Nova telling them when their instances are active, deleted, stopped, started, error, etc. Also, I really liked Sandy's suggestion to use the notifications on the backend, and then funnel them into something that the user can consume. The project they have, yagi, for putting them into atom feeds is pretty interesting. If we could give people a simple API that says subscribe to Nova/Cinder/Heat/etc. notifications for instance X, and put them in an atom feed, that seems like something that would make sense as an under-the-cloud service that would be relatively low cost and would ultimately reduce load on API servers. THIS! Yes. It would be so good to pull apart the state-machine that is Nova and just emit completed actions via notifications. Then, have something like TaskFlow externalize the orchestration. Do away with RPC-over-AMQP. Sorry for being nitpicky but, saying RPC-over-AMQP is way too generic. What AMQP version? On top of what technology? Considering all the issues OPs have with our current broker story, I think considering implementing this on top of pure AMQP (which is how that phrase reads) would not be good. If you meant RPC-over-messaging then I think you should just keep using oslo.nmessaging, which abstracts the problem of picking one broker. Unfortunately, this means users will need to consume this messages from the messaging source using oslo.messaging as well. I say unfortunately because I believe the API - or even the protocol - as it is exposed through this library - or simply the broker - is not something users should deal with. There are services that try to make this interaction simpler - yes, Zaqar. Flavio And, anyone that is interested in the transitions can eavesdrop on the notifications. In our transition from StackTach.v2 to StackTach.v3 in production we simply cloned the notification feeds so the two systems can run in parallel*. No changes to OpenStack, no disruption of service. Later, we'll just kill off the v2 queues. -S * we did this in Yagi, since olso-messaging doesn't support multiple queues from one routing key. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org? subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco pgpVs5uWMugWE.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
Re: [openstack-dev] [all] how to send messages (and events) to our users
I would agree that the reason monasca had to roll their own notification system is likely because one wasn't available, but whether that should be a separate (stand alone separate project) vs integrated service (part of something like monasca) is, to some extent, debatable. No argument on the fact that this is a cross-cutting concern, and as previous posts said it would be great to have a common mechanism for publishing notifications to users. Whether the notification system/service is separated or integrated, what would be the best method to use? Angus started the thread asking whether such a service should be something that pushes to other existing endpoints that the user supplies (syslog), or a publicly available message queue (zaqar), and there was a suggestion to use something like AMQP as well. I assert a public/user facing notification system should be web centric/native for the reason I cited before, one of the consumers of such a notification system will very likely be web browsers (perhaps even horizon itself). If there’s agreement on this point (which I guess there isn’t, or nobody has really chimed in on it yet), then the next thing would be to identify the best protocol/transport for communicating the notifications. I threw out Atom as a potential method, but by no means am I advocating Atom, just that it be a web centric protocol. Also, I’m of the mind that notification/event messaging there should be a multi-tiered approach to notification/event messaging, where there’s probably an internal message bus (be it rabbitmq, kafka, activemq, or what have you) that all services publish to for consumption by other services, and a consumer of said internal message bus that then publishes the events publicly to users in a web native protocol. BTW, I don’t mean open access public, I mean public facing public. There should still be access controls on consuming the public facing notifications. - Min On Wed, Apr 8, 2015, 5:46 PM Halterman, Jonathan jonathan.halter...@hp.com wrote: The ability to send general purpose notifications is clearly a cross-cutting concern. The absence of an AWS SNS like service in OpenStack is the reason that services like Monasca had to roll their own notifications. This has been a gaping hole in the OpenStack portfolio for a while, and I I think the right way to think of a solution is as a new service built around a pub/sub notification API (again, see SNS) as opposed to something which merely exposes OpenStack’s internal messaging infrastructure in some way (that would be inappropriate). Cheers, Jonathan From: Vipul Sabhaya vip...@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Wednesday, April 8, 2015 at 5:18 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [all] how to send messages (and events) to our users On Wed, Apr 8, 2015 at 4:45 PM, Min Pae sputni...@gmail.com wrote: an under-the-clould service ? - That is not what I am after here. I think the thread went off on a tangent and this point got lost. A user facing notification system absolutely should be a web centric protocol, as I imagine one of the big consumers of such a system will be monitoring dashboards which is trending more and more toward rich client side “Single Page Applications”. AMQP would not work well in such cases. So is the yagi + atom hopper solution something we can point end-users to? Is it per-tenant etc... While I haven’t seen it yet, if that solution provides a means to expose the atom events to end users, it seems like a promising start. The thing that’s required, though, is authentication/authorization that’s tied in to keystone, so that notification regarding a tenant’s resource is available only to that tenant. Sandy, do you have a write up somewhere on how to set this up so I can experiment a bit? Maybe this needs to be a part of Cue? Sorry, Cue’s goal is to provision Message Queue/Broker services and manage them, just like Trove provisions and manages databases. Cue would be ideally used to stand up and scale the RabbitMQ cluster providing messaging for an application backend, but it does not provide messaging itself (that would be Zaqar). Agree — I don’t think a multi-tenant notification service (which we seem to be after here) is the goal of Cue. That said, Monasca https://wiki.openstack.org/wiki/Monasca seems have implemented the collection, aggregation, and notification of these events. What may be missing is in Monasca is a mechanism for the tenant to consume these events via something other than AMQP. - Min __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject: unsubscribe
Re: [openstack-dev] [all] how to send messages (and events) to our users
On 06/04/15 22:55, Angus Salkeld wrote: Hi all For quite some time we (Heat team) have wanted to be able to send messages to our users (by user I do not mean the Operator, but the User that is interacting with the client). What do I mean by user messages, and how do they differ from our current log messages and notifications? - Our current logs are for the operator and have information that the user should not have (ip addresses, hostnames, configuration options, other tenant info etc..) - Our notifications (that Ceilometer uses) *could* be used, but I am not sure if it quite fits. (they seem a bit heavy weight for a log message and aimed at higher level events) These messages could be (based on Heat's use case): - Specific user oriented log messages (distinct from our normal operator logs) - Deprecation messages (if they are using old resource properties/template features) - Progress and resource state changes (an application doesn't want to poll an api for a state change) - Automated actions (autoscaling events, time based actions) - Potentially integrated server logs (from in guest agents) I wanted to raise this to [all] as it would be great to have a general solution that all projects can make use of. What do we have now: - The user can not get any kind of log message from services. The closest thing ATM is the notifications in Ceilometer, but I have the feeling that these have a different aim. - nova console log - Heat has a DB event table for users (we have long wanted to get rid of this) What do other clouds provide: - https://devcenter.heroku.com/articles/logging - https://cloud.google.com/logging/docs/ - https://aws.amazon.com/blogs/aws/cloudwatch-log-service/ - http://aws.amazon.com/cloudtrail/ (other examples...) What are some options we could investigate: 1. remote syslog The user provides a rsyslog server IP/port and we send their messages to that. [pros] simple, and the user could also send their server's log messages to the same rsyslog - great visibility into what is going on. There are great tools like loggly/logstash/papertrailapp that source logs from remote syslog It leaves the user in control of what tools they get to use. [cons] Would we become a spam agent (just sending traffic to an IP/Port) - I guess that's how remote syslog works. I am not sure if this is an issue or not? This might be a lesser solution for the use case of an application doesn't want to poll an api for a state change I am not sure how we would integrate this with horizon. 2. Zaqar We send the messages to a queue in Zaqar. [pros] multi tenant OpenStack project for messaging! [cons] I don't think Zaqar is installed in most installations (tho' please correct me here if this is wrong). I know Mirantis does not currently support I think you're correct for now, but I also think that the ability to send messages to the user side is a valuable enough feature to convince many cloud operators to deploy it. Everybody wins in that case. Zaqar, so that would be a problem for me. There is not the level of external tooling like in option 1 (logstash and friends) IMO whatever solution we choose is going to end up requiring the same semantics as Zaqar: durable storage, timeouts of stale messages, arbitrary scale-out, multi-tenancy with Keystone auth, pub-sub, and so on. That leaves us with 3 options: 1) Use Zaqar 2) Write a ground-up replacement for Zaqar. I hope we can agree that this is insane. Particularly if our reason for not using Zaqar is that it isn't widely deployed enough yet. 3) Write or make use of something much simpler than Zaqar that implements only the exact subset of Zaqar's semantics that we need. However, IMHO that is very likely to turn out to be substantially all of them, and once again this is unlikely to solve the problem of wide deployment before Zaqar. So, in conclusion, +1 Zaqar. However, there are some other interesting questions posed by your email. - Do we need a separate service to condition the output to a different protocol? IMHO no - or, rather, not beyond the ones already proposed for Zaqar (long polling, WebSockets, SNS-style notifications). Even if there was some better protocol (in principle I'm not opposed to Atom, for example), I think we'd benefit more by just adding support for it in Zaqar - if it works for this use case then it will work for others that users commonly have. - What should be feeding inputs into the queue? Some service that consumes the existing oslo messaging notifications and sanitises them for the user? Or would every service publish its user notifications directly to the queue? I think this might vary on a case-by-case basis. For things like log messages, warnings and the like, I can see that having a single place to configure it would be valuable. For
Re: [openstack-dev] [all] how to send messages (and events) to our users
From: Angus Salkeld asalk...@mirantis.com Sent: Wednesday, April 8, 2015 8:24 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all] how to send messages (and events) to our users I also want to point out that what I'd actually rather see is that all of the services provide functionality like this. Users would be served by having an event stream from Nova telling them when their instances are active, deleted, stopped, started, error, etc. Also, I really liked Sandy's suggestion to use the notifications on the backend, and then funnel them into something that the user can consume. The project they have, yagi, for putting them into atom feeds is pretty interesting. If we could give people a simple API that says subscribe to Nova/Cinder/Heat/etc. notifications for instance X, and put them in an atom feed, that seems like something that would make sense as an under-the-cloud service that would be relatively low cost and would ultimately reduce load on API servers. an under-the-clould service ? - That is not what I am after here. Yeah, we're using this as an under cloud service. Our notifications are only consumed internally, so it's not a multi-tenant/SaaS solution. What I am really after is a general OpenStack solution for how end users can consume service notifications (and replace heat event-list). Right now there is ceilometer event-list, but as some Ceilometer devs have said, they don't want to store every notification that comes. So is the yagi + atom hopper solution something we can point end-users to? Is it per-tenant etc... However, there is a team within Rax working on this SaaS offering: Peter Kazmir and Joe Savak. I'll let them respond with their lessons on AtomHopper, etc. Sandy, do you have a write up somewhere on how to set this up so I can experiment a bit? Yagi: https://github.com/rackerlabs/yagi AtomHopper: http://atomhopper.org/ (java warning) The StackTach.v3 sandbox is DevStack-for-Notifications. It simulates notifications (no openstack deploy needed) and it has Yagi set up to consume them. There's also Vagrant scripts to get you going. http://www.stacktach.com/install.html https://github.com/stackforge/stacktach-sandbox and some, slightly older, screencasts on the Sandbox here: http://www.stacktach.com/screencasts.html? We're in the #stacktach channel, by all means ping us if you run into problems. Or if a Hangout works better for you, just scream :) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
Excerpts from Angus Salkeld's message of 2015-04-06 19:55:37 -0700: Hi all For quite some time we (Heat team) have wanted to be able to send messages to our users (by user I do not mean the Operator, but the User that is interacting with the client). What do I mean by user messages, and how do they differ from our current log messages and notifications? - Our current logs are for the operator and have information that the user should not have (ip addresses, hostnames, configuration options, other tenant info etc..) - Our notifications (that Ceilometer uses) *could* be used, but I am not sure if it quite fits. (they seem a bit heavy weight for a log message and aimed at higher level events) These messages could be (based on Heat's use case): - Specific user oriented log messages (distinct from our normal operator logs) These currently go in the Heat events API, yes? - Deprecation messages (if they are using old resource properties/template features) I think this could fit with the bits above. - Progress and resource state changes (an application doesn't want to poll an api for a state change) These also go in the current Heat events. - Automated actions (autoscaling events, time based actions) As do these? - Potentially integrated server logs (from in guest agents) I wanted to raise this to [all] as it would be great to have a general solution that all projects can make use of. What do we have now: - The user can not get any kind of log message from services. The closest thing ATM is the notifications in Ceilometer, but I have the feeling that these have a different aim. - nova console log - Heat has a DB event table for users (we have long wanted to get rid of this) So if we forget the DB part of it, the API is also lacking things like pagination and search that one would want in an event/logging API. What do other clouds provide: - https://devcenter.heroku.com/articles/logging - https://cloud.google.com/logging/docs/ - https://aws.amazon.com/blogs/aws/cloudwatch-log-service/ - http://aws.amazon.com/cloudtrail/ (other examples...) What are some options we could investigate: 1. remote syslog The user provides a rsyslog server IP/port and we send their messages to that. [pros] simple, and the user could also send their server's log messages to the same rsyslog - great visibility into what is going on. There are great tools like loggly/logstash/papertrailapp that source logs from remote syslog It leaves the user in control of what tools they get to use. [cons] Would we become a spam agent (just sending traffic to an IP/Port) - I guess that's how remote syslog works. I am not sure if this is an issue or not? This might be a lesser solution for the use case of an application doesn't want to poll an api for a state change I am not sure how we would integrate this with horizon. I think this one puts too much burden on the user to setup a good receiver. 2. Zaqar We send the messages to a queue in Zaqar. [pros] multi tenant OpenStack project for messaging! [cons] I don't think Zaqar is installed in most installations (tho' please correct me here if this is wrong). I know Mirantis does not currently support Zaqar, so that would be a problem for me. There is not the level of external tooling like in option 1 (logstash and friends) I agree with your con, and would also add that after the long discussions we had in the past we had some concerns about scaling. 3. Other options: Please chip in with suggestions/links! There's this: https://wiki.openstack.org/wiki/Cue I think that could be a bit like 1, but provide the user with an easy target for the messages. I also want to point out that what I'd actually rather see is that all of the services provide functionality like this. Users would be served by having an event stream from Nova telling them when their instances are active, deleted, stopped, started, error, etc. Also, I really liked Sandy's suggestion to use the notifications on the backend, and then funnel them into something that the user can consume. The project they have, yagi, for putting them into atom feeds is pretty interesting. If we could give people a simple API that says subscribe to Nova/Cinder/Heat/etc. notifications for instance X, and put them in an atom feed, that seems like something that would make sense as an under-the-cloud service that would be relatively low cost and would ultimately reduce load on API servers. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
From a security point, it certainly scares the hell out of me On 7 April 2015 at 08:45, Chris Friesen chris.frie...@windriver.com wrote: On 04/06/2015 10:08 PM, Angus Salkeld wrote: On Tue, Apr 7, 2015 at 1:53 PM, Chris Friesen chris.frie...@windriver.com mailto:chris.frie...@windriver.com wrote: On 04/06/2015 08:55 PM, Angus Salkeld wrote: Hi all For quite some time we (Heat team) have wanted to be able to send messages to our users (by user I do not mean the Operator, but the User that is interacting with the client). What do I mean by user messages, and how do they differ from our current log messages and notifications? - Our current logs are for the operator and have information that the user should not have (ip addresses, hostnames, configuration options, other tenant info etc..) - Our notifications (that Ceilometer uses) *could* be used, but I am not sure if it quite fits. (they seem a bit heavy weight for a log message and aimed at higher level events) snip What are some options we could investigate: 1. remote syslog 2. Zaqar 3. Other options: Please chip in with suggestions/links! What about a per-user notification topic using the existing notification backend? Wouldn't that require the Operator to provide the end user with access to the message bus? Seems scary to me. AMQP supports access controls, so is it really all that scary? Maybe set up a virtual host per user if we want to be paranoid? (Just throwing it out there as an option since we're already using it...) Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Duncan Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
On Wed, 8 Apr 2015, Sandy Walsh wrote: Yes. It would be so good to pull apart the state-machine that is Nova and just emit completed actions via notifications. Then, have something like TaskFlow externalize the orchestration. Do away with RPC-over-AMQP. YES! I've got notes going back to my first few weeks in OpenStack land that essentially say What's with this RPC? Let's have (observable) events! It was basically the first thing I noticed that stood out as a significant limitation. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
Yeah, I don't think anyone would give access to the production rabbitmq directly. We use Yagi [1] to pipe it to AtomHopper [2] for downstream consumption/sanitizing. -S [1] https://github.com/rackerlabs/yagi [2] http://atomhopper.org/ From: Duncan Thomas duncan.tho...@gmail.com Sent: Wednesday, April 8, 2015 2:03 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all] how to send messages (and events) to our users From a security point, it certainly scares the hell out of me On 7 April 2015 at 08:45, Chris Friesen chris.frie...@windriver.commailto:chris.frie...@windriver.com wrote: On 04/06/2015 10:08 PM, Angus Salkeld wrote: On Tue, Apr 7, 2015 at 1:53 PM, Chris Friesen chris.frie...@windriver.commailto:chris.frie...@windriver.com mailto:chris.frie...@windriver.commailto:chris.frie...@windriver.com wrote: On 04/06/2015 08:55 PM, Angus Salkeld wrote: Hi all For quite some time we (Heat team) have wanted to be able to send messages to our users (by user I do not mean the Operator, but the User that is interacting with the client). What do I mean by user messages, and how do they differ from our current log messages and notifications? - Our current logs are for the operator and have information that the user should not have (ip addresses, hostnames, configuration options, other tenant info etc..) - Our notifications (that Ceilometer uses) *could* be used, but I am not sure if it quite fits. (they seem a bit heavy weight for a log message and aimed at higher level events) snip What are some options we could investigate: 1. remote syslog 2. Zaqar 3. Other options: Please chip in with suggestions/links! What about a per-user notification topic using the existing notification backend? Wouldn't that require the Operator to provide the end user with access to the message bus? Seems scary to me. AMQP supports access controls, so is it really all that scary? Maybe set up a virtual host per user if we want to be paranoid? (Just throwing it out there as an option since we're already using it...) Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Duncan Thomas __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
From: Ryan Brown rybr...@redhat.com Sent: Wednesday, April 8, 2015 9:42 AM The trend in the monitoring space seems to be: 1. Alarms are issued from Metrics as Events. (events can issue alarms too, but conventional alarming is metric based) 2. Multiple events are analyzed to produce Metrics (stream processing) 3. Go to Step 1 Indeed. I sort of envisioned heat sending out events that are then consumed both as metrics and by the user (where appropriate). In StackTach I can see that being implemented as /-- resource events other tools Heat -- Winchester --- notifications stream-- user \-- metrics stream -- alerts --/ Yep, you can get a lot of great info from a notification. And a lot of metrics can be produced from them. We use them for debugging, usage/billing and performance monitoring/tuning. Contextual data ftw! :) Events start as structured data. More so, we're looking at establishing AVRO-based schema definitions on top of these events (slow progress). Yeah, I'd really like to have a schema for Heat events so we can have a single event stream and repackage events for different consumption goals (metrics, notifications, programmatic interaction, etc). Yep, that's the right approach. There are some people at Rax looking at getting this nailed down soon. Having to build filters is a relatively error-prone approach compared to the methods described above. I wasn't saying *we* should do the unstructured message + regex filters strategy, I was just pointing out the CW solution for folks who hadn't used it. Gotcha ... agreed. [snip] The Fujitsu team have already added logging support to Monasca (with an elasticsearch backend) and HP is currently adding StackTach.v3 support for notification-event conversion as well as our Winchester event stream processing engine. Also, this is based on Kafka vs. RabbitMQ, which has better scaling characteristics for this kind of data. Oooh, I'll have a look into that, Kafka as an event bus sounds like a good fit. I have the same concern Angus voiced earlier about Zaqar though. What's the deployment of StackTach.v3 across OpenStack installations? Is it mostly deployed for Helion/Rackspace, or are smaller deployers using it as well? We're in the short strokes of rolling STv3 into production at Rax now. No issues with the libraries, it's all hiccups with downstream system integration. HP have some good requirements they want added around hosted monitoring. People are still installing and playing around with STv2. It's battle proven and solves the immediate OpenStack concerns. But it's more rigid than STv3. If you want to get going today, I'd recommend STv2, but all new efforts and partner work is going into STv3. This could be extended to richer JSON events that include the stack, resources affected in the update, stats like num-deleted-resources or num-replaced-resources, autoscaling actions, and info about stack errors. Some of these sound more like a metrics than notifications. We should be careful not to misuse the two. I think they're events, and have facets that are quantifiable as metrics (num-replaced-resources on an update action) and that should be user-visible (update is complete, or autoscaling actions taken). Yep, tricky to discern sometimes. Perhaps a better way to decide if it's an event or a metric is to consider the frequency they're generated or how much context they contain? Is there a way for users as-is to view those raw notifications, not just the indexed k/v pairs? In StackTach.v3 we ship the raw notifications to HDFS for archiving, but expose the reduced event via the API. The message-id links the two. Lots more here: http://www.stacktach.com Thanks! I'll have to read up. By all means, reach out if you have questions. The more people we have that see the value in events, the better. Looking at the rise of packages like storm, spark, reimann.io, etc. it's clear it's a big change in the distributed computing monitoring space. -S __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
On 04/07/2015 02:34 PM, Sandy Walsh wrote:
Tooling in general seems to be moving towards richer event data as well.
The logging tools (Loggly/Logstash/PaperTrail/zillions of others) are
intended to take your unstructured logs and turn them into events, so
why not have Heat output structured events that we can present to the
user with Ceilometer rather than sending log lines (through syslog or
otherwise) and using tooling to reassemble them into events later.
The trend in the monitoring space seems to be:
1. Alarms are issued from Metrics as Events.
(events can issue alarms too, but conventional alarming is metric based)
2. Multiple events are analyzed to produce Metrics (stream processing)
3. Go to Step 1
Indeed. I sort of envisioned heat sending out events that are then
consumed both as metrics and by the user (where appropriate). In
StackTach I can see that being implemented as
/-- resource events other tools
Heat -- Winchester --- notifications stream-- user
\-- metrics stream -- alerts --/
TL;DR: I think what we really want is a place to send and react to
*events*. Logs are a way to do that, of course, but the Ceilometer way
sounds pretty attractive.
The difference is structured vs. unstructured data. Elasticsearch-based
solutions tend to ignore structure by looking at keywords. Some solutions,
like TopLog, infer a structure by gleaning regexs from logs.
Events start as structured data. More so, we're looking at establishing
AVRO-based schema definitions on top of these events (slow progress).
Yeah, I'd really like to have a schema for Heat events so we can have a
single event stream and repackage events for different consumption goals
(metrics, notifications, programmatic interaction, etc).
If anything we should consider changing the logging library to use structured
messages. Specifically:
log(The %(foo)s did %(thing)s % {'foo':'x', 'thing':'action'})
it would be
log({'message':The %(foo)s did %(thing)s, 'foo':'x', 'thing':'action'})
Which can still be formatted for conventional logs, but also sent as a
notification or as a higher-level structure to things like ES, TopLog, etc.
The driver can decide.
* CloudWatch has you send unstructured log messages, then build filters
to parse them into quantifiable events, then set alarms on those metrics.
Having to build filters is a relatively error-prone approach compared to the
methods described above.
I wasn't saying *we* should do the unstructured message + regex filters
strategy, I was just pointing out the CW solution for folks who hadn't
used it.
[snip]
The Fujitsu team have already added logging support to Monasca (with an
elasticsearch backend) and HP is currently adding StackTach.v3 support for
notification-event conversion as well as our Winchester event stream
processing engine. Also, this is based on Kafka vs. RabbitMQ, which has better
scaling characteristics for this kind of data.
Oooh, I'll have a look into that, Kafka as an event bus sounds like a
good fit. I have the same concern Angus voiced earlier about Zaqar
though. What's the deployment of StackTach.v3 across OpenStack
installations? Is it mostly deployed for Helion/Rackspace, or are
smaller deployers using it as well?
This could be extended to richer JSON events that include the stack,
resources affected in the update, stats like num-deleted-resources or
num-replaced-resources, autoscaling actions, and info about stack errors.
Some of these sound more like a metrics than notifications. We should be
careful not to misuse the two.
I think they're events, and have facets that are quantifiable as metrics
(num-replaced-resources on an update action) and that should be
user-visible (update is complete, or autoscaling actions taken).
Is there a way for users as-is to view those raw notifications, not just
the indexed k/v pairs?
In StackTach.v3 we ship the raw notifications to HDFS for archiving, but
expose the reduced event via the API. The message-id links the two.
Lots more here: http://www.stacktach.com
Thanks! I'll have to read up.
--
Ryan Brown / Software Engineer, Openstack / Red Hat, Inc.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
Yeah, I'd really like to have a schema for Heat events so we can have a
single event stream and repackage events for different consumption goals
(metrics, notifications, programmatic interaction, etc).
Keystone and parts of Ceilometer use the CADF schema to build notification
messages[1]. you can see example usage in ceilometermiddleware[2] but as an
example, it basically builds a notification similar to:
{
'typeURI': 'http: //schemas.dmtf.org/cloud/audit/1.0/event',
'eventTime': '2015-01-30T16: 38: 43.233621',
'target': { # target of event
'typeURI': 'service/storage/object',
'id': 'account',
},
'observer': {
'id': 'target'
},
'eventType': 'activity',
'measurements': [ # measurements if appplicable
{
'metric': {
'metricId': 'openstack: uuid',
'name': 'storage.objects.outgoing.bytes',
'unit': 'B'
},
'result': 28
}
],
'initiator': { # who is triggering event
'typeURI': 'service/security/account/user',
'project_id': None,
'id': 'openstack: 288f6260-bf37-4737-a178-5038c84ba244'
},
'action': 'read',
'outcome': 'success',
'id': 'openstack: 69972bb6-14dd-46e4-bdaf-3148014363dc'
}
a few of the fields are generated if not given. i just mention this because
i've found it immensely easier as a consumer to work off a consistent format.
[1] http://docs.openstack.org/developer/pycadf/
[2]
https://github.com/openstack/ceilometermiddleware/blob/master/ceilometermiddleware/swift.py#L198-L227
cheers,
gord
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
On 07/04/15 12:55 +1000, Angus Salkeld wrote: Hi all For quite some time we (Heat team) have wanted to be able to send messages to our users (by user I do not mean the Operator, but the User that is interacting with the client). What do I mean by user messages, and how do they differ from our current log messages and notifications? - Our current logs are for the operator and have information that the user should not have (ip addresses, hostnames, configuration options, other tenant info etc..) - Our notifications (that Ceilometer uses) *could* be used, but I am not sure if it quite fits. (they seem a bit heavy weight for a log message and aimed at higher level events) These messages could be (based on Heat's use case): - Specific user oriented log messages (distinct from our normal operator logs) - Deprecation messages (if they are using old resource properties/template features) - Progress and resource state changes (an application doesn't want to poll an api for a state change) - Automated actions (autoscaling events, time based actions) - Potentially integrated server logs (from in guest agents) I wanted to raise this to [all] as it would be great to have a general solution that all projects can make use of. What do we have now: - The user can not get any kind of log message from services. The closest thing ATM is the notifications in Ceilometer, but I have the feeling that these have a different aim. - nova console log - Heat has a DB event table for users (we have long wanted to get rid of this) What do other clouds provide: - https://devcenter.heroku.com/articles/logging - https://cloud.google.com/logging/docs/ - https://aws.amazon.com/blogs/aws/cloudwatch-log-service/ - http://aws.amazon.com/cloudtrail/ (other examples...) What are some options we could investigate: 1. remote syslog The user provides a rsyslog server IP/port and we send their messages to that. [pros] simple, and the user could also send their server's log messages to the same rsyslog - great visibility into what is going on. There are great tools like loggly/logstash/papertrailapp that source logs from remote syslog It leaves the user in control of what tools they get to use. [cons] Would we become a spam agent (just sending traffic to an IP/Port) - I guess that's how remote syslog works. I am not sure if this is an issue or not? This might be a lesser solution for the use case of an application doesn't want to poll an api for a state change I am not sure how we would integrate this with horizon. 2. Zaqar We send the messages to a queue in Zaqar. [pros] multi tenant OpenStack project for messaging! [cons] I don't think Zaqar is installed in most installations (tho' please correct me here if this is wrong). I know Mirantis does not currently support Zaqar, so that would be a problem for me. This is, unfortunately, true. However, adoption needs to start somewhere and I believe this would be a good thing to use Zaqar for. Kilo was a heads-down cycle for the Zaqar team but I believe we could help out with making this happen in heat during Liberty. At the Kilo summit, we discussed what was needed for Heat to consume Zaqar and we've completed some of those features. I'm saying this to highlight that Zaqar is now closer to Heat's needs. There is not the level of external tooling like in option 1 (logstash and friends) True again :( There's an almost-complete puppet manifest but other than that, tools are yet to be built. With my Zaqar hat on, I hope we can make this happen and any help on tools/adoption is more than appreciated. Flavio -- @flaper87 Flavio Percoco pgpTGmVFmKyfa.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
From: Clint Byrum cl...@fewbar.com Sent: Wednesday, April 8, 2015 1:15 PM There's this: https://wiki.openstack.org/wiki/Cue Hmm, that looks interesting. Will read. I also want to point out that what I'd actually rather see is that all of the services provide functionality like this. Users would be served by having an event stream from Nova telling them when their instances are active, deleted, stopped, started, error, etc. Also, I really liked Sandy's suggestion to use the notifications on the backend, and then funnel them into something that the user can consume. The project they have, yagi, for putting them into atom feeds is pretty interesting. If we could give people a simple API that says subscribe to Nova/Cinder/Heat/etc. notifications for instance X, and put them in an atom feed, that seems like something that would make sense as an under-the-cloud service that would be relatively low cost and would ultimately reduce load on API servers. THIS! Yes. It would be so good to pull apart the state-machine that is Nova and just emit completed actions via notifications. Then, have something like TaskFlow externalize the orchestration. Do away with RPC-over-AMQP. And, anyone that is interested in the transitions can eavesdrop on the notifications. In our transition from StackTach.v2 to StackTach.v3 in production we simply cloned the notification feeds so the two systems can run in parallel*. No changes to OpenStack, no disruption of service. Later, we'll just kill off the v2 queues. -S * we did this in Yagi, since olso-messaging doesn't support multiple queues from one routing key. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
On 08/04/15 16:38 +, Sandy Walsh wrote: From: Clint Byrum cl...@fewbar.com Sent: Wednesday, April 8, 2015 1:15 PM There's this: https://wiki.openstack.org/wiki/Cue Hmm, that looks interesting. Will read. I also want to point out that what I'd actually rather see is that all of the services provide functionality like this. Users would be served by having an event stream from Nova telling them when their instances are active, deleted, stopped, started, error, etc. Also, I really liked Sandy's suggestion to use the notifications on the backend, and then funnel them into something that the user can consume. The project they have, yagi, for putting them into atom feeds is pretty interesting. If we could give people a simple API that says subscribe to Nova/Cinder/Heat/etc. notifications for instance X, and put them in an atom feed, that seems like something that would make sense as an under-the-cloud service that would be relatively low cost and would ultimately reduce load on API servers. THIS! Yes. It would be so good to pull apart the state-machine that is Nova and just emit completed actions via notifications. Then, have something like TaskFlow externalize the orchestration. Do away with RPC-over-AMQP. Sorry for being nitpicky but, saying RPC-over-AMQP is way too generic. What AMQP version? On top of what technology? Considering all the issues OPs have with our current broker story, I think considering implementing this on top of pure AMQP (which is how that phrase reads) would not be good. If you meant RPC-over-messaging then I think you should just keep using oslo.nmessaging, which abstracts the problem of picking one broker. Unfortunately, this means users will need to consume this messages from the messaging source using oslo.messaging as well. I say unfortunately because I believe the API - or even the protocol - as it is exposed through this library - or simply the broker - is not something users should deal with. There are services that try to make this interaction simpler - yes, Zaqar. Flavio And, anyone that is interested in the transitions can eavesdrop on the notifications. In our transition from StackTach.v2 to StackTach.v3 in production we simply cloned the notification feeds so the two systems can run in parallel*. No changes to OpenStack, no disruption of service. Later, we'll just kill off the v2 queues. -S * we did this in Yagi, since olso-messaging doesn't support multiple queues from one routing key. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco pgpYZZIXGaONk.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
On 8 April 2015 at 20:34, Chris Dent chd...@redhat.com wrote: On Wed, 8 Apr 2015, Sandy Walsh wrote: Yes. It would be so good to pull apart the state-machine that is Nova and just emit completed actions via notifications. Then, have something like TaskFlow externalize the orchestration. Do away with RPC-over-AMQP. YES! I've got notes going back to my first few weeks in OpenStack land that essentially say What's with this RPC? Let's have (observable) events! It was basically the first thing I noticed that stood out as a significant limitation. I'm a bit worried here that a sensible, small, limited and easily implementable piece of work (stream events to tenants) is trying to morph into 'lets re-write nova' - can we keep the two separate, please? Maybe start a new thread on the pros and cons of moving away from the current RPC model (which I'd love to hear - I don't see what's wrong with what we have, but that probably just means I don't understand what you're proposing)... __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
On 08/04/15 15:35 -0700, Min Pae wrote: Uh sorry to nitpick, I think he said “let’s do away with” not “let’s use” RPC-over-AMQP How is that different? I honestly don't see the difference but I'm surre I'm missing something in my translation. On Wed, Apr 8, 2015 at 10:56 AM, Flavio Percoco fla...@redhat.com wrote: On 08/04/15 16:38 +, Sandy Walsh wrote: From: Clint Byrum cl...@fewbar.com Sent: Wednesday, April 8, 2015 1:15 PM There's this: https://wiki.openstack.org/wiki/Cue Hmm, that looks interesting. Will read. I also want to point out that what I'd actually rather see is that all of the services provide functionality like this. Users would be served by having an event stream from Nova telling them when their instances are active, deleted, stopped, started, error, etc. Also, I really liked Sandy's suggestion to use the notifications on the backend, and then funnel them into something that the user can consume. The project they have, yagi, for putting them into atom feeds is pretty interesting. If we could give people a simple API that says subscribe to Nova/Cinder/Heat/etc. notifications for instance X, and put them in an atom feed, that seems like something that would make sense as an under-the-cloud service that would be relatively low cost and would ultimately reduce load on API servers. THIS! Yes. It would be so good to pull apart the state-machine that is Nova and just emit completed actions via notifications. Then, have something like TaskFlow externalize the orchestration. Do away with RPC-over-AMQP. Sorry for being nitpicky but, saying RPC-over-AMQP is way too generic. What AMQP version? On top of what technology? Considering all the issues OPs have with our current broker story, I think considering implementing this on top of pure AMQP (which is how that phrase reads) would not be good. If you meant RPC-over-messaging then I think you should just keep using oslo.nmessaging, which abstracts the problem of picking one broker. Unfortunately, this means users will need to consume this messages from the messaging source using oslo.messaging as well. I say unfortunately because I believe the API - or even the protocol - as it is exposed through this library - or simply the broker - is not something users should deal with. There are services that try to make this interaction simpler - yes, Zaqar. Flavio And, anyone that is interested in the transitions can eavesdrop on the notifications. In our transition from StackTach.v2 to StackTach.v3 in production we simply cloned the notification feeds so the two systems can run in parallel*. No changes to OpenStack, no disruption of service. Later, we'll just kill off the v2 queues. -S * we did this in Yagi, since olso-messaging doesn't support multiple queues from one routing key. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org? subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco pgpCO0u00zDmL.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
Sandy Walsh wrote: From: Clint Byrumcl...@fewbar.com Sent: Wednesday, April 8, 2015 1:15 PM There's this: https://wiki.openstack.org/wiki/Cue Hmm, that looks interesting. Will read. I also want to point out that what I'd actually rather see is that all of the services provide functionality like this. Users would be served by having an event stream from Nova telling them when their instances are active, deleted, stopped, started, error, etc. Also, I really liked Sandy's suggestion to use the notifications on the backend, and then funnel them into something that the user can consume. The project they have, yagi, for putting them into atom feeds is pretty interesting. If we could give people a simple API that says subscribe to Nova/Cinder/Heat/etc. notifications for instance X, and put them in an atom feed, that seems like something that would make sense as an under-the-cloud service that would be relatively low cost and would ultimately reduce load on API servers. THIS! Yes. It would be so good to pull apart the state-machine that is Nova and just emit completed actions via notifications. Then, have something like TaskFlow externalize the orchestration. Do away with RPC-over-AMQP. Sounds good to me ;) http://docs.openstack.org/developer/taskflow/notifications.html were designed for this purpose; some of the implementations at: http://docs.openstack.org/developer/taskflow/notifications.html#implementations I know that http://www.rackspace.com/cloud/big-data/ is using one of these (among others) and using it to do various tracking of workflows/tasks, and such and gathering the information at whatever level is desired. The neat thing is that it allows for post-workflow addition of listeners so if at some future point you want to know the timing of your workflow, well u can just plug another listener in that gathers this information (for example http://docs.openstack.org/developer/taskflow/notifications.html#taskflow.listeners.timing.EventTimeListener)... -Josh And, anyone that is interested in the transitions can eavesdrop on the notifications. In our transition from StackTach.v2 to StackTach.v3 in production we simply cloned the notification feeds so the two systems can run in parallel*. No changes to OpenStack, no disruption of service. Later, we'll just kill off the v2 queues. -S * we did this in Yagi, since olso-messaging doesn't support multiple queues from one routing key. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
Hope this helps: 'Let's do away with' == 'remove it/no longer use it' vs 'let's use RPC-over-AMQP' which means continue using it/use it more. Flavio Percoco wrote: On 08/04/15 15:35 -0700, Min Pae wrote: Uh sorry to nitpick, I think he said “let’s do away with” not “let’s use” RPC-over-AMQP How is that different? I honestly don't see the difference but I'm surre I'm missing something in my translation. On Wed, Apr 8, 2015 at 10:56 AM, Flavio Percoco fla...@redhat.com wrote: On 08/04/15 16:38 +, Sandy Walsh wrote: From: Clint Byrum cl...@fewbar.com Sent: Wednesday, April 8, 2015 1:15 PM There's this: https://wiki.openstack.org/wiki/Cue Hmm, that looks interesting. Will read. I also want to point out that what I'd actually rather see is that all of the services provide functionality like this. Users would be served by having an event stream from Nova telling them when their instances are active, deleted, stopped, started, error, etc. Also, I really liked Sandy's suggestion to use the notifications on the backend, and then funnel them into something that the user can consume. The project they have, yagi, for putting them into atom feeds is pretty interesting. If we could give people a simple API that says subscribe to Nova/Cinder/Heat/etc. notifications for instance X, and put them in an atom feed, that seems like something that would make sense as an under-the-cloud service that would be relatively low cost and would ultimately reduce load on API servers. THIS! Yes. It would be so good to pull apart the state-machine that is Nova and just emit completed actions via notifications. Then, have something like TaskFlow externalize the orchestration. Do away with RPC-over-AMQP. Sorry for being nitpicky but, saying RPC-over-AMQP is way too generic. What AMQP version? On top of what technology? Considering all the issues OPs have with our current broker story, I think considering implementing this on top of pure AMQP (which is how that phrase reads) would not be good. If you meant RPC-over-messaging then I think you should just keep using oslo.nmessaging, which abstracts the problem of picking one broker. Unfortunately, this means users will need to consume this messages from the messaging source using oslo.messaging as well. I say unfortunately because I believe the API - or even the protocol - as it is exposed through this library - or simply the broker - is not something users should deal with. There are services that try to make this interaction simpler - yes, Zaqar. Flavio And, anyone that is interested in the transitions can eavesdrop on the notifications. In our transition from StackTach.v2 to StackTach.v3 in production we simply cloned the notification feeds so the two systems can run in parallel*. No changes to OpenStack, no disruption of service. Later, we'll just kill off the v2 queues. -S * we did this in Yagi, since olso-messaging doesn't support multiple queues from one routing key. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org? subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
Uh sorry to nitpick, I think he said “let’s do away with” not “let’s use” RPC-over-AMQP On Wed, Apr 8, 2015 at 10:56 AM, Flavio Percoco fla...@redhat.com wrote: On 08/04/15 16:38 +, Sandy Walsh wrote: From: Clint Byrum cl...@fewbar.com Sent: Wednesday, April 8, 2015 1:15 PM There's this: https://wiki.openstack.org/wiki/Cue Hmm, that looks interesting. Will read. I also want to point out that what I'd actually rather see is that all of the services provide functionality like this. Users would be served by having an event stream from Nova telling them when their instances are active, deleted, stopped, started, error, etc. Also, I really liked Sandy's suggestion to use the notifications on the backend, and then funnel them into something that the user can consume. The project they have, yagi, for putting them into atom feeds is pretty interesting. If we could give people a simple API that says subscribe to Nova/Cinder/Heat/etc. notifications for instance X, and put them in an atom feed, that seems like something that would make sense as an under-the-cloud service that would be relatively low cost and would ultimately reduce load on API servers. THIS! Yes. It would be so good to pull apart the state-machine that is Nova and just emit completed actions via notifications. Then, have something like TaskFlow externalize the orchestration. Do away with RPC-over-AMQP. Sorry for being nitpicky but, saying RPC-over-AMQP is way too generic. What AMQP version? On top of what technology? Considering all the issues OPs have with our current broker story, I think considering implementing this on top of pure AMQP (which is how that phrase reads) would not be good. If you meant RPC-over-messaging then I think you should just keep using oslo.nmessaging, which abstracts the problem of picking one broker. Unfortunately, this means users will need to consume this messages from the messaging source using oslo.messaging as well. I say unfortunately because I believe the API - or even the protocol - as it is exposed through this library - or simply the broker - is not something users should deal with. There are services that try to make this interaction simpler - yes, Zaqar. Flavio And, anyone that is interested in the transitions can eavesdrop on the notifications. In our transition from StackTach.v2 to StackTach.v3 in production we simply cloned the notification feeds so the two systems can run in parallel*. No changes to OpenStack, no disruption of service. Later, we'll just kill off the v2 queues. -S * we did this in Yagi, since olso-messaging doesn't support multiple queues from one routing key. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject: unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
On Wed, Apr 8, 2015 at 4:45 PM, Min Pae sputni...@gmail.com wrote: an under-the-clould service ? - That is not what I am after here. I think the thread went off on a tangent and this point got lost. A user facing notification system absolutely should be a web centric protocol, as I imagine one of the big consumers of such a system will be monitoring dashboards which is trending more and more toward rich client side “Single Page Applications”. AMQP would not work well in such cases. So is the yagi + atom hopper solution something we can point end-users to? Is it per-tenant etc... While I haven’t seen it yet, if that solution provides a means to expose the atom events to end users, it seems like a promising start. The thing that’s required, though, is authentication/authorization that’s tied in to keystone, so that notification regarding a tenant’s resource is available only to that tenant. Sandy, do you have a write up somewhere on how to set this up so I can experiment a bit? Maybe this needs to be a part of Cue? Sorry, Cue’s goal is to provision Message Queue/Broker services and manage them, just like Trove provisions and manages databases. Cue would be ideally used to stand up and scale the RabbitMQ cluster providing messaging for an application backend, but it does not provide messaging itself (that would be Zaqar). Agree — I don’t think a multi-tenant notification service (which we seem to be after here) is the goal of Cue. That said, Monasca https://wiki.openstack.org/wiki/Monasca seems have implemented the collection, aggregation, and notification of these events. What may be missing is in Monasca is a mechanism for the tenant to consume these events via something other than AMQP. - Min __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
The ability to send general purpose notifications is clearly a cross-cutting concern. The absence of an AWS SNS like service in OpenStack is the reason that services like Monasca had to roll their own notifications. This has been a gaping hole in the OpenStack portfolio for a while, and I I think the right way to think of a solution is as a new service built around a pub/sub notification API (again, see SNS) as opposed to something which merely exposes OpenStack¹s internal messaging infrastructure in some way (that would be inappropriate). Cheers, Jonathan From: Vipul Sabhaya vip...@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Wednesday, April 8, 2015 at 5:18 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [all] how to send messages (and events) to our users On Wed, Apr 8, 2015 at 4:45 PM, Min Pae sputni...@gmail.com wrote: an under-the-clould service ? - That is not what I am after here. I think the thread went off on a tangent and this point got lost. A user facing notification system absolutely should be a web centric protocol, as I imagine one of the big consumers of such a system will be monitoring dashboards which is trending more and more toward rich client side ³Single Page Applications². AMQP would not work well in such cases. So is the yagi + atom hopper solution something we can point end-users to? Is it per-tenant etc... While I haven¹t seen it yet, if that solution provides a means to expose the atom events to end users, it seems like a promising start. The thing that¹s required, though, is authentication/authorization that¹s tied in to keystone, so that notification regarding a tenant¹s resource is available only to that tenant. Sandy, do you have a write up somewhere on how to set this up so I can experiment a bit? Maybe this needs to be a part of Cue? Sorry, Cue¹s goal is to provision Message Queue/Broker services and manage them, just like Trove provisions and manages databases. Cue would be ideally used to stand up and scale the RabbitMQ cluster providing messaging for an application backend, but it does not provide messaging itself (that would be Zaqar). Agree I don¹t think a multi-tenant notification service (which we seem to be after here) is the goal of Cue. That said, Monasca https://wiki.openstack.org/wiki/Monasca seems have implemented the collection, aggregation, and notification of these events. What may be missing is in Monasca is a mechanism for the tenant to consume these events via something other than AMQP. - Min __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev smime.p7s Description: S/MIME cryptographic signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
an under-the-clould service ? - That is not what I am after here. I think the thread went off on a tangent and this point got lost. A user facing notification system absolutely should be a web centric protocol, as I imagine one of the big consumers of such a system will be monitoring dashboards which is trending more and more toward rich client side “Single Page Applications”. AMQP would not work well in such cases. So is the yagi + atom hopper solution something we can point end-users to? Is it per-tenant etc... While I haven’t seen it yet, if that solution provides a means to expose the atom events to end users, it seems like a promising start. The thing that’s required, though, is authentication/authorization that’s tied in to keystone, so that notification regarding a tenant’s resource is available only to that tenant. Sandy, do you have a write up somewhere on how to set this up so I can experiment a bit? Maybe this needs to be a part of Cue? Sorry, Cue’s goal is to provision Message Queue/Broker services and manage them, just like Trove provisions and manages databases. Cue would be ideally used to stand up and scale the RabbitMQ cluster providing messaging for an application backend, but it does not provide messaging itself (that would be Zaqar). - Min __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
On Thu, Apr 9, 2015 at 2:15 AM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Angus Salkeld's message of 2015-04-06 19:55:37 -0700: Hi all For quite some time we (Heat team) have wanted to be able to send messages to our users (by user I do not mean the Operator, but the User that is interacting with the client). What do I mean by user messages, and how do they differ from our current log messages and notifications? - Our current logs are for the operator and have information that the user should not have (ip addresses, hostnames, configuration options, other tenant info etc..) - Our notifications (that Ceilometer uses) *could* be used, but I am not sure if it quite fits. (they seem a bit heavy weight for a log message and aimed at higher level events) These messages could be (based on Heat's use case): - Specific user oriented log messages (distinct from our normal operator logs) These currently go in the Heat events API, yes? Well I wanted a replacement for our events table that could also take smaller messages and fulfil some other use cases (be a stream, that the user doesn't have to poll). We are sending RPC notifications at the moment: stack.action.begin/end/error autoscaling.begin/end/error We need to add: per resource notifications (to replace the event table) Add some useful log-like messages: x property is deprecated, please update your template to use Y There are also a bunch of other log messages that should really be going to the user, but just end up in the Operator's log file. - Deprecation messages (if they are using old resource properties/template features) I think this could fit with the bits above. - Progress and resource state changes (an application doesn't want to poll an api for a state change) These also go in the current Heat events. - Automated actions (autoscaling events, time based actions) As do these? - Potentially integrated server logs (from in guest agents) I wanted to raise this to [all] as it would be great to have a general solution that all projects can make use of. What do we have now: - The user can not get any kind of log message from services. The closest thing ATM is the notifications in Ceilometer, but I have the feeling that these have a different aim. - nova console log - Heat has a DB event table for users (we have long wanted to get rid of this) So if we forget the DB part of it, the API is also lacking things like pagination and search that one would want in an event/logging API. I'd almost rather spend the time getting an OpenStack wide solution then switch our client (and API) to use that. What do other clouds provide: - https://devcenter.heroku.com/articles/logging - https://cloud.google.com/logging/docs/ - https://aws.amazon.com/blogs/aws/cloudwatch-log-service/ - http://aws.amazon.com/cloudtrail/ (other examples...) What are some options we could investigate: 1. remote syslog The user provides a rsyslog server IP/port and we send their messages to that. [pros] simple, and the user could also send their server's log messages to the same rsyslog - great visibility into what is going on. There are great tools like loggly/logstash/papertrailapp that source logs from remote syslog It leaves the user in control of what tools they get to use. [cons] Would we become a spam agent (just sending traffic to an IP/Port) - I guess that's how remote syslog works. I am not sure if this is an issue or not? This might be a lesser solution for the use case of an application doesn't want to poll an api for a state change I am not sure how we would integrate this with horizon. I think this one puts too much burden on the user to setup a good receiver. 2. Zaqar We send the messages to a queue in Zaqar. [pros] multi tenant OpenStack project for messaging! [cons] I don't think Zaqar is installed in most installations (tho' please correct me here if this is wrong). I know Mirantis does not currently support Zaqar, so that would be a problem for me. There is not the level of external tooling like in option 1 (logstash and friends) I agree with your con, and would also add that after the long discussions we had in the past we had some concerns about scaling. 3. Other options: Please chip in with suggestions/links! There's this: https://wiki.openstack.org/wiki/Cue I think that could be a bit like 1, but provide the user with an easy target for the messages. I also want to point out that what I'd actually rather see is that all of the services provide functionality like this. Users would be served by having an event stream from Nova telling them when their instances are active, deleted, stopped,
Re: [openstack-dev] [all] how to send messages (and events) to our users
+1 for using taskflow to implement workflows… workflow reliability is a non-trivial problem that is best solved in one place imho I have doubts as to whether AMQP is the right protocol for notifications. Web service interfaces, thus far, being the “standard” interface for interacting with Openstack, it would seem like the best thing to do is to stay with a web centric protocol like Atom or RSS for delivering notifications. Also, as to where the notification should be served out of, whether it’s provided by the service itself vs a common shared service (like Ceilometer or Zaqar), I feel like the service APIs are management interfaces and should not be for event notifications, since the scaling requirements on a management interface (which might have to support complex workflows) and a notification interface (which would be simple in logic but very high in volume) would be different. Does that mean each service should provide a separate web stack for serving notifications? I think there’s a benefit to being able to go to a single place for notifications, and if I’m an operator I’d rather have a fewer things to maintain, so a centralized notification service seems like the better choice. - Min On Wed, Apr 8, 2015 at 3:14 PM, Joshua Harlow harlo...@outlook.com wrote: Sandy Walsh wrote: From: Clint Byrumcl...@fewbar.com Sent: Wednesday, April 8, 2015 1:15 PM There's this: https://wiki.openstack.org/wiki/Cue Hmm, that looks interesting. Will read. I also want to point out that what I'd actually rather see is that all of the services provide functionality like this. Users would be served by having an event stream from Nova telling them when their instances are active, deleted, stopped, started, error, etc. Also, I really liked Sandy's suggestion to use the notifications on the backend, and then funnel them into something that the user can consume. The project they have, yagi, for putting them into atom feeds is pretty interesting. If we could give people a simple API that says subscribe to Nova/Cinder/Heat/etc. notifications for instance X, and put them in an atom feed, that seems like something that would make sense as an under-the-cloud service that would be relatively low cost and would ultimately reduce load on API servers. THIS! Yes. It would be so good to pull apart the state-machine that is Nova and just emit completed actions via notifications. Then, have something like TaskFlow externalize the orchestration. Do away with RPC-over-AMQP. Sounds good to me ;) http://docs.openstack.org/developer/taskflow/notifications.html were designed for this purpose; some of the implementations at: http://docs.openstack.org/developer/taskflow/notifications.html# implementations I know that http://www.rackspace.com/cloud/big-data/ is using one of these (among others) and using it to do various tracking of workflows/tasks, and such and gathering the information at whatever level is desired. The neat thing is that it allows for post-workflow addition of listeners so if at some future point you want to know the timing of your workflow, well u can just plug another listener in that gathers this information (for example http://docs.openstack.org/developer/taskflow/ notifications.html#taskflow.listeners.timing.EventTimeListener)... -Josh And, anyone that is interested in the transitions can eavesdrop on the notifications. In our transition from StackTach.v2 to StackTach.v3 in production we simply cloned the notification feeds so the two systems can run in parallel*. No changes to OpenStack, no disruption of service. Later, we'll just kill off the v2 queues. -S * we did this in Yagi, since olso-messaging doesn't support multiple queues from one routing key. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject: unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
Notifications were added for this very purpose. At Rax, we have a downstream consumer (yagi) that routes notifications to an appropriate ATOM/pubsub feed (based on tenant-id). Notifications are only as heavy as you want to make them. The only required fields are event_type and timestamp. ? From: Angus Salkeld asalk...@mirantis.com Sent: Monday, April 6, 2015 11:55 PM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [all] how to send messages (and events) to our users Hi all For quite some time we (Heat team) have wanted to be able to send messages to our users (by user I do not mean the Operator, but the User that is interacting with the client). What do I mean by user messages, and how do they differ from our current log messages and notifications? - Our current logs are for the operator and have information that the user should not have (ip addresses, hostnames, configuration options, other tenant info etc..) - Our notifications (that Ceilometer uses) *could* be used, but I am not sure if it quite fits. (they seem a bit heavy weight for a log message and aimed at higher level events) These messages could be (based on Heat's use case): - Specific user oriented log messages (distinct from our normal operator logs) - Deprecation messages (if they are using old resource properties/template features) - Progress and resource state changes (an application doesn't want to poll an api for a state change) - Automated actions (autoscaling events, time based actions) - Potentially integrated server logs (from in guest agents) I wanted to raise this to [all] as it would be great to have a general solution that all projects can make use of. What do we have now: - The user can not get any kind of log message from services. The closest thing ATM is the notifications in Ceilometer, but I have the feeling that these have a different aim. - nova console log - Heat has a DB event table for users (we have long wanted to get rid of this) What do other clouds provide: - https://devcenter.heroku.com/articles/logging - https://cloud.google.com/logging/docs/ - https://aws.amazon.com/blogs/aws/cloudwatch-log-service/ - http://aws.amazon.com/cloudtrail/ (other examples...) What are some options we could investigate: 1. remote syslog The user provides a rsyslog server IP/port and we send their messages to that. [pros] simple, and the user could also send their server's log messages to the same rsyslog - great visibility into what is going on. There are great tools like loggly/logstash/papertrailapp that source logs from remote syslog It leaves the user in control of what tools they get to use. [cons] Would we become a spam agent (just sending traffic to an IP/Port) - I guess that's how remote syslog works. I am not sure if this is an issue or not? This might be a lesser solution for the use case of an application doesn't want to poll an api for a state change I am not sure how we would integrate this with horizon. 2. Zaqar We send the messages to a queue in Zaqar. [pros] multi tenant OpenStack project for messaging! [cons] I don't think Zaqar is installed in most installations (tho' please correct me here if this is wrong). I know Mirantis does not currently support Zaqar, so that would be a problem for me. There is not the level of external tooling like in option 1 (logstash and friends) 3. Other options: Please chip in with suggestions/links! https://blueprints.launchpad.net/heat/+spec/user-visible-logs Regards Angus __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
- Specific user oriented log messages (distinct from our normal operator logs) - Deprecation messages (if they are using old resource properties/template features) - Progress and resource state changes (an application doesn't want to poll an api for a state change) - Automated actions (autoscaling events, time based actions) - Potentially integrated server logs (from in guest agents) is the idea that Heat would build events from the logs or would you want to send the log messages to another service to be process? so for example, Nova doesn't send all logs messages to the queue but they do send a set of messages relating to certain actions and errors that occur (beyond just CRUD events). as the use cases above seem to target specific actions/logs and not all logs, i would think the processing could be done on the initiators service end and not on the consumer end. to give an example of what Ceilometer is capable of; Ceilometer currently takes JSON messages from the MQ from *most* services and from there we capture the entire raw notification and index on a select set of key-value pairs. i think it's entirely possible to take in non-json log messages and build an indexer around that if needed. cheers, gord Date: Tue, 7 Apr 2015 12:55:37 +1000 From: asalk...@mirantis.com To: openstack-dev@lists.openstack.org Subject: [openstack-dev] [all] how to send messages (and events) to our users Hi all For quite some time we (Heat team) have wanted to be able to send messages to ourusers (by user I do not mean the Operator, but the User that is interacting with the client). What do I mean by user messages, and how do they differ from our current log messagesand notifications?- Our current logs are for the operator and have information that the user should not have (ip addresses, hostnames, configuration options, other tenant info etc..)- Our notifications (that Ceilometer uses) *could* be used, but I am not sure if it quite fits. (they seem a bit heavy weight for a log message and aimed at higher level events) These messages could be (based on Heat's use case): - Specific user oriented log messages (distinct from our normal operator logs)- Deprecation messages (if they are using old resource properties/template features)- Progress and resource state changes (an application doesn't want to poll an api for a state change) - Automated actions (autoscaling events, time based actions)- Potentially integrated server logs (from in guest agents) I wanted to raise this to [all] as it would be great to have a general solution that all projects can make use of. What do we have now: - The user can not get any kind of log message from services. The closest thing ATM is the notifications in Ceilometer, but I have the feeling that these have a different aim.- nova console log- Heat has a DB event table for users (we have long wanted to get rid of this) What do other clouds provide:- https://devcenter.heroku.com/articles/logging- https://cloud.google.com/logging/docs/- https://aws.amazon.com/blogs/aws/cloudwatch-log-service/- http://aws.amazon.com/cloudtrail/(other examples...) What are some options we could investigate:1. remote syslogThe user provides a rsyslog server IP/port and we send their messages to that.[pros] simple, and the user could also send their server's log messages to the same rsyslog - great visibility into what is going on. There are great tools like loggly/logstash/papertrailapp that source logs from remote syslog It leaves the user in control of what tools they get to use. [cons] Would we become a spam agent (just sending traffic to an IP/Port) - I guess that's how remote syslog works. I am not sure if this is an issue or not? This might be a lesser solution for the use case of an application doesn't want to poll an api for a state change I am not sure how we would integrate this with horizon. 2. ZaqarWe send the messages to a queue in Zaqar.[pros] multi tenant OpenStack project for messaging! [cons] I don't think Zaqar is installed in most installations (tho' please correct me here if this is wrong). I know Mirantis does not currently support Zaqar, so that would be a problem for me. There is not the level of external tooling like in option 1 (logstash and friends) 3. Other options: Please chip in with suggestions/links! https://blueprints.launchpad.net/heat/+spec/user-visible-logs RegardsAngus __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing
Re: [openstack-dev] [all] how to send messages (and events) to our users
On 04/07/2015 10:10 AM, gordon chung wrote: - Specific user oriented log messages (distinct from our normal operator logs) - Deprecation messages (if they are using old resource properties/template features) - Progress and resource state changes (an application doesn't want to poll an api for a state change) - Automated actions (autoscaling events, time based actions) - Potentially integrated server logs (from in guest agents) Angus mentions that we want user messages but I'd argue that an events interface (not to be confused with our current stack events) would be a great fit. We could define schemas (building on API-WG's error message guide[1]) for heat events so users can programmatically interact with heat w/o polling all the time. I think moving slightly up the abstraction ladder from here's a log message to here's a structured event with extra metadata too would be great, and do a better job helping users than unstructured log messages. The difference between the AWS options is instructive here because the older service (CloudWatch) is log-line oriented*, while the newer service (CloudTrail) provides structured events. Tooling in general seems to be moving towards richer event data as well. The logging tools (Loggly/Logstash/PaperTrail/zillions of others) are intended to take your unstructured logs and turn them into events, so why not have Heat output structured events that we can present to the user with Ceilometer rather than sending log lines (through syslog or otherwise) and using tooling to reassemble them into events later. TL;DR: I think what we really want is a place to send and react to *events*. Logs are a way to do that, of course, but the Ceilometer way sounds pretty attractive. * CloudWatch has you send unstructured log messages, then build filters to parse them into quantifiable events, then set alarms on those metrics. is the idea that Heat would build events from the logs or would you want to send the log messages to another service to be process? so for example, Nova doesn't send all logs messages to the queue but they do send a set of messages relating to certain actions and errors that occur (beyond just CRUD events). as the use cases above seem to target specific actions/logs and not all logs, i would think the processing could be done on the initiators service end and not on the consumer end. to give an example of what Ceilometer is capable of; Ceilometer currently takes JSON messages from the MQ from *most* services and from there we capture the entire raw notification and index on a select set of key-value pairs. i think it's entirely possible to take in non-json log messages and build an indexer around that if needed. I don't think it would be too hard for us to package up events like stack state transitions, failures (with as much debug info as is reasonable), autoscaling actions, etc. (please pardon any gross terminology-mangling, I'm not very familiar with Ceilometer) We already use notifications to send what I'd term sparse events that only include the affected stack ID, meaning there's not much to slice on in Ceilometer. This could be extended to richer JSON events that include the stack, resources affected in the update, stats like num-deleted-resources or num-replaced-resources, autoscaling actions, and info about stack errors. Is there a way for users as-is to view those raw notifications, not just the indexed k/v pairs? Thanks, Ryan [1]: https://review.openstack.org/#/c/167793/ -- Ryan Brown / Software Engineer, Openstack / Red Hat, Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
This could be extended to richer JSON events that include the stack, resources affected in the update, stats like num-deleted-resources or num-replaced-resources, autoscaling actions, and info about stack errors. Is there a way for users as-is to view those raw notifications, not just the indexed k/v pairs? from Ceilometer POV, currently, raw notifications are optionally stored (for auditing, postmortem analysis, etc...) but they are not queryable from the api (we don't support it as the performance will [arguably] suffer depending on the backend). the basic structure of an Event in Ceilometer is: id, timestamp, event_type, a list of traits (queryable k/v pairs pulled from notification), and raw (full json notifications) in Liberty, we intend on support action/alerting on events so maybe it's something we should collaborate on to ensure the right functionality is provided. cheers, gord __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
Tooling in general seems to be moving towards richer event data as well.
The logging tools (Loggly/Logstash/PaperTrail/zillions of others) are
intended to take your unstructured logs and turn them into events, so
why not have Heat output structured events that we can present to the
user with Ceilometer rather than sending log lines (through syslog or
otherwise) and using tooling to reassemble them into events later.
The trend in the monitoring space seems to be:
1. Alarms are issued from Metrics as Events.
(events can issue alarms too, but conventional alarming is metric based)
2. Multiple events are analyzed to produce Metrics (stream processing)
3. Go to Step 1
TL;DR: I think what we really want is a place to send and react to
*events*. Logs are a way to do that, of course, but the Ceilometer way
sounds pretty attractive.
The difference is structured vs. unstructured data. Elasticsearch-based
solutions tend to ignore structure by looking at keywords. Some solutions,
like TopLog, infer a structure by gleaning regexs from logs.
Events start as structured data. More so, we're looking at establishing
AVRO-based schema definitions on top of these events (slow progress).
If anything we should consider changing the logging library to use structured
messages. Specifically:
log(The %(foo)s did %(thing)s % {'foo':'x', 'thing':'action'})
it would be
log({'message':The %(foo)s did %(thing)s, 'foo':'x', 'thing':'action'})
Which can still be formatted for conventional logs, but also sent as a
notification or as a higher-level structure to things like ES, TopLog, etc.
The driver can decide.
* CloudWatch has you send unstructured log messages, then build filters
to parse them into quantifiable events, then set alarms on those metrics.
Having to build filters is a relatively error-prone approach compared to the
methods described above.
[snip]
The Fujitsu team have already added logging support to Monasca (with an
elasticsearch backend) and HP is currently adding StackTach.v3 support for
notification-event conversion as well as our Winchester event stream
processing engine. Also, this is based on Kafka vs. RabbitMQ, which has better
scaling characteristics for this kind of data.
This could be extended to richer JSON events that include the stack,
resources affected in the update, stats like num-deleted-resources or
num-replaced-resources, autoscaling actions, and info about stack errors.
Some of these sound more like a metrics than notifications. We should be
careful not to misuse the two.
Is there a way for users as-is to view those raw notifications, not just
the indexed k/v pairs?
In StackTach.v3 we ship the raw notifications to HDFS for archiving, but
expose the reduced event via the API. The message-id links the two.
Lots more here: http://www.stacktach.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
On 04/07/2015 11:34 AM, Sandy Walsh wrote:
log({'message':The %(foo)s did %(thing)s, 'foo':'x', 'thing':'action'})
I pity the foo that did that thing.
-jay
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
On Tue, Apr 7, 2015 at 1:53 PM, Chris Friesen chris.frie...@windriver.com wrote: On 04/06/2015 08:55 PM, Angus Salkeld wrote: Hi all For quite some time we (Heat team) have wanted to be able to send messages to our users (by user I do not mean the Operator, but the User that is interacting with the client). What do I mean by user messages, and how do they differ from our current log messages and notifications? - Our current logs are for the operator and have information that the user should not have (ip addresses, hostnames, configuration options, other tenant info etc..) - Our notifications (that Ceilometer uses) *could* be used, but I am not sure if it quite fits. (they seem a bit heavy weight for a log message and aimed at higher level events) snip What are some options we could investigate: 1. remote syslog 2. Zaqar 3. Other options: Please chip in with suggestions/links! What about a per-user notification topic using the existing notification backend? Wouldn't that require the Operator to provide the end user with access to the message bus? Seems scary to me. -Angus Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
On 04/06/2015 08:55 PM, Angus Salkeld wrote: Hi all For quite some time we (Heat team) have wanted to be able to send messages to our users (by user I do not mean the Operator, but the User that is interacting with the client). What do I mean by user messages, and how do they differ from our current log messages and notifications? - Our current logs are for the operator and have information that the user should not have (ip addresses, hostnames, configuration options, other tenant info etc..) - Our notifications (that Ceilometer uses) *could* be used, but I am not sure if it quite fits. (they seem a bit heavy weight for a log message and aimed at higher level events) snip What are some options we could investigate: 1. remote syslog 2. Zaqar 3. Other options: Please chip in with suggestions/links! What about a per-user notification topic using the existing notification backend? Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] how to send messages (and events) to our users
On 04/06/2015 10:08 PM, Angus Salkeld wrote: On Tue, Apr 7, 2015 at 1:53 PM, Chris Friesen chris.frie...@windriver.com mailto:chris.frie...@windriver.com wrote: On 04/06/2015 08:55 PM, Angus Salkeld wrote: Hi all For quite some time we (Heat team) have wanted to be able to send messages to our users (by user I do not mean the Operator, but the User that is interacting with the client). What do I mean by user messages, and how do they differ from our current log messages and notifications? - Our current logs are for the operator and have information that the user should not have (ip addresses, hostnames, configuration options, other tenant info etc..) - Our notifications (that Ceilometer uses) *could* be used, but I am not sure if it quite fits. (they seem a bit heavy weight for a log message and aimed at higher level events) snip What are some options we could investigate: 1. remote syslog 2. Zaqar 3. Other options: Please chip in with suggestions/links! What about a per-user notification topic using the existing notification backend? Wouldn't that require the Operator to provide the end user with access to the message bus? Seems scary to me. AMQP supports access controls, so is it really all that scary? Maybe set up a virtual host per user if we want to be paranoid? (Just throwing it out there as an option since we're already using it...) Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
