Re: BigQuery TIMESTAMP and TimestampedValue()

2020-01-23 Thread Sandy Walsh
hey > change/fix this it could break your code. > > Kenn > > On Mon, Jan 20, 2020 at 2:45 PM Sandy Walsh wrote: > >> [image: :wave:] Newb here for what will certainly be the first of many >> silly questions ... >> >> I'm working on a dataflow pipeline

BigQuery TIMESTAMP and TimestampedValue()

2020-01-20 Thread Sandy Walsh
[image: :wave:] Newb here for what will certainly be the first of many silly questions ... I'm working on a dataflow pipeline using python SDK (local runners currently). It's a bounded source from BigQuery. One column is a TIMESTAMP. I'm trying to assign the timestamp using beam.window.Timestampe

[Openstack-operators] StackTach.v3 now in production ...

2015-04-22 Thread Sandy Walsh
(sorry for cross-post, but this is appropriate to both audiences) Hey y'all! For those of you that don't know, StackTach is a notification-based debugging, monitoring and usage tool for OpenStack. We're happy to announce that we've recently rolled StackTach.v3 into production at one of the Rax

[openstack-dev] StackTach.v3 now in production ...

2015-04-22 Thread Sandy Walsh
(sorry for cross-post, but this is appropriate to both audiences) Hey y'all! For those of you that don't know, StackTach is a notification-based debugging, monitoring and usage tool for OpenStack. We're happy to announce that we've recently rolled StackTach.v3 into production at one of the Rax

Re: [openstack-dev] [nova][oslo_messaging]Enabling notifications from nova

2015-04-16 Thread Sandy Walsh
http://www.stacktach.com/about.html#enabling? From: Eduard Matei Sent: Thursday, April 16, 2015 6:04 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [nova][oslo_messaging]Enabling notifications from nova Hi, I've

Re: [openstack-dev] [all] how to send messages (and events) to our users

2015-04-09 Thread Sandy Walsh
>From: Angus Salkeld >Sent: Wednesday, April 8, 2015 8:24 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] [all] how to send messages (and events) to our >users > > >I also want to point out that what I'd actually rather see is that all >of the se

Re: [openstack-dev] [all] how to send messages (and events) to our users

2015-04-08 Thread Sandy Walsh
Yeah, I don't think anyone would give access to the production rabbitmq directly. We use Yagi [1] to pipe it to AtomHopper [2] for downstream consumption/sanitizing. -S [1] https://github.com/rackerlabs/yagi [2] http://atomhopper.org/ From: Duncan Thomas

Re: [openstack-dev] [all] how to send messages (and events) to our users

2015-04-08 Thread Sandy Walsh
>From: Clint Byrum >Sent: Wednesday, April 8, 2015 1:15 PM > >There's this: > >https://wiki.openstack.org/wiki/Cue Hmm, that looks interesting. Will read. >I also want to point out that what I'd actually rather see is that all >of the services provide fun

Re: [openstack-dev] [all] how to send messages (and events) to our users

2015-04-08 Thread Sandy Walsh
>From: Ryan Brown >Sent: Wednesday, April 8, 2015 9:42 AM > >> The trend in the monitoring space seems to be: >> >> 1. Alarms are issued from Metrics as Events. >> (events can issue alarms too, but conventional alarming is metric based) >> 2. Multiple events are analyzed to produce Metrics (st

Re: [openstack-dev] [all] how to send messages (and events) to our users

2015-04-07 Thread Sandy Walsh
>Tooling in general seems to be moving towards richer event data as well. >The logging tools (Loggly/Logstash/PaperTrail/zillions of others) are >intended to take your unstructured logs and turn them into events, so >why not have Heat output structured events that we can present to the >user with

Re: [openstack-dev] [all] how to send messages (and events) to our users

2015-04-07 Thread Sandy Walsh
Notifications were added for this very purpose. At Rax, we have a downstream consumer (yagi) that routes notifications to an appropriate ATOM/pubsub feed (based on tenant-id). Notifications are only as heavy as you want to make them. The only required fields are event_type and timestamp. ?

Re: [openstack-dev] [stacktach] [oslo] stachtach -> kombu -> pika ??

2015-03-16 Thread Sandy Walsh
> >From: Sean Dague >Sent: Thursday, March 12, 2015 9:09 AM >To: openstack-dev@lists.openstack.org >Subject: Re: [openstack-dev] [stacktach] [oslo] stachtach -> kombu -> pika ?? > >On 03/10/2015 12:56 PM, Joshua Harlow wrote: >> I saw the following on >> htt

Re: [openstack-dev] [stacktach] [oslo] stachtach -> kombu -> pika ??

2015-03-10 Thread Sandy Walsh
No, we're adding this to Yagi first and perhaps Notabene later. We don't need rpc support, so it's too big a change for us to take on. From: gordon chung Sent: Tuesday, March 10, 2015 3:58 PM To: OpenStack Development Mailing List not for usage questions Subjec

Re: [openstack-dev] [stacktach] [oslo] stachtach -> kombu -> pika ??

2015-03-10 Thread Sandy Walsh
Hey J, Our (old) notification consumer was using carrot, which is dead but worked. Lately though there have been conflicts with carrot and msgpack, so we had to change. Around the same time, we ran into a bug where we were writing to an unnamed exchange (completely valid, but too easy to do und

Re: [openstack-dev] [oslo_messaging]Notification listener for nova

2015-03-04 Thread Sandy Walsh
I assume you have notifications enabled in nova.conf? See the "Enabling notifications in OpenStack" section http://www.stacktach.com/about.html Oh, there are couple more flags that could be useful too: notify_on_state_change=vm_and_task_state notify_on_any_change=True ? __

Re: [Openstack] Action notification in nova for delete an instance

2015-03-02 Thread Sandy Walsh
Enable notifications on the compute nodes and you'll get compute.instance.create.end and compute.instance.delete.end notifications for these operations. It sounds like you only have them enabled on the scheduler/api nodes currently. https://wiki.openstack.org/wiki/SystemUsageData __

[Openstack-operators] StackTach.v3 docs ...

2015-02-26 Thread Sandy Walsh
First pass at new docs are available here http://www.stacktach.com/ (API and glossary to follow) Feedback and patches welcome! -Sandy ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailm

[openstack-dev] StackTach.v3 docs ...

2015-02-26 Thread Sandy Walsh
First pass at new docs are available here http://www.stacktach.com/ (API and glossary to follow) Feedback and patches welcome! -Sandy __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-de

Re: [openstack-dev] [oslo.messaging] extending notification MessagingDriver

2015-02-26 Thread Sandy Walsh
Cool, I'm interested in creating some notification drivers outside of olso-messaging as well (for Kafka support and schema-based notifications). Do you have a repo started on this? I'd be keen to have a look. -S From: Boden Russell Sent: Thursday, Febr

Re: [Openstack] Prometheus and open stack

2015-02-26 Thread Sandy Walsh
?Been keeping an eye on that project, but don't know of anyone running it against OpenStack. Certainly shows promise though. You may want to look at Monasca first though ... a little more tailored to the OpenStack world. The monitoring/logging and big data world seem to be circling on Kafka-ba

Re: [Openstack-operators] [Ceilometer] Real world experience with Ceilometer deployments - Feedback requested

2015-02-12 Thread Sandy Walsh
Yagi [1] is a really easy way to consume notifications and do stuff with them. We use it as the basis for our STv3 consumption. Very easy to write a Handler (fill in the handle_events() method) and you're off to the races. Yagi has a proper worker than can be daemonized and flexible logging.

Re: [Openstack-operators] [Ceilometer] Real world experience with Ceilometer deployments - Feedback requested

2015-02-12 Thread Sandy Walsh
btw> if you want to know how StackTach handles billing, here's the salient part from our Hong Kong presentation [1]. Back when we were attempting Ceilometer integration. [1] http://youtu.be/c8zZtSL0t00?t=8m26s ___ OpenStack-operators mailing list OpenS

Re: [Openstack-operators] [Ceilometer] Real world experience with Ceilometer deployments - Feedback requested

2015-02-12 Thread Sandy Walsh
Hey Tim! Thanks for the mention. I'm keen to hear the responses on this as well. I haven't been very active on the ML recently, so perhaps it's a good time for an update (or an intro for those not familiar with StackTach [1]) StackTach started out as a diagnostics tool. It consumes notification

Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-01-30 Thread Sandy Walsh
> >From: Johannes Erdfelt [johan...@erdfelt.com] >Sent: Thursday, January 29, 2015 9:18 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues > >On Thu, Jan 29, 2

Re: [openstack-dev] Notification Schemas ...

2015-01-19 Thread Sandy Walsh
n 01/18/2015 04:39 PM, Sandy Walsh wrote: > > > Eddie Sheffield has pulled together a strawman set of notification > > schemas for Nova and Glance. > > > > https://github.com/StackTach/notification-schemas > > Some important things that I see are missing, so far...

[openstack-dev] Notification Schemas ...

2015-01-18 Thread Sandy Walsh
Hey y'all Eddie Sheffield has pulled together a strawman set of notification schemas for Nova and Glance. Looks like a great start for further discussion. He's going to add JSON-Schema validation next as a form of unit test. Then I guess we have to start thinking about a library to digest these

Re: [openstack-dev] Where should Schema files live?

2014-12-08 Thread Sandy Walsh
>From: Sandy Walsh [sandy.wa...@rackspace.com] Monday, December 01, 2014 9:29 AM > >>From: Duncan Thomas [duncan.tho...@gmail.com] >>Sent: Sunday, November 30, 2014 5:40 AM >>To: OpenStack Development Mailing List >>Subject: Re: [openstack-dev] Where should Schema

Re: [openstack-dev] Where should Schema files live?

2014-12-01 Thread Sandy Walsh
>From: Duncan Thomas [duncan.tho...@gmail.com] >Sent: Sunday, November 30, 2014 5:40 AM >To: OpenStack Development Mailing List >Subject: Re: [openstack-dev] Where should Schema files live? > >Duncan Thomas >On Nov 27, 2014 10:32 PM, "Sandy Walsh" wrote: >>

Re: [openstack-dev] [nova] is there a way to simulate thousands or millions of compute nodes?

2014-11-27 Thread Sandy Walsh
>From: Michael Still [mi...@stillhq.com] Thursday, November 27, 2014 6:57 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] [nova] is there a way to simulate thousands or >millions of compute nodes? > >I would say that supporting millions of compute

Re: [openstack-dev] Where should Schema files live?

2014-11-27 Thread Sandy Walsh
>From: Eoghan Glynn [egl...@redhat.com] Tuesday, November 25, 2014 1:49 PM >To: OpenStack Development Mailing List (not for usage questions) >Subject: Re: [openstack-dev] Where should Schema files live? > >> I think Doug's suggestion of keeping the schema files in-tree and pushing >> them to a well

Re: [openstack-dev] Where should Schema files live?

2014-11-27 Thread Sandy Walsh
>From: Jay Pipes [jaypi...@gmail.com] Wednesday, November 26, 2014 5:22 PM >On 11/20/2014 08:12 AM, Sandy Walsh wrote: >> >> We're early in thinking about these problems, but would like to start the >> conversation now to get your opinions. >> > >OK, so th

Re: [openstack-dev] Where should Schema files live?

2014-11-24 Thread Sandy Walsh
>From: Eoghan Glynn [egl...@redhat.com] Friday, November 21, 2014 11:03 AM >> >> Some problems / options: >> >> a. Unlike Python, there is no simple pip install for text files. No >> >> version control per se. Basically whatever we pull from the repo. The >> >> problem with a git clone is we need t

Re: [openstack-dev] Where should Schema files live?

2014-11-21 Thread Sandy Walsh
>From: Doug Hellmann [d...@doughellmann.com] Thursday, November 20, 2014 5:09 PM >On Nov 20, 2014, at 3:40 PM, Sandy Walsh wrote: >> From: Doug Hellmann [d...@doughellmann.com] Thursday, November 20, 2014 3:51 >> PM >>> On Nov 20, 2014, at 8:12 AM, Sandy Walsh

Re: [openstack-dev] Where should Schema files live?

2014-11-21 Thread Sandy Walsh
> >From: Eoghan Glynn [egl...@redhat.com] Thursday, November 20, 2014 5:34 PM > >Some questions/observations inline. > >> Hey y'all, >> >> To avoid cross-posting, please inform your -infra / -operations buddies >> about this post. >> >> We've just started th

Re: [openstack-dev] Where should Schema files live?

2014-11-20 Thread Sandy Walsh
From: Doug Hellmann [d...@doughellmann.com] Thursday, November 20, 2014 3:51 PM >On Nov 20, 2014, at 8:12 AM, Sandy Walsh wrote: >> Hey y'all, >> >> To avoid cross-posting, please inform your -infra / -operations buddies about this post. >> >

[openstack-dev] Where should Schema files live?

2014-11-20 Thread Sandy Walsh
Hey y'all, To avoid cross-posting, please inform your -infra / -operations buddies about this post. We've just started thinking about where notification schema files should live and how they should be deployed. Kind of a tricky problem. We could really use your input on this problem ... The

[openstack-dev] Summit Recap: Notification Schema

2014-11-18 Thread Sandy Walsh
https://etherpad.openstack.org/p/kilo-crossproject-notifications The big takeaways: 1. We want the schema to be external so other languages can utilize them. 2. JSON-Schema seems fine, but AVRO has traction in the Big Data world and should be considered. 3. The challenge of have text-file based

Re: [openstack-dev] [Ceilometer] Notifications as a contract summit prep

2014-10-24 Thread Sandy Walsh
Thanks ... we'll be sure to address your concerns. And there's the list we've compiled here: https://etherpad.openstack.org/p/kilo-crossproject-summit-topics (section 4) -S From: Chris Dent [chd...@redhat.com] Sent: Friday, October 24, 2014 2:45 PM To: O

[openstack-dev] StackTach users?

2014-10-24 Thread Sandy Walsh
Hey y'all! I'm taking a page from Angus and trying to pull together a list of StackTach users. We're moving quickly on our V3 implementation and I'd like to ensure we're addressing the problems you've faced/are facing with older versions. For example, I know initial setup has been a concern an

Re: [openstack-dev] [all] How can we get more feedback from users?

2014-10-24 Thread Sandy Walsh
Nice work Angus ... great idea. Would love to see more of this. -S From: Angus Salkeld [asalk...@mirantis.com] Sent: Friday, October 24, 2014 1:32 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [all] How can we get mor

Re: [openstack-dev] [oslo] request_id deprecation strategy question

2014-10-20 Thread Sandy Walsh
20, 2014 at 02:17:54PM +, Sandy Walsh wrote: > Does this mean we're losing request-id's? No, it just means the implementation has moved from oslo-incubator[1] to oslo.middleware[2]. The issue I'm highlighting is that those projects using the code now have to update their a

Re: [openstack-dev] [oslo] request_id deprecation strategy question

2014-10-20 Thread Sandy Walsh
Does this mean we're losing request-id's? Will they still appear in the Context objects? And there was the effort to keep consistent request-id's in cross-service requests, will this deprecation affect that? -S From: Steven Hardy [sha...@redhat.com] Sen

Re: [openstack-dev] Treating notifications as a contract

2014-10-14 Thread Sandy Walsh
From: Doug Hellmann [d...@doughellmann.com] Tuesday, October 14, 2014 7:19 PM > It might be more appropriate to put it on the cross-project session list: > https://etherpad.openstack.org/p/kilo-crossproject-summit-topics Done ... thanks! ___ OpenStack

Re: [Openstack] Messaging reliability/durability expectations

2014-10-14 Thread Sandy Walsh
Sort of. Openstack RPC-over-AMQP (oslo.messaging) automatically ack()'s all messages that are received. So, it becomes the responsibility of the sender to retry. For example, the scheduler in Nova does this. However, if the client fails before getting the message, AMQP will automatically requeu

Re: [openstack-dev] Treating notifications as a contract

2014-10-07 Thread Sandy Walsh
> >From: Sandy Walsh [sandy.wa...@rackspace.com] Tuesday, October 07, 2014 6:07 PM > >Haven't had any time to get anything written down (pressing deadlines with >StackTach.v3) but open to suggestions. Perhaps we should just add something to >the olso.messaging etherpad to

Re: [openstack-dev] Treating notifications as a contract

2014-10-07 Thread Sandy Walsh
>From: Chris Dent [chd...@redhat.com] Tuesday, October 07, 2014 12:07 PM > >On Wed, 3 Sep 2014, Sandy Walsh wrote: > >> Good goals. When Producer and Consumer know what to expect, things are >> good ... "I know to find the Instance ID ". When the consumer >>

Re: [Question #255075]: End of body / message terminator in .txt archive?

2014-09-28 Thread Sandy Walsh
Question #255075 on mailman in Ubuntu changed: https://answers.launchpad.net/ubuntu/+source/mailman/+question/255075 Status: Answered => Solved Sandy Walsh confirmed that the question is solved: Thanks Mark ... you've been a great help. Will do. Btw> found the proper url for the

Re: [Question #255075]: End of body / message terminator in .txt archive?

2014-09-28 Thread Sandy Walsh
Question #255075 on mailman in Ubuntu changed: https://answers.launchpad.net/ubuntu/+source/mailman/+question/255075 Status: Needs information => Open Sandy Walsh gave more information on the question: Thanks Mark, I don't have admin privileges to the list, so the various mbox opt

Re: [Question #255075]: End of body / message terminator in .txt archive?

2014-09-27 Thread Sandy Walsh
Question #255075 on mailman in Ubuntu changed: https://answers.launchpad.net/ubuntu/+source/mailman/+question/255075 Sandy Walsh gave more information on the question: Likewise for This is the first body line >From here to eternity third This is the forth body line as per h

Re: [Question #255075]: End of body / message terminator in .txt archive?

2014-09-27 Thread Sandy Walsh
Question #255075 on mailman in Ubuntu changed: https://answers.launchpad.net/ubuntu/+source/mailman/+question/255075 Status: Solved => Open Sandy Walsh is still having a problem: Hmm, close ... but it seems leading "From " in the body of the message is not getting escaped. This

Re: [Question #255075]: End of body / message terminator in .txt archive?

2014-09-27 Thread Sandy Walsh
Question #255075 on mailman in Ubuntu changed: https://answers.launchpad.net/ubuntu/+source/mailman/+question/255075 Sandy Walsh posted a new comment: http://en.wikipedia.org/wiki/Mbox -- You received this question notification because you are a member of Mailman Coders, which is an answer

Re: [Question #255075]: End of body / message terminator in .txt archive?

2014-09-27 Thread Sandy Walsh
Question #255075 on mailman in Ubuntu changed: https://answers.launchpad.net/ubuntu/+source/mailman/+question/255075 Status: Open => Solved Sandy Walsh confirmed that the question is solved: nm, I should have read this closer. There is no EOM marker, just a BOM marker. "From &qu

[Question #255075]: End of body / message terminator in .txt archive?

2014-09-27 Thread Sandy Walsh
New question #255075 on mailman in Ubuntu: https://answers.launchpad.net/ubuntu/+source/mailman/+question/255075 I have the .gz archives for my list, which contains the single .txt file. I'm trying to determine the terminator for the body of each message? I know the blank line after the headers

Re: [Openstack] Billing System Recommendation

2014-09-24 Thread Sandy Walsh
While not a billing system. Rackspace (and some others) use StackTach to collect accurate (and verified) usage data from Nova and Glance. https://github.com/stackforge/stacktach We're getting close to GA for our new StackTach architecture (v3), which we will be demoing at the Paris summit. If

Re: [openstack-dev] [Nova] - do we need .start and .end notifications in all cases ?

2014-09-22 Thread Sandy Walsh
> >From: Jay Pipes [jaypi...@gmail.com] >Sent: Monday, September 22, 2014 11:51 AM >To: openstack-dev@lists.openstack.org >Subject: Re: [openstack-dev] [Nova] - do we need .start and .end notifications >in all cases ? > >On 09/22/20

Re: [openstack-dev] [Nova] - do we need .start and .end notifications in all cases ?

2014-09-22 Thread Sandy Walsh
+1, the high-level code should deal with top-level exceptions and generate .error notifications (though it's a little spotty). Ideally we shouldn't need three events for simple operations. The use of .start/.end vs. logging is a bit of a blurry line. At its heart a notification should provide

Re: [openstack-dev] [Nova] - do we need .start and .end notifications in all cases ?

2014-09-22 Thread Sandy Walsh
Hey Phil, (sorry for top-post, web client) There's no firm rule for requiring .start/.end and I think your criteria defines it well. Long running transactions (or multi complex-step transactions). The main motivator behind .start/.end code was .error notifications not getting generated in many

Re: [OpenStack-Infra] Setting the bar higher for stackforge

2014-09-15 Thread Sandy Walsh
> >From: Jeremy Stanley [fu...@yuggoth.org] >Sent: Monday, September 15, 2014 2:51 PM > >On 2014-09-15 17:25:48 + (+0000), Sandy Walsh wrote: >> Hmm, my understanding was that branches are not accepted without a >> signed IC

Re: [OpenStack-Infra] Setting the bar higher for stackforge

2014-09-15 Thread Sandy Walsh
> >From: Jeremy Stanley [fu...@yuggoth.org] >Sent: Monday, September 15, 2014 2:05 PM > >On 2014-09-15 16:47:13 + (+0000), Sandy Walsh wrote: >> It's the Corporate CLA that's needed. The companies that want to >> cont

Re: [OpenStack-Infra] Setting the bar higher for stackforge

2014-09-15 Thread Sandy Walsh
>From: Jeremy Stanley [fu...@yuggoth.org] >Sent: Monday, September 15, 2014 1:06 PM > >On 2014-09-15 14:36:59 +0000 (+0000), Sandy Walsh wrote: >> For our group, the greatest value of StackForge is CLA management. >[...] > >Out of curiosity, why is that a value t

Re: [OpenStack-Infra] Setting the bar higher for stackforge

2014-09-15 Thread Sandy Walsh
On 2014-09-15 23:44:39 +1000 (+1000), Joshua Hesketh wrote: >I think it goes without saying that the stackforge programme is an >important asset to OpenStack. However, with our current challenges and >limitations of the gate and resources I wonder if we should set the bar >higher for accepting

Re: [openstack-dev] Treating notifications as a contract

2014-09-10 Thread Sandy Walsh
> Jay Pipes - Wednesday, September 10, 2014 3:56 PM >On 09/03/2014 11:21 AM, Sandy Walsh wrote: >> On 9/3/2014 11:32 AM, Chris Dent wrote: >>> I took some notes on this a few weeks ago and extracted what seemed >>> to be the two main threads or ideas the were reveale

Re: [Openstack] is anyone using zeromq for RPC?

2014-09-06 Thread Sandy Walsh
>From: Doug Hellmann [d...@doughellmann.com] >On Sep 5, 2014, at 11:13 PM, Sandy Walsh wrote: >>> From: Doug Hellmann [d...@doughellmann.com] >>> The zmq driver in oslo.messaging, used for internal communication >>> between OpenStack services, has been without a

Re: [Openstack] is anyone using zeromq for RPC?

2014-09-05 Thread Sandy Walsh
> From: Doug Hellmann [d...@doughellmann.com] >The zmq driver in oslo.messaging, used for internal communication >between OpenStack services, has been without a maintainer for a >significant period of time. It isn’t actively tested, and it isn’t >clear whether or not it works. The Oslo team would l

Re: [Openstack] Zabbix

2014-09-05 Thread Sandy Walsh
From: Michael Mester (micmeste) [micme...@cisco.com] >Guys, >Im curious on who is using Zabbix to monitor their openstack deployment. I >know there is quite a bit of value to having the single pane of glass as well >as the scripting and API capabilities. I hav

Re: [openstack-dev] StackTach.v3 - Screencasts ...

2014-09-05 Thread Sandy Walsh
For those of you playing the home game ... just added four new screencasts to the StackTach.v3 playlist. These are technical deep dives into the code added over the last week or so, with demos. For the more complex topics I spend a little time on the background and rationale. StackTach.v3:

Re: [openstack-dev] Treating notifications as a contract (CADF)

2014-09-04 Thread Sandy Walsh
see them) [1] with the advantages/disadvantages of each approach. It's just a strawman, so bend/spindle/mutilate. Look forward to feedback! -S [1] https://wiki.openstack.org/wiki/NotificationsAndCADF On 9/3/2014 12:30 PM, Sandy Walsh wrote: > On 9/3/2014 11:32 AM, Chris Dent wrote: >&

Re: [openstack-dev] Treating notifications as a contract

2014-09-03 Thread Sandy Walsh
On 9/3/2014 11:32 AM, Chris Dent wrote: > On Wed, 3 Sep 2014, Sandy Walsh wrote: > >> We're chatting with IBM about CADF and getting down to specifics on >> their applicability to notifications. Once I get StackTach.v3 into >> production I'm keen to get star

Re: [openstack-dev] Treating notifications as a contract

2014-09-03 Thread Sandy Walsh
Is there anything slated for the Paris summit around this? I just spent nearly a week parsing Nova notifications and the pain of no schema has overtaken me. We're chatting with IBM about CADF and getting down to specifics on their applicability to notifications. Once I get StackTach.v3 into pr

[openstack-dev] StackTach.v3 - Screencasts ...

2014-08-22 Thread Sandy Walsh
Hey y'all, We've started a screencast series on the StackTach.v3 dev efforts [1]. It's still early-days, so subscribe to the playlist for updates. The videos start with the StackTach/Ceilometer integration presentation at the Hong Kong summit, which is useful for background and motivation but t

Re: [openstack-dev] [all] The future of the integrated release

2014-08-19 Thread Sandy Walsh
On 8/18/2014 9:27 AM, Thierry Carrez wrote: > Clint Byrum wrote: >> Here's why folk are questioning Ceilometer: >> >> Nova is a set of tools to abstract virtualization implementations. >> Neutron is a set of tools to abstract SDN/NFV implementations. >> Cinder is a set of tools to abstract block-de

Re: [openstack-dev] [all] The future of the integrated release

2014-08-16 Thread Sandy Walsh
On 8/16/2014 10:09 AM, Chris Dent wrote: > On Fri, 15 Aug 2014, Sandy Walsh wrote: > >> I recently suggested that the Ceilometer API (and integration tests) >> be separated from the implementation (two repos) so others might plug >> in a different implementation while m

Re: [openstack-dev] [all] The future of the integrated release

2014-08-15 Thread Sandy Walsh
On 8/14/2014 6:42 PM, Doug Hellmann wrote: On Aug 14, 2014, at 4:41 PM, Joe Gordon mailto:joe.gord...@gmail.com>> wrote: On Wed, Aug 13, 2014 at 12:24 PM, Doug Hellmann mailto:d...@doughellmann.com>> wrote: On Aug 13, 2014, at 3:05 PM, Eoghan Glynn mailto:egl...@redhat.com>> wrote: > >>>

Re: [openstack-dev] [nova][core] Expectations of core reviewers

2014-08-15 Thread Sandy Walsh
Maybe we need to think about this from a distributed software perspective? * Divide and Conquer? Can we split the topics to create more manageable sub-groups? This way it's not core-vs-non-core but intererested-vs-moderately-interested. (of course, this is much the way the mailing list works)

Re: [openstack-dev] [nova][core] Expectations of core reviewers

2014-08-14 Thread Sandy Walsh
On 8/14/2014 11:28 AM, Russell Bryant wrote: > On 08/14/2014 10:04 AM, CARVER, PAUL wrote: >> Daniel P. Berrange [mailto:berra...@redhat.com] wrote: >> >>> Depending on the usage needs, I think Google hangouts is a quite useful >>> technology. For many-to-many session its limit of 10 participants c

Re: [openstack-dev] Annoucing CloudKitty : an OpenSource Rating-as-a-Service project for OpenStack

2014-08-14 Thread Sandy Walsh
Sounds very interesting. We're currently collecting detailed (and verified) usage information in StackTach and are keen to see what CloudKitty is able to offer. My one wish is that you keep the components as small pip redistributables with low coupling to promote reuse with other projects. Many

Re: [openstack-dev] [tc][ceilometer] Some background on the gnocchi project

2014-08-11 Thread Sandy Walsh
On 8/11/2014 6:49 PM, Eoghan Glynn wrote: > On 8/11/2014 4:22 PM, Eoghan Glynn wrote: >> Hi Eoghan, >> >> Thanks for the note below. However, one thing the overview below does >> not >> cover is why InfluxDB ( http://influxdb.com/ ) is not being leveraged. >> Many >

Re: [openstack-dev] [tc][ceilometer] Some background on the gnocchi project

2014-08-11 Thread Sandy Walsh
On 8/11/2014 5:29 PM, Eoghan Glynn wrote: > >> On 8/11/2014 4:22 PM, Eoghan Glynn wrote: Hi Eoghan, Thanks for the note below. However, one thing the overview below does not cover is why InfluxDB ( http://influxdb.com/ ) is not being leveraged. Many folks feel that thi

Re: [openstack-dev] [tc][ceilometer] Some background on the gnocchi project

2014-08-11 Thread Sandy Walsh
On 8/11/2014 4:22 PM, Eoghan Glynn wrote: > >> Hi Eoghan, >> >> Thanks for the note below. However, one thing the overview below does not >> cover is why InfluxDB ( http://influxdb.com/ ) is not being leveraged. Many >> folks feel that this technology is a viable solution for the problem space >> d

Re: [openstack-dev] [Ceilometer] Generate Event or Notification in Ceilometer

2014-07-30 Thread Sandy Walsh
If all you want to do is publish a notification you can use oslo.messaging directly. Or, for something lighter weight, we have Notabene, which is a small wrapper on Kombu. An example of how our notification simulator/generator uses it is available here: https://github.com/StackTach/notigen/blob

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-16 Thread Sandy Walsh
On 7/11/2014 6:08 AM, Chris Dent wrote: > On Fri, 11 Jul 2014, Lucas Alvares Gomes wrote: > >> The data format that Ironic will send was part of the spec proposed >> and could have been reviewed. I think there's still time to change it >> tho, if you have a better format talk to Haomeng which is th

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-15 Thread Sandy Walsh
On 7/15/2014 3:51 AM, Mark McLoughlin wrote: > On Fri, 2014-07-11 at 10:04 +0100, Chris Dent wrote: >> On Fri, 11 Jul 2014, Lucas Alvares Gomes wrote: >> >>> The data format that Ironic will send was part of the spec proposed >>> and could have been reviewed. I think there's still time to change it

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Sandy Walsh
On 7/11/2014 6:04 AM, Chris Dent wrote: > On Fri, 11 Jul 2014, Eoghan Glynn wrote: > >> But I guess you're suggesting not only that we version/schematize >> individual notification payloads, but that we do so in a way that's >> global across event types and emitters? > That's partially correct. I'm

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-11 Thread Sandy Walsh
On 7/10/2014 12:10 PM, Chris Dent wrote: > On Thu, 10 Jul 2014, Julien Danjou wrote: > >> My initial plan was to leverage a library like voluptuous to do schema >> based validation on the sender side. That would allow for receiver to >> introspect schema and know the data structure to expect. I did

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Sandy Walsh
On 7/10/2014 2:59 PM, Daniel Dyer wrote: > From my perspective, the requirement is to be able to have a consistent and > predictable format for notifications that are being sent from all services. > This means: > 1. a set of required fields that all events contain and have consistent > meaning >

Re: [openstack-dev] [all] Treating notifications as a contract

2014-07-10 Thread Sandy Walsh
On 7/10/2014 5:52 AM, Eoghan Glynn wrote: > TL;DR: do we need to stabilize notifications behind a versioned >and discoverable contract? Thanks for dusting this off. Versioning and published schemas for notifications are important to the StackTach team. It would be nice to get this resolve

Re: [openstack-dev] [oslo] Openstack and SQLAlchemy

2014-06-30 Thread Sandy Walsh
woot! From: Mike Bayer [mba...@redhat.com] Sent: Monday, June 30, 2014 1:56 PM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [oslo] Openstack and SQLAlchemy Hi all - For those who don't know me, I'm Mike Bayer

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

2014-06-27 Thread Sandy Walsh
7;t need this feature. We should be able to disable it for RPC in oslo.messaging. (I say "should" because I'm not positive some aspect of openstack doesn't depend on the queue existing. Thinking about the scheduler mostly) -S On 06/27/2014 05:16 PM, Sandy Walsh wrote: Some

Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring

2014-06-27 Thread Sandy Walsh
Something to consider is the "create the queue in advance" feature is done for notifications, so we don't drop important messages on the floor by having an Exchange with no associated Queue. For RPC operations, this may not be required (we assume the service is available). If this check is trul

Re: [openstack-dev] [nova] Why is there a 'None' task_state between 'SCHEDULING' & 'BLOCK_DEVICE_MAPPING'?

2014-06-26 Thread Sandy Walsh
Nice ... that's always bugged me. From: wu jiang [win...@gmail.com] Sent: Thursday, June 26, 2014 9:30 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [nova] Why is there a 'None' task_state between 'SCHEDULING' & '

Re: [openstack-dev] [taskflow] Recommendations for the granularity of tasks and their "stickiness" to workers

2014-06-17 Thread Sandy Walsh
On 6/17/2014 7:04 AM, Eoghan Glynn wrote: > Folks, > > A question for the taskflow ninjas. > > Any thoughts on best practice WRT $subject? > > Specifically I have in mind this ceilometer review[1] which adopts > the approach of using very fine-grained tasks (at the level of an > individual alarm ev

Re: [Openstack] [metering] Considerations and Roadmap in Ceilometer (or other component) for Financial Management

2014-06-02 Thread Sandy Walsh
Hi Lee, At Rackspace we use StackTach to gather our billing/usage information (with verification and reconciliation). We are heavy into our v3 development currently. While we haven't addressed the financial aspects of billing it might be something worth talking about, though it's a little fur

Re: [openstack-dev] [Trove][Notifications] Notifications refactoring

2014-05-23 Thread Sandy Walsh
Great to see efforts to ensure we only send useful notifications. On 5/23/2014 7:48 AM, Denis Makogon wrote: Good day, Trove community. I would like to start thread related to Trove notification framework. Notification design was defined as: “Trove will emit events for resources as

Re: [Openstack] Getting eventing information

2014-05-21 Thread Sandy Walsh
Side note: many groups are using StackTach to collect and persist the events and then just accessing the StackTach database directly to generate their usage/performance reports. From: Narayanan, Krishnaprasad [naray...@uni-mainz.de] Sent: Tuesday, May 20, 2014 4

Re: [Openstack] Getting eventing information

2014-05-20 Thread Sandy Walsh
On 5/20/2014 7:32 AM, Narayanan, Krishnaprasad wrote: Hallo all, In our project, I need information about the VM related operations such as creation, deletion, resizing, migration, reboot, etc. In one of my earlier emails dated on 16th April 2014, I received replies from the forum about the ava

Re: [openstack-dev] Monitoring as a Service

2014-05-06 Thread Sandy Walsh
On 5/6/2014 1:48 PM, Thierry Carrez wrote: > Sandy Walsh wrote: >> I'd be curious to know more what "managed" means in this situation? Is >> the core project expected to allocate time in the IRC meeting to the >> concerns of these adjacent projects? What if th

Re: [openstack-dev] Monitoring as a Service

2014-05-06 Thread Sandy Walsh
On 5/6/2014 10:04 AM, Thierry Carrez wrote: > John Dickinson wrote: >> One of the advantages of the program concept within OpenStack is that >> separate code projects with complementary goals can be managed under the >> same program without needing to be the same codebase. The most obvious >> ex

Re: [openstack-dev] How to re-compile Devstack Code

2014-04-24 Thread Sandy Walsh
Also, I find setting this in my localrc/local.conf helps debugging: # get an actual log file vs. screen scrollback LOGFILE=/opt/stack/logs/stack.sh.log # gimme all the info VERBOSE=True # don't pull from git every time I run stack.sh RECLONE=False # make the logs readable LOG_COLOR=False

Re: [Openstack] VM events from OpenStack

2014-04-16 Thread Sandy Walsh
Hi! Nova can publish notifications on the queue whenever anything important happens. https://wiki.openstack.org/wiki/SystemUsageData Tools like YAGI can consume these events and send them to downstream systems. Or you could use something like StackTach to consume them and expose them via a RES

Re: [openstack-dev] Payload within RabbitMQ messages for Nova related exchanges

2014-04-15 Thread Sandy Walsh
On 04/15/2014 10:07 AM, George Monday wrote: > Hey there, > > I've got a quick question about the RabbitMQ exchanges. We are writing > listeners > for the RabbitMQ exchanges. The basic information about the tasks like > compute.instance.create.[start|stop] etc. as stored in the 'payload' > attri

  1   2   3   4   5   6   >