On 01/02/2014 09:44 PM, Clark Boylan wrote:
This is great stuff. Out of curiousity is doing the graphing with
pandas and ES vs graphite so that we can graph things in a more ad hoc

So, we need to go to ES for the fingerprints anyway (because that's where we mine them from), which means we need a way to process ES data into TimeSeries. In order to calculate frequencies we need largely equivalent TimeSeries that are base lines for # of jobs run of particular types. Given that we can get that with an ES query, it prevents the need of having to have a different data transformation process to get to the same kind of TimeSeries.

It also lets us bulk query. With 1 ~20second ES query we get all states, of all jobs, across all queues, over the last 7 days (as well as information on review). And the transform to slice is super easy because it's 10s of thousands of records that are dictionaries, which makes for good input. You'd need to do a bunch of unbinning and transforms to massage the graphite data to pair with what we have in the fingerprint data.

Eventually having tools to do the same thing with graphite is probe ably a good thing, largely for other analysis people want to do on that (I think long term having some data kits for our bulk data to let people play with it is goodness). I'd just put it after a 1.0 as I think it's not really needed.

Also, for the dashboard, Kibana3 does a lot more stuff than
Kibana2 which we currently use. I have been meaning to get Kibana3
running alongside Kibana2 and I think it may be able to do multi line
queries (I need to double check that but it has a lot more query and
graphing capability). I think Kibana3 is worth looking into as well
before we go too far down the road of custom UI.

Absolutely. There is a reason that's all the way at the bottom of the list, and honestly, something I almost didn't put in there. But I figured we needed to understand the implications of multi line matches with our current UI, and the fact that they will make some things better, but discovering those matches will be harder with the existing UI.

If Kibana3 solves it, score. One less thing to do. Because I'd really like to not be in the business of maintaining a custom web UI just for discovery of fingerprints.


Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com

OpenStack-dev mailing list

Reply via email to