Re: Send Juju logs to different database?
On 6 May 2015 at 04:57, Menno Smits menno.sm...@canonical.com wrote: It is more likely that Juju will grow the ability to send logs to external log services using the syslog protocol (and perhaps others). You could use this to log to your own log aggregator or database. This feature has been discussed but hasn't been planned in any detail yet (pull requests would be most welcome!). syslog seems a bad fit, as the logs are now structured data and I'd like to keep it that way. I guess people want it as an option, but I'd consider it the legacy option here. My own use case would be to make a more readable debug-logs, rather than attempting to parse the debug-logs output ;) Hmm... I may be able to do this already via the Juju API. -- Stuart Bishop stuart.bis...@canonical.com -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Send Juju logs to different database?
I'll also note that the issues with stale reads mentioned in the call-me-maybe article don't really apply at all for something like logging. They also don't really apply for how we are using Mongo today. I don't think Mongo is the best DB for append-only (delete in bulk infrequently) logging, but it certainly suffices for the amount of data we are processing today. The stale-reads mentioned in the article is much more about a single point of data that multiple sources are trying to change concurrently, which isn't how we interact with mongo from Juju today. For Logging, you are only appending data, so you aren't mutating data in place, thus there isn't really a stale read problem, as the data is either there or not, it can't be partially out of date. For other Juju state data, we use client-side transactions on top of Mongo for most of our operations, and while I haven't gone through every possible interaction, it should insulate us from this particular problem. (if you were partway through a write and master got split brain, when we switch to a new master we should either finish committing, or rollback transactions that were in progress. Because every transaction requires multiple layers of writing before it is completed, in a split brain situation the master cannot return that the transaction is successful without coordination with its peers.) John =:- On Wed, May 6, 2015 at 1:57 AM, Menno Smits menno.sm...@canonical.com wrote: On 6 May 2015 at 06:49, Lauri Ojansivu x...@xet7.org wrote: Hi, today at Ubuntu Developer Summit: http://summit.ubuntu.com/uos-1505/meeting/22437/juju-share-and-juju-sos/ Speakers talked about changing all logging to go to MongoDB, so I asked question at IRC: xet7 Would it be possible to use some different database than MongoDB for logs, because of current problems in MongoDB ? https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-reads It is more likely that Juju will grow the ability to send logs to external log services using the syslog protocol (and perhaps others). You could use this to log to your own log aggregator or database. This feature has been discussed but hasn't been planned in any detail yet (pull requests would be most welcome!). The changes currently being made to Juju to support logging to MongoDB also lay the groundwork for logging to external services. This should land in the coming weeks. - Menno -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Send Juju logs to different database?
On 6 May 2015 at 19:53, Stuart Bishop stuart.bis...@canonical.com wrote: On 6 May 2015 at 04:57, Menno Smits menno.sm...@canonical.com wrote: It is more likely that Juju will grow the ability to send logs to external log services using the syslog protocol (and perhaps others). You could use this to log to your own log aggregator or database. This feature has been discussed but hasn't been planned in any detail yet (pull requests would be most welcome!). syslog seems a bad fit, as the logs are now structured data and I'd like to keep it that way. I guess people want it as an option, but I'd consider it the legacy option here. You're right. Structured logs would be much more useful. My own use case would be to make a more readable debug-logs, rather than attempting to parse the debug-logs output ;) Hmm... I may be able to do this already via the Juju API. Not yet. The API used by debug-log currently emits formatted log lines, not structured log data. -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Send Juju logs to different database?
On 7 May 2015 at 00:58, Charles Butler charles.but...@canonical.com wrote: On Wed, May 6, 2015 at 3:53 AM, Stuart Bishop stuart.bis...@canonical.com wrote: My own use case would be to make a more readable debug-logs, rather than attempting to parse the debug-logs output ;) Hmm... I may be able to do this already via the Juju API. Personally - i feel like this is a *great* use case for an opengrok filter w/ logstash. Translate that structured data into something meaningful in elasticsearch and search/aggregate/tail like a champ. Agreed - that would be awesome. When we get to this we should aim to have Juju emit structured log data that could be used to feed things like logstash. If we do it right then hopefully grok won't be needed because the log data will already be structured. No promises about when this might happen though. At least the logging work currently being done provides a good basis to add logging to external systems. -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Send Juju logs to different database?
On 6 May 2015 at 06:49, Lauri Ojansivu x...@xet7.org wrote: Hi, today at Ubuntu Developer Summit: http://summit.ubuntu.com/uos-1505/meeting/22437/juju-share-and-juju-sos/ Speakers talked about changing all logging to go to MongoDB, so I asked question at IRC: xet7 Would it be possible to use some different database than MongoDB for logs, because of current problems in MongoDB ? https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-reads It is more likely that Juju will grow the ability to send logs to external log services using the syslog protocol (and perhaps others). You could use this to log to your own log aggregator or database. This feature has been discussed but hasn't been planned in any detail yet (pull requests would be most welcome!). The changes currently being made to Juju to support logging to MongoDB also lay the groundwork for logging to external services. This should land in the coming weeks. - Menno -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Send Juju logs to different database?
Hi, today at Ubuntu Developer Summit: http://summit.ubuntu.com/uos-1505/meeting/22437/juju-share-and-juju-sos/ Speakers talked about changing all logging to go to MongoDB, so I asked question at IRC: xet7 Would it be possible to use some different database than MongoDB for logs, because of current problems in MongoDB ? https://aphyr.com/posts/322-call-me-maybe-mongodb-stale-reads I also read from juju website that you are readying for MongoDB 3.0: http://blog.labix.org/2015/01/24/readying-mgo-for-mongodb-3-0 And also that some have switched from MongoDB to PostgreSQL: https://news.ycombinator.com/item?id=9178773 http://developer.olery.com/blog/goodbye-mongodb-hello-postgresql/ So my questions are: 1) Would it be possible to use for example PostgreSQL for logs etc? 2) Are the issues in mentioned articles linked above still valid in MongoDB 3.0 ? BR, Lauri -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
