Re: [systemd-devel] Looking for experiences formalizing an API for journal messages
On Wed, Jul 29, 2015 at 03:02:26PM -0400, Anne Mulhern wrote: What I'm wondering about is the existence of some processes (not systemd), that have an agreement on a set of key-value pairs that they communicate with through the journal. There was work done on converting abrt to use the journal. We extended our set of metadata fields for coredumps which were already used internally by coredumpctl (see http://cgit.freedesktop.org/systemd/systemd/commit/?id=3f132692e3). I think this work is not finished yet, because of some issues that abrt would have to copy the coredump file (?), but abrt is becoming an external consumer. fail2ban has a systemd backend which uses the journal. It uses the python API for journal to add matches (the mechanism is general and the matches themselves are specified by filters). This is the same functionality that journalctl uses. HTH, Zbyszek ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Looking for experiences formalizing an API for journal messages
On Wed, 29.07.15 15:02, Anne Mulhern (amulh...@redhat.com) wrote: systemd of courses uses its on its own, and we tried to document the fields we use in systemd.journal-fields(7), though it might be slightly incomplete. But yes, this is indeed API, and deserves complete documentation, like any C API, any D-Bus API or a any REST API would need too. Lennart -- Lennart Poettering, Red Hat What I'm wondering about is the existence of some processes (not systemd), that have an agreement on a set of key-value pairs that they communicate with through the journal. There are multiple bindings for the journal logging API on github for various languages. It might be worth searching for users of these APIs, as well as our C API. libvirt talks the journal logging protocol natively btw, it might be a codebase to check for this. It seems like, even though there is a mechanism for adding additional key/value pairs, regular expression matching is still kind of the norm. Well, yes, regexp searching is definitely more popular. But do note that the journal will implicitly append a big number of structured fields, such as user, process, MAC, unit information, so each record will have a good number of metadata fields already without clients sending the metadata themselves. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Looking for experiences formalizing an API for journal messages
- Original Message - From: Zbigniew Jędrzejewski-Szmek zbys...@in.waw.pl To: Anne Mulhern amulh...@redhat.com Cc: systemd-devel@lists.freedesktop.org Sent: Thursday, July 30, 2015 10:01:54 AM Subject: Re: [systemd-devel] Looking for experiences formalizing an API for journal messages On Wed, Jul 29, 2015 at 03:02:26PM -0400, Anne Mulhern wrote: What I'm wondering about is the existence of some processes (not systemd), that have an agreement on a set of key-value pairs that they communicate with through the journal. There was work done on converting abrt to use the journal. We extended our set of metadata fields for coredumps which were already used internally by coredumpctl (see http://cgit.freedesktop.org/systemd/systemd/commit/?id=3f132692e3). I think this work is not finished yet, because of some issues that abrt would have to copy the coredump file (?), but abrt is becoming an external consumer. Thanks! That's a helpful datapoint. fail2ban has a systemd backend which uses the journal. It uses the python API for journal to add matches (the mechanism is general and the matches themselves are specified by filters). This is the same functionality that journalctl uses. I did take a look at fail2ban. I didn't study it in depth, but it looks like it processes journal entries into a different format and then does regular expression matching on the result. So it didn't really feel like a good example for what I had in mind. I've been exercising the systemd-python package a little and AFAICT it's a pretty straightforward mapping onto the journal C API, with a few extra bits from the journalctl front-end thrown in for convenience. HTH, Zbyszek - mulhern ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Looking for experiences formalizing an API for journal messages
On Thu, Jul 30, 2015 at 10:56:42AM -0400, Anne Mulhern wrote: On Wed, Jul 29, 2015 at 03:02:26PM -0400, Anne Mulhern wrote: What I'm wondering about is the existence of some processes (not systemd), that have an agreement on a set of key-value pairs that they communicate with through the journal. There was work done on converting abrt to use the journal. We extended our set of metadata fields for coredumps which were already used internally by coredumpctl (see http://cgit.freedesktop.org/systemd/systemd/commit/?id=3f132692e3). I think this work is not finished yet, because of some issues that abrt would have to copy the coredump file (?), but abrt is becoming an external consumer. Thanks! That's a helpful datapoint. fail2ban has a systemd backend which uses the journal. It uses the python API for journal to add matches (the mechanism is general and the matches themselves are specified by filters). This is the same functionality that journalctl uses. I did take a look at fail2ban. I didn't study it in depth, but it looks like it processes journal entries into a different format and then does regular expression matching on the result. So it didn't really feel like a good example for what I had in mind. It seems to have (https://github.com/fail2ban/fail2ban/blob/HEAD/fail2ban/protocol.py) [set JAIL addjournalmatch MATCH, adds MATCH to the journal filter of JAIL], so it looks like it is at least possible. I don't know if it is used. I've been exercising the systemd-python package a little and AFAICT it's a pretty straightforward mapping onto the journal C API, with a few extra bits from the journalctl front-end thrown in for convenience. That's a valid description. Zbyszek ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Looking for experiences formalizing an API for journal messages
- Original Message - From: Lennart Poettering lenn...@poettering.net To: Anne Mulhern amulh...@redhat.com Cc: systemd-devel@lists.freedesktop.org Sent: Thursday, July 23, 2015 9:21:44 AM Subject: Re: [systemd-devel] Looking for experiences formalizing an API for journal messages On Thu, 23.07.15 09:17, Anne Mulhern (amulh...@redhat.com) wrote: Hi! We all know that using the journald native API it is possible to enrich the log entry data w/ key/value pairs, although this facility is Linux only. The set of key/value pairs which a message may log to the journal can constitute an API with which a logging entity can communicate alerts to consumers of this information. Clearly this requires an agreement on an API between the logging entity and the consumer of the journal entries. Are there existing projects that have used this facility in a principled, coordinated way with some success or have there been any interesting failures along those same lines? systemd of courses uses its on its own, and we tried to document the fields we use in systemd.journal-fields(7), though it might be slightly incomplete. But yes, this is indeed API, and deserves complete documentation, like any C API, any D-Bus API or a any REST API would need too. Lennart -- Lennart Poettering, Red Hat What I'm wondering about is the existence of some processes (not systemd), that have an agreement on a set of key-value pairs that they communicate with through the journal. It seems like, even though there is a mechanism for adding additional key/value pairs, regular expression matching is still kind of the norm. I was hoping for an inspiring counter example that uses mostly or exclusively key/value pairs instead of expression matching, and maybe some hints as to best practices. - mulhern ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Looking for experiences formalizing an API for journal messages
On 29 July 2015 at 20:02, Anne Mulhern amulh...@redhat.com wrote: - Original Message - From: Lennart Poettering lenn...@poettering.net To: Anne Mulhern amulh...@redhat.com Cc: systemd-devel@lists.freedesktop.org Sent: Thursday, July 23, 2015 9:21:44 AM Subject: Re: [systemd-devel] Looking for experiences formalizing an API for journal messages On Thu, 23.07.15 09:17, Anne Mulhern (amulh...@redhat.com) wrote: Hi! We all know that using the journald native API it is possible to enrich the log entry data w/ key/value pairs, although this facility is Linux only. The set of key/value pairs which a message may log to the journal can constitute an API with which a logging entity can communicate alerts to consumers of this information. Clearly this requires an agreement on an API between the logging entity and the consumer of the journal entries. Are there existing projects that have used this facility in a principled, coordinated way with some success or have there been any interesting failures along those same lines? systemd of courses uses its on its own, and we tried to document the fields we use in systemd.journal-fields(7), though it might be slightly incomplete. But yes, this is indeed API, and deserves complete documentation, like any C API, any D-Bus API or a any REST API would need too. Lennart -- Lennart Poettering, Red Hat What I'm wondering about is the existence of some processes (not systemd), that have an agreement on a set of key-value pairs that they communicate with through the journal. It seems like, even though there is a mechanism for adding additional key/value pairs, regular expression matching is still kind of the norm. I was hoping for an inspiring counter example that uses mostly or exclusively key/value pairs instead of expression matching, and maybe some hints as to best practices. I've heard of https://github.com/rsyslog/liblogging/tree/master/stdlog but didn't look to deep into it. It supposed to do generic structured logging for multiple destinations, one of them being journal. Maybe it's of any help. I don't think that project is stable, nor widely used at the moment. -- Regards, Dimitri. Pura Vida! https://clearlinux.org Open Source Technology Center Intel Corporation (UK) Ltd. - Co. Reg. #1134945 - Pipers Way, Swindon SN3 1RJ. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Looking for experiences formalizing an API for journal messages
On 23 July 2015 at 23:17, Anne Mulhern amulh...@redhat.com wrote: Hi! We all know that using the journald native API it is possible to enrich the log entry data w/ key/value pairs, although this facility is Linux only. The set of key/value pairs which a message may log to the journal can constitute an API with which a logging entity can communicate alerts to consumers of this information. Clearly this requires an agreement on an API between the logging entity and the consumer of the journal entries. Are there existing projects that have used this facility in a principled, coordinated way with some success or have there been any interesting failures along those same lines? Thanks for your help, - mulhern The newer syslog standard (RFC 5424) defines a key/value area. See https://tools.ietf.org/html/rfc5424#section-6.3 ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] Looking for experiences formalizing an API for journal messages
Hi! We all know that using the journald native API it is possible to enrich the log entry data w/ key/value pairs, although this facility is Linux only. The set of key/value pairs which a message may log to the journal can constitute an API with which a logging entity can communicate alerts to consumers of this information. Clearly this requires an agreement on an API between the logging entity and the consumer of the journal entries. Are there existing projects that have used this facility in a principled, coordinated way with some success or have there been any interesting failures along those same lines? Thanks for your help, - mulhern ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Looking for experiences formalizing an API for journal messages
On Thu, 23.07.15 09:17, Anne Mulhern (amulh...@redhat.com) wrote: Hi! We all know that using the journald native API it is possible to enrich the log entry data w/ key/value pairs, although this facility is Linux only. The set of key/value pairs which a message may log to the journal can constitute an API with which a logging entity can communicate alerts to consumers of this information. Clearly this requires an agreement on an API between the logging entity and the consumer of the journal entries. Are there existing projects that have used this facility in a principled, coordinated way with some success or have there been any interesting failures along those same lines? systemd of courses uses its on its own, and we tried to document the fields we use in systemd.journal-fields(7), though it might be slightly incomplete. But yes, this is indeed API, and deserves complete documentation, like any C API, any D-Bus API or a any REST API would need too. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
