Re: [systemd-devel] [PATCH] journalctl: allow customizable output formats
On Wed, Oct 08, 2014 at 11:53:38PM +0200, Lennart Poettering wrote:
On Mon, 22.09.14 16:33, Daniel P. Berrange (berra...@redhat.com) wrote:
The current '--output FORMAT' argument defines a number of
common output formats, but there are some useful cases it
does cover. In particular when reading application logs it
is often desirable to display the code file name, line number
and function name. Rather than defining yet more fixed output
formats, this patch introduces user defined output formats.
The format string is an arbitrary string which contains a
mixture of literal text and variable subsistitions. Each
variable name corresponds to a journal field name. A variable
name can be optionally followed by a data type, and in the
case of string types, a length limit.
Hmm, hmm, hmm.
I am quite afraid about inventing a new template language for this. I
can see the usecase though, and I sympasize with it.
I am particularly afraid of the entire type thing. The fact that the
journal is more or less typeless is after all by design: i really
didn't want to invent a new type system. Adding this to the formatter
now, kinda feels like adding it after all, but through the backdoor...
So, I am not against this in general, but I'd really be careful with
the language we define here, and try to make this as similar to an
existing language (like the python/java one Zbigniew mentioned) as we
can.
Or even better, we already have a very limited formatting language in
place, which is the specifier logic, that can replace %i, %f and
such things in unit files, maybe we can build on this, and allow
specifiers to take a field name as parameter. Then, if we really need
formatters for different field types, we could just give them
high-level characters or so?
Hmm, also, we already have a really bad formatter in place for the
journal catalog files (which only replaces @foo@ by the value of field
foo). We should probably use the same code for this new journalctl
formatter and the catalog code. In fact the catalog formatter could
really use some improvement...
Ok, I didn't know about the catalog files until now, so I'll
investigate that and see what I can do about unifying the code
for these two options.
Do you consider the catalog file format to be part of the stable
ABI ? ie, do we need to preserve support for @foo@ if we took
the %s{FOO} approach ?
Maybe something like this:
journalctl -O %t %s{CODE_FILE}:%s{CODE_LINE} %d{_SOURCE_REALTIME_TIMESTAMP}
or something like that, where %t would simply map to the timestamp,
and %s{} maps to a field name, and %d{} the same, but reformats the
field as timestamp, assuming it is a UNIX timestamp formatted as
number...
Or something like that...
Yep, that would work for me. I'll cook up another patch to demonstrate
that approach along with catalog support.
I'm about to be travelling for KVM Forum / LinuxCon so probably won't get
a chance to send an updated patch for a week or two.
Regards,
Daniel
--
|: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH] journalctl: allow customizable output formats
On Thu, 09.10.14 09:58, Daniel P. Berrange (berra...@redhat.com) wrote:
Hmm, also, we already have a really bad formatter in place for the
journal catalog files (which only replaces @foo@ by the value of field
foo). We should probably use the same code for this new journalctl
formatter and the catalog code. In fact the catalog formatter could
really use some improvement...
Ok, I didn't know about the catalog files until now, so I'll
investigate that and see what I can do about unifying the code
for these two options.
That would be fantastic!
Do you consider the catalog file format to be part of the stable
ABI ? ie, do we need to preserve support for @foo@ if we took
the %s{FOO} approach ?
I figure the @xyz@ syntax needs to stay. I mean, I doubt this is
widely used yet (we never really advertised it that much), but then
again it has been in place for a while now. So I figure we should
change the catalog files we ship to the new syntax, but still
alternatively support the old syntax, but only in the catalog files...
Maybe something like this:
journalctl -O %t %s{CODE_FILE}:%s{CODE_LINE}
%d{_SOURCE_REALTIME_TIMESTAMP}
or something like that, where %t would simply map to the timestamp,
and %s{} maps to a field name, and %d{} the same, but reformats the
field as timestamp, assuming it is a UNIX timestamp formatted as
number...
Or something like that...
Yep, that would work for me. I'll cook up another patch to demonstrate
that approach along with catalog support.
Thanks! That'd be lovely!
I'm about to be travelling for KVM Forum / LinuxCon so probably won't get
a chance to send an updated patch for a week or two.
See you in Düsseldorf!
Lennart
--
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH] journalctl: allow customizable output formats
On Mon, 22.09.14 16:33, Daniel P. Berrange (berra...@redhat.com) wrote:
The current '--output FORMAT' argument defines a number of
common output formats, but there are some useful cases it
does cover. In particular when reading application logs it
is often desirable to display the code file name, line number
and function name. Rather than defining yet more fixed output
formats, this patch introduces user defined output formats.
The format string is an arbitrary string which contains a
mixture of literal text and variable subsistitions. Each
variable name corresponds to a journal field name. A variable
name can be optionally followed by a data type, and in the
case of string types, a length limit.
Hmm, hmm, hmm.
I am quite afraid about inventing a new template language for this. I
can see the usecase though, and I sympasize with it.
I am particularly afraid of the entire type thing. The fact that the
journal is more or less typeless is after all by design: i really
didn't want to invent a new type system. Adding this to the formatter
now, kinda feels like adding it after all, but through the backdoor...
So, I am not against this in general, but I'd really be careful with
the language we define here, and try to make this as similar to an
existing language (like the python/java one Zbigniew mentioned) as we
can.
Or even better, we already have a very limited formatting language in
place, which is the specifier logic, that can replace %i, %f and
such things in unit files, maybe we can build on this, and allow
specifiers to take a field name as parameter. Then, if we really need
formatters for different field types, we could just give them
high-level characters or so?
Hmm, also, we already have a really bad formatter in place for the
journal catalog files (which only replaces @foo@ by the value of field
foo). We should probably use the same code for this new journalctl
formatter and the catalog code. In fact the catalog formatter could
really use some improvement...
Maybe something like this:
journalctl -O %t %s{CODE_FILE}:%s{CODE_LINE} %d{_SOURCE_REALTIME_TIMESTAMP}
or something like that, where %t would simply map to the timestamp,
and %s{} maps to a field name, and %d{} the same, but reformats the
field as timestamp, assuming it is a UNIX timestamp formatted as
number...
Or something like that...
Other ideas?
Lennart
--
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH] journalctl: allow customizable output formats
On Fri, Oct 03, 2014 at 02:13:51AM +0200, Zbigniew Jędrzejewski-Szmek wrote:
On Mon, Sep 22, 2014 at 04:33:28PM +0100, Daniel P. Berrange wrote:
The current '--output FORMAT' argument defines a number of
common output formats, but there are some useful cases it
does cover. In particular when reading application logs it
is often desirable to display the code file name, line number
and function name. Rather than defining yet more fixed output
formats, this patch introduces user defined output formats.
Hi,
I think this makes sense. But I think that the format strings you
propose are damn ugly :). Using %() for variables seems too heavy.
Also, journal fields are all text, so I don't think that specifying
the type is useful.
Well there are two virtual fields which are timestamps which the existing
hardcoded output modes convert into a date string in various ways. I want
the format strings we define here to be able to express the semantics of
the current hardcoded output modes, so this neccessitates a way to ask
for various date formats.
Also although the physically stored journal fields are strings per the
journal API storage backend, they can be simple string versions of
other data types. eg an application defined journal field could be used
to store an integer, floating point, boolean, etc. It would be natural
for the app to use many decimal places if storing a floating point value
in the journal, so being able to give data types in the output mode lets
us alter the precision displayed when extracting it again. Of course my
patch didn't try todo this, it only deal with dates.
Maybe we could adopt the {} format from Java and Python, as
implemented in Python [1]. It has a fairly rich and consistent field
formatting language. We would care only about the part relevant
to strings, at least in the beginning.
I'll see what I can cook up along these lines, but the existing python
language is focused on C data types and doesn't directly provide types
for the various date formats to support, so we can't use it 100% as-is.
Regards,
Daniel
--
|: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH] journalctl: allow customizable output formats
On Mon, Sep 22, 2014 at 04:33:28PM +0100, Daniel P. Berrange wrote:
The current '--output FORMAT' argument defines a number of
common output formats, but there are some useful cases it
does cover. In particular when reading application logs it
is often desirable to display the code file name, line number
and function name. Rather than defining yet more fixed output
formats, this patch introduces user defined output formats.
Hi,
I think this makes sense. But I think that the format strings you
propose are damn ugly :). Using %() for variables seems too heavy.
Also, journal fields are all text, so I don't think that specifying
the type is useful.
Maybe we could adopt the {} format from Java and Python, as
implemented in Python [1]. It has a fairly rich and consistent field
formatting language. We would care only about the part relevant
to strings, at least in the beginning.
[1] https://docs.python.org/3.5/library/string.html#formatspec
Zbyszek
The format string is an arbitrary string which contains a
mixture of literal text and variable subsistitions. Each
variable name corresponds to a journal field name. A variable
name can be optionally followed by a data type, and in the
case of string types, a length limit.
This is best illustrated with an example:
$ journalctl -o format:%(__REALTIME_TIMESTAMP) \
[%(CODE_FILE):%(CODE_LINE):%(CODE_FUNC)] \
%(MESSAGE:string:80)\n _COMM=libvirtd
-- Logs begin at Mon 2013-12-23 16:31:41 GMT, end at Mon 2014-09-22
16:13:00 BST. --
Dec 23 17:19:25 [util/virlog.c:877:virLogVMessage] libvirt version:
1.1.3.1, package: 2.fc20 (Fedora Project, 2013-11-17-23:28:43, ...
Dec 23 17:19:25 [conf/storage_conf.c:854:virStoragePoolDefParseXML] XML
error: unknown storage pool type btrfs
Dec 23 17:19:30 [conf/domain_conf.c:12671:virDomainObjParseNode] XML error:
unexpected root element domain, expecting domstatus
Dec 23 17:24:45 [qemu/qemu_monitor.c:653:qemuMonitorIO] internal error: End
of file from monitor
Dec 23 20:12:00 [qemu/qemu_monitor.c:653:qemuMonitorIO] internal error: End
of file from monitor
-- Reboot --
Dec 23 21:06:14 [util/virlog.c:877:virLogVMessage] libvirt version:
1.1.3.1, package: 2.fc20 (Fedora Project, 2013-11-17-23:28:43, ...
Dec 23 21:06:21 [conf/storage_conf.c:854:virStoragePoolDefParseXML] XML
error: unknown storage pool type btrfs
Signed-off-by: Daniel P. Berrange berra...@redhat.com
---
man/journalctl.xml| 76 +
src/journal-remote/journal-gatewayd.c | 11 +-
src/journal/journalctl.c | 39 ++-
src/shared/logs-show.c| 532
++
src/shared/logs-show.h| 16 +-
src/shared/output-mode.h | 1 +
src/systemctl/systemctl.c | 20 +-
7 files changed, 615 insertions(+), 80 deletions(-)
diff --git a/man/journalctl.xml b/man/journalctl.xml
index acd75a6..bd8c2bd 100644
--- a/man/journalctl.xml
+++ b/man/journalctl.xml
@@ -375,6 +375,21 @@
/para
/listitem
/varlistentry
+
+varlistentry
+term
+
optionformat:FMT/option
+/term
+listitem
+paragenerates
output
+according to the
format
+specification given
in
+the FMT string. See
the
+OUTPUT FORMAT STRINGS
+section for details
+/para
+/listitem
+/varlistentry
/variablelist
/listitem
/varlistentry
@@ -878,6 +893,64 @@
/refsect1
refsect1
+titleOutput Format Strings/title
+
+paraAn output format string provides precise control how
journal
+data records are formatted for output. A format string
consists of
+mixture of literal text and variables to be substituted with
journal
+data records. A variable takes the general form/para
+
+programlisting$(NAME:TYPE:LEN)/programlisting
+
+paraThe NAME component corresponds to any journal entry
field
+(eg MESSAGE, _SYSTEMD_UNIT,
Re: [systemd-devel] [PATCH] journalctl: allow customizable output formats
On Mon, Sep 22, 2014 at 12:43:28PM -0400, Daurnimator wrote: On 22 September 2014 11:33, Daniel P. Berrange berra...@redhat.com wrote: The current '--output FORMAT' argument defines a number of common output formats, but there are some useful cases it does cover. In particular when reading application logs it is often desirable to display the code file name, line number and function name. Rather than defining yet more fixed output formats, this patch introduces user defined output formats. The format string is an arbitrary string which contains a mixture of literal text and variable subsistitions. Each variable name corresponds to a journal field name. A variable name can be optionally followed by a data type, and in the case of string types, a length limit. As an opposing point of view, I've been accomplishing this by piping output through a script that parses and displays JSON. I rather this style of composability than passing format strings to journalctl itself. Sure you could do that, but it is really madness to expect anyone who just wants to display a slightly different set of fields to write a script to parse JSON and re-write it. When I have end users doing troubleshooting of libvirt for bug reports, I want to be able to just tell them to run a direct journalctl command to collect data I need, not have to write or download some extra script to parse JSON. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH] journalctl: allow customizable output formats
[Strangely, this thread up in my inbox rather than my systemd Maildir.] On Mon, Sep 22, 2014 at 12:43:28PM -0400, Daurnimator wrote: On 22 September 2014 11:33, Daniel P. Berrange berra...@redhat.com wrote: The current '--output FORMAT' argument defines a number of common output formats, but there are some useful cases it does cover. In particular when reading application logs it is often desirable to display the code file name, line number and function name. Rather than defining yet more fixed output formats, this patch introduces user defined output formats. The format string is an arbitrary string which contains a mixture of literal text and variable subsistitions. Each variable name corresponds to a journal field name. A variable name can be optionally followed by a data type, and in the case of string types, a length limit. As an opposing point of view, I've been accomplishing this by piping output through a script that parses and displays JSON. I rather this style of composability than passing format strings to journalctl itself. FWIW, as a heavy user and frequent tester of libvirt, I'd absolutely welcome the proposed change which clearly illustrated what I wished for (code file, line, function name the error string) at times when debugging myself or helping others debug. When at a critical time, I'd rather much depend on `journalctl` having such a useful option handy than depending on yet another random script from the inter-webs. -- /kashyap ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] [PATCH] journalctl: allow customizable output formats
The current '--output FORMAT' argument defines a number of common output formats, but there are some useful cases it does cover. In particular when reading application logs it is often desirable to display the code file name, line number and function name. Rather than defining yet more fixed output formats, this patch introduces user defined output formats. The format string is an arbitrary string which contains a mixture of literal text and variable subsistitions. Each variable name corresponds to a journal field name. A variable name can be optionally followed by a data type, and in the case of string types, a length limit. This is best illustrated with an example: $ journalctl -o format:%(__REALTIME_TIMESTAMP) \ [%(CODE_FILE):%(CODE_LINE):%(CODE_FUNC)] \ %(MESSAGE:string:80)\n _COMM=libvirtd -- Logs begin at Mon 2013-12-23 16:31:41 GMT, end at Mon 2014-09-22 16:13:00 BST. -- Dec 23 17:19:25 [util/virlog.c:877:virLogVMessage] libvirt version: 1.1.3.1, package: 2.fc20 (Fedora Project, 2013-11-17-23:28:43, ... Dec 23 17:19:25 [conf/storage_conf.c:854:virStoragePoolDefParseXML] XML error: unknown storage pool type btrfs Dec 23 17:19:30 [conf/domain_conf.c:12671:virDomainObjParseNode] XML error: unexpected root element domain, expecting domstatus Dec 23 17:24:45 [qemu/qemu_monitor.c:653:qemuMonitorIO] internal error: End of file from monitor Dec 23 20:12:00 [qemu/qemu_monitor.c:653:qemuMonitorIO] internal error: End of file from monitor -- Reboot -- Dec 23 21:06:14 [util/virlog.c:877:virLogVMessage] libvirt version: 1.1.3.1, package: 2.fc20 (Fedora Project, 2013-11-17-23:28:43, ... Dec 23 21:06:21 [conf/storage_conf.c:854:virStoragePoolDefParseXML] XML error: unknown storage pool type btrfs Signed-off-by: Daniel P. Berrange berra...@redhat.com --- man/journalctl.xml| 76 + src/journal-remote/journal-gatewayd.c | 11 +- src/journal/journalctl.c | 39 ++- src/shared/logs-show.c| 532 ++ src/shared/logs-show.h| 16 +- src/shared/output-mode.h | 1 + src/systemctl/systemctl.c | 20 +- 7 files changed, 615 insertions(+), 80 deletions(-) diff --git a/man/journalctl.xml b/man/journalctl.xml index acd75a6..bd8c2bd 100644 --- a/man/journalctl.xml +++ b/man/journalctl.xml @@ -375,6 +375,21 @@ /para /listitem /varlistentry + +varlistentry +term + optionformat:FMT/option +/term +listitem +paragenerates output +according to the format +specification given in +the FMT string. See the +OUTPUT FORMAT STRINGS +section for details +/para +/listitem +/varlistentry /variablelist /listitem /varlistentry @@ -878,6 +893,64 @@ /refsect1 refsect1 +titleOutput Format Strings/title + +paraAn output format string provides precise control how journal +data records are formatted for output. A format string consists of +mixture of literal text and variables to be substituted with journal +data records. A variable takes the general form/para + +programlisting$(NAME:TYPE:LEN)/programlisting + +paraThe NAME component corresponds to any journal entry field +(eg MESSAGE, _SYSTEMD_UNIT, CODE_FUNC, etc). The TYPE component +determines the data format to use for printing the value. If +omitted, it defaults to a sensible format for the NAME of the +field. The LEN component places an upper limit on the length of +strings being printed, beyond which they will be ellipsized. +The valid data types for TYPE are:/para + +variablelist +varlistentry +termstring/term +listitemparadisplayed if a printable string. If the value +contains non-printable characters, it will be
Re: [systemd-devel] [PATCH] journalctl: allow customizable output formats
On 22 September 2014 11:33, Daniel P. Berrange berra...@redhat.com wrote: The current '--output FORMAT' argument defines a number of common output formats, but there are some useful cases it does cover. In particular when reading application logs it is often desirable to display the code file name, line number and function name. Rather than defining yet more fixed output formats, this patch introduces user defined output formats. The format string is an arbitrary string which contains a mixture of literal text and variable subsistitions. Each variable name corresponds to a journal field name. A variable name can be optionally followed by a data type, and in the case of string types, a length limit. As an opposing point of view, I've been accomplishing this by piping output through a script that parses and displays JSON. I rather this style of composability than passing format strings to journalctl itself. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH] journalctl: allow customizable output formats
On Monday 22 September 2014 at 12:43:28, Daurnimator wrote: On 22 September 2014 11:33, Daniel P. Berrange berra...@redhat.com wrote: The current '--output FORMAT' argument defines a number of common output formats, but there are some useful cases it does cover. In particular when reading application logs it is often desirable to display the code file name, line number and function name. Rather than defining yet more fixed output formats, this patch introduces user defined output formats. The format string is an arbitrary string which contains a mixture of literal text and variable subsistitions. Each variable name corresponds to a journal field name. A variable name can be optionally followed by a data type, and in the case of string types, a length limit. As an opposing point of view, I've been accomplishing this by piping output through a script that parses and displays JSON. I rather this style of composability than passing format strings to journalctl itself. I think that using dedicated ad-hoc JSON parser for customizing output is an overkill. For that matter, Git has support for custom output format in its log command, and I've found it useful many times. -- Ivan Shapovalov / intelfx / signature.asc Description: This is a digitally signed message part. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
