Re: [systemd-devel] Journalctl Unit Completion Optmization

2015-04-23 Thread Zbigniew Jędrzejewski-Szmek
On Wed, Apr 22, 2015 at 06:37:06PM +0200, Lennart Poettering wrote:
 On Tue, 03.03.15 23:02, Justin Brown (justin.br...@fandingo.org) wrote:
 
  Hello,
  
  I've recently been using journalctl on one of my longer-lived systems, and
  I'm running into a slight annoyance. Most of the time I'm using `journalctl
  -u unit -e`. I use tab completion for the unit name, and the annoyance
  is the long pause journalctl takes to complete a unit. With cold cache
  (meaning writing 3 to /proc/sys/vm/drop_caches), this takes 23 seconds on a
  system using Btrfs on a SATA3 SSD. With warm cache, it still takes 3-4
  seconds. Both cases are quite a bit slower than normal tab completion
  behavior.
 
 Is it really that bad if the files on btrfs are defragmented properly?
 Note that btrfs is notoriously bad at handling journald's write
 pattern currently, which results in heavily fragmented files. Use the
 filefrag tool to check how fragmented they are, and use btrfs defrag
 to the fix this.
 
 Before we look into other options I'd really prefer if you could check
 if this already fixes your issues.
 
 (Note that very recent journald will set the btrfs NOCOW flag now on
 the journal files, which should work around this issue.)
Also make sure that you're running recent enough and have commit 
v219-99-g950c07d421.

Zbyszek
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


Re: [systemd-devel] Journalctl Unit Completion Optmization

2015-04-22 Thread Lennart Poettering
On Tue, 03.03.15 23:02, Justin Brown (justin.br...@fandingo.org) wrote:

 Hello,
 
 I've recently been using journalctl on one of my longer-lived systems, and
 I'm running into a slight annoyance. Most of the time I'm using `journalctl
 -u unit -e`. I use tab completion for the unit name, and the annoyance
 is the long pause journalctl takes to complete a unit. With cold cache
 (meaning writing 3 to /proc/sys/vm/drop_caches), this takes 23 seconds on a
 system using Btrfs on a SATA3 SSD. With warm cache, it still takes 3-4
 seconds. Both cases are quite a bit slower than normal tab completion
 behavior.

Is it really that bad if the files on btrfs are defragmented properly?
Note that btrfs is notoriously bad at handling journald's write
pattern currently, which results in heavily fragmented files. Use the
filefrag tool to check how fragmented they are, and use btrfs defrag
to the fix this.

Before we look into other options I'd really prefer if you could check
if this already fixes your issues.

(Note that very recent journald will set the btrfs NOCOW flag now on
the journal files, which should work around this issue.)

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel


[systemd-devel] Journalctl Unit Completion Optmization

2015-03-03 Thread Justin Brown
Hello,

I've recently been using journalctl on one of my longer-lived systems, and
I'm running into a slight annoyance. Most of the time I'm using `journalctl
-u unit -e`. I use tab completion for the unit name, and the annoyance
is the long pause journalctl takes to complete a unit. With cold cache
(meaning writing 3 to /proc/sys/vm/drop_caches), this takes 23 seconds on a
system using Btrfs on a SATA3 SSD. With warm cache, it still takes 3-4
seconds. Both cases are quite a bit slower than normal tab completion
behavior.

I'm wondering if there has been any discussion or examination on optimizing
the bash completion to speed it up.

The Bash completion support for journalctl calls `journalctl -F
'_SYSTEMD_UNIT'` to assemble a list of acceptable units. That leads to
get_possible_units() in src/journal/journalctl.c, which calls the macro
SD_JOURNAL_FOREACH_UNIQUE and then sd_journal_enumerate_unique() in
sr/journal/sd-journal.c.

The problem from the user's perspective is that this scans through all the
journals, including user journals, which in my case is 68. That leads me to
a few possible optimization strategies, although each has downsides:

1) Instead of scanning through the journals for tab completion, use a more
simplistic search that only uses installed units. By installed, I mean
that tab completion only provides units that presently have a unit file on
the system (/etc/systemd/system/, /lib/systemd/system/, etc.) regardless of
enabled or activated state. Basically, just scan those directories for file
names.

The obvious downside is that the user wouldn't be able to tab complete
units which have since been deleted.

A simplistic POC solution would be to change L76-78 of
shell-completion/bash/journalctl from

--unit|-u)
comps=$(journalctl -F '_SYSTEMD_UNIT'
2/dev/null)
;;

to something like

--unit|-u)
comps=$(ls /lib/systemd/system
/etc/systemd/system | grep '\.service$\|\.timer$\|\.socket$')
;;


2) Have the journal maintain an index of units, which have emitted log
messages. That way tab completion would only need to scan this index to
populate `journalctl -F '_SYSTEMD_UNIT'`. The index could either be read
from disk or kept in memory.

The downside is two-fold. First, for every log message the journal would
need to account for whether that unit is already in the index. If the index
was constantly updated on disk, this adds an amount of addition IO. If it's
in memory, the journal would consume additional runtime memory. Second, as
old journals are removed due to cleaning, this index would need to be
updated.

3) A combination of the two approaches. If this is even possible for shell
completion, have the first tab completion attempt use method #1 or #2 and a
second tab completion attempt use either #2 or the current behavior. The
benefit is that a user using tab completion for convenience (instead of
obtaining an exhaustive list of all units) and knows what she is looking
for (eg. Networktab for NetworkManager.service) gets a fast answer.

The downside is whether shell completion can even handle multiple behaviors
and causing user confusion by altering the user interface.

Thoughts?

Thanks,
Justin
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel