On Wed, Apr 16, 2014 at 12:50:53PM -0400, Martin Langhoff wrote:
> On Mon, Apr 14, 2014 at 9:07 AM, Jaroslav Reznik <jrez...@redhat.com> wrote:
> > The communication between the two daemons is done over standard HTTPS,
> I hear you holler "OMG you have to build full redundancy in your
> logging backend"; and... I have not seen a single operation where the
> logging backed was fully redundant.
Hi,
the upload client is like any other journal client — it is fully asynchronous
wrt. to journald writing log entries. (It's something like
'journalctl -o export|curl -X POST https://some.where/upload'.)
So it is assumed that local storage of journal logs is enabled (either
in /var/log/journal or in /run/log/journal), and the uploader simply
reads data from there at its leisure. In fact, the uploader is only
started late during boot, after the network is up, while journald
writes logs from very early in the boot.
There's no guarantee that the logs will ever be uploaded, but in
practice they will only be lost if enough new logs come in to force
the old entries out. Even with the journal capped at a few megabytes,
this shouldn't happen but rarely.
> And in fact it may be too much to ask -- in most setups log entries
> are not _that_ precious. I know I can reconfigure a syslog server and
> restart it without the 1K VMs that talk to it glitching. A recent
> "loop" in our syslog configuration was a relatively minor problem
> because it just dropped the traffic it couldn't handle for a brief
> time while we fixed things.
>
> This is the reality of system configs I know. Fully redundant,
> "perfect" log servers are very hard to run, might not be worth it, and
> anyway such a change won't happen overnight.
>
> To avoid gridlocking operations, IMO this logging will need a local
> queue, (for later retry), with a "drop anything older than X" escape
> hatch...
I think that this describes my current implementation to a large extent.
The client remembers the position ("cursor") of the last uploaded entry
and restarts from there. "drop anything older than X" feature sounds
useful, and should be easy to add.
Zbyszek
--
devel mailing list
devel@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/devel
Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
