Re: [systemd-devel] [PATCH] journald: Introduce RFC 5424 syslog
On Thu, Feb 19, 2015 at 12:10:04PM +0100, Lennart Poettering wrote: On Thu, 19.02.15 13:28, Susant Sahani (sus...@redhat.com) wrote: This patch adds support for RFC 5424 syslog format to journald. Journald can now forward logs to a multicast UDP group. RFC 5424 format: PRIVERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP [SD-ID]s SP MSG Hmm, wasn't the last proposal we discussed to do this in an auxiliary daemmon, possibly in systemd-journal-upload or so, but not in journald? We discussed both... From http://lists.freedesktop.org/archives/systemd-devel/2014-December/026202.html: Having this in systems-journald and extend the forward to syslog config with the target host was our expectation anyway. The difference is in how the logs are accessed: if journald itself does the jobs, they would be forwarded live. If anything else, the uploader would be a client which reads the files in /var/log/journal/. The are advantages to both solutions: the first one might be more robust if writing the logs fails or stops for whatever reason. The second one will probably send more logs, because sending of logs can be delayed until the network is up. In the second version, the uploader can also forward logs from other machines (containers). Now that I spelled it out, the second version seems nicer. After rereading the old discussion, I have to agree with Lennart that *not* doing it in systemd-journald directly seems better. Reasons below... I see two problems with journald: first of all, for security reasons I am conservative about making it deal with the network directly. Opening up such a basic daemon to the network is a something i'd prefer to avoid. I don't see how opening a socket to send UDP messages is dangerous. But yeah, sd-journald runs as root with full capabilities. sd-journal-upload runs as an unprivileged user. The other thing is that journald runs really really early during boot, at a time where the network is unlikely to be up. This means that early boot msgs could never be delivered via syslog... And this is a convincing argument for me. Essentially, by doing it in a separate tool we get reliability which we could never have with journald. I'd really prefer a scheme where this syslog broadcaster can be run relatively late at boot and where it tries to repeatedly send the messages, until sendmsg() actually succeeds. i.e. using the journal cursor logic it would not send a log message until the point where the previous message was delivered with a successful sendmsg(). Wth such a scheme all early boot msgs would be dumped on the network the moment the network is up. Zbigniew, do you have more ideas about this? Yep, sounds right. Susant, sorry! I think we should at look at adding this to sd-journal-upload, or a separate similar tool which reuses some code of sd-journal-upload. Zbyszek ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH] journald: Introduce RFC 5424 syslog
On Fri, 20 Feb 2015 22:14:20 +0530, Zbigniew Jędrzejewski-Szmek zbys...@in.waw.pl wrote: On Thu, Feb 19, 2015 at 12:10:04PM +0100, Lennart Poettering wrote: On Thu, 19.02.15 13:28, Susant Sahani (sus...@redhat.com) wrote: This patch adds support for RFC 5424 syslog format to journald. Journald can now forward logs to a multicast UDP group. RFC 5424 format: PRIVERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP [SD-ID]s SP MSG Hmm, wasn't the last proposal we discussed to do this in an auxiliary daemmon, possibly in systemd-journal-upload or so, but not in journald? We discussed both... From http://lists.freedesktop.org/archives/systemd-devel/2014-December/026202.html: Having this in systems-journald and extend the forward to syslog config with the target host was our expectation anyway. The difference is in how the logs are accessed: if journald itself does the jobs, they would be forwarded live. If anything else, the uploader would be a client which reads the files in /var/log/journal/. The are advantages to both solutions: the first one might be more robust if writing the logs fails or stops for whatever reason. The second one will probably send more logs, because sending of logs can be delayed until the network is up. In the second version, the uploader can also forward logs from other machines (containers). Now that I spelled it out, the second version seems nicer. After rereading the old discussion, I have to agree with Lennart that *not* doing it in systemd-journald directly seems better. Reasons below... I see two problems with journald: first of all, for security reasons I am conservative about making it deal with the network directly. Opening up such a basic daemon to the network is a something i'd prefer to avoid. I don't see how opening a socket to send UDP messages is dangerous. But yeah, sd-journald runs as root with full capabilities. sd-journal-upload runs as an unprivileged user. The other thing is that journald runs really really early during boot, at a time where the network is unlikely to be up. This means that early boot msgs could never be delivered via syslog... And this is a convincing argument for me. Essentially, by doing it in a separate tool we get reliability which we could never have with journald. I'd really prefer a scheme where this syslog broadcaster can be run relatively late at boot and where it tries to repeatedly send the messages, until sendmsg() actually succeeds. i.e. using the journal cursor logic it would not send a log message until the point where the previous message was delivered with a successful sendmsg(). Wth such a scheme all early boot msgs would be dumped on the network the moment the network is up. Zbigniew, do you have more ideas about this? Yep, sounds right. Susant, sorry! I think we should at look at adding this to sd-journal-upload, or a separate similar tool which reuses some code of sd-journal-upload. Yes :) . I will start working on it. just have to plug in this patch with the new daemon. Susant ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH] journald: Introduce RFC 5424 syslog
Hi Susant,
On Thu, Feb 19, 2015 at 8:58 AM, Susant Sahani sus...@redhat.com wrote:
This patch adds support for RFC 5424 syslog format to journald. Journald
can now forward logs to a multicast UDP group.
RFC 5424 format:
PRIVERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP
[SD-ID]s SP MSG
Example conf:
file: journald.conf
SysLogAddress=239.0.0.1:6000
---
Makefile.am | 1 +
man/journald.conf.xml | 12 ++
src/journal/journald-gperf.gperf | 1 +
src/journal/journald-native.c | 3 +
src/journal/journald-server.c | 40 +-
src/journal/journald-server.h | 14 ++
src/journal/journald-stream.c | 4 +
src/journal/journald-syslog-network.c | 246
++
src/journal/journald-syslog.c | 3 +
src/journal/journald-syslog.h | 2 +
10 files changed, 325 insertions(+), 1 deletion(-)
create mode 100644 src/journal/journald-syslog-network.c
diff --git a/Makefile.am b/Makefile.am
index ba63f68..b015f69 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -4487,6 +4487,7 @@ libsystemd_journal_core_la_SOURCES = \
src/journal/journald-kmsg.h \
src/journal/journald-syslog.c \
src/journal/journald-syslog.h \
+ src/journal/journald-syslog-network.c \
src/journal/journald-stream.c \
src/journal/journald-stream.h \
src/journal/journald-server.c \
diff --git a/man/journald.conf.xml b/man/journald.conf.xml
index 364b58f..4fb037b 100644
--- a/man/journald.conf.xml
+++ b/man/journald.conf.xml
@@ -355,6 +355,18 @@
/varlistentry
varlistentry
+termvarnameSysLogAddress=/varname/term
+listitemparaControls whether log messages received by the
+journal daemon shall be forwarded to a multicast UDP network
+group in syslog RFC 5424 format./para
+
+paraThe the address string format is similar to socket units. See
Double the.
+
citerefentryrefentrytitlesystemd.socket/refentrytitlemanvolnum1/manvolnum/citerefentry
+/para
+/listitem
+ /varlistentry
+
+ varlistentry
termvarnameTTYPath=/varname/term
listitemparaChange the console TTY to use if
diff --git a/src/journal/journald-gperf.gperf
b/src/journal/journald-gperf.gperf
index 74554c1..9cdffbc 100644
--- a/src/journal/journald-gperf.gperf
+++ b/src/journal/journald-gperf.gperf
@@ -40,3 +40,4 @@ Journal.MaxLevelKMsg, config_parse_log_level, 0,
offsetof(Server, max_lev
Journal.MaxLevelConsole,config_parse_log_level, 0, offsetof(Server,
max_level_console)
Journal.MaxLevelWall, config_parse_log_level, 0, offsetof(Server,
max_level_wall)
Journal.SplitMode, config_parse_split_mode, 0, offsetof(Server,
split_mode)
+Journal.SysLogAddress, config_parse_syslog_network_address, 0,
offsetof(Server, syslog_addr)
diff --git a/src/journal/journald-native.c b/src/journal/journald-native.c
index 851625d..9fd370f 100644
--- a/src/journal/journald-native.c
+++ b/src/journal/journald-native.c
@@ -273,6 +273,9 @@ void server_process_native_message(
if (s-forward_to_syslog)
server_forward_syslog(s, priority, identifier,
message, ucred, tv);
+if (s-forward_to_network)
+server_forward_syslog_network(s, priority,
identifier, message, ucred, tv);
+
if (s-forward_to_kmsg)
server_forward_kmsg(s, priority, identifier,
message, ucred);
diff --git a/src/journal/journald-server.c b/src/journal/journald-server.c
index 7ee8174..de4ef50 100644
--- a/src/journal/journald-server.c
+++ b/src/journal/journald-server.c
@@ -86,7 +86,7 @@ static const char* const split_mode_table[_SPLIT_MAX] = {
DEFINE_STRING_TABLE_LOOKUP(split_mode, SplitMode);
DEFINE_CONFIG_PARSE_ENUM(config_parse_split_mode, split_mode, SplitMode,
Failed to parse split mode setting);
-static uint64_t available_space(Server *s, bool verbose) {
+uint64_t available_space(Server *s, bool verbose) {
char ids[33];
_cleanup_free_ char *p = NULL;
sd_id128_t machine;
@@ -1356,6 +1356,35 @@ static int server_parse_config_file(Server *s) {
false, s);
}
+int config_parse_syslog_network_address(const char *unit,
+const char *filename,
+unsigned line,
+const char *section,
+unsigned section_line,
+const char *lvalue,
+int ltype,
+const char *rvalue,
+void *data,
+void
Re: [systemd-devel] [PATCH] journald: Introduce RFC 5424 syslog
On Thu, 19.02.15 13:28, Susant Sahani (sus...@redhat.com) wrote: This patch adds support for RFC 5424 syslog format to journald. Journald can now forward logs to a multicast UDP group. RFC 5424 format: PRIVERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP [SD-ID]s SP MSG Hmm, wasn't the last proposal we discussed to do this in an auxiliary daemmon, possibly in systemd-journal-upload or so, but not in journald? I see two problems with journald: first of all, for security reasons I am conservative about making it deal with the network directly. Opening up such a basic daemon to the network is a something i'd prefer to avoid. The other thing is that journald runs really really early during boot, at a time where the network is unlikely to be up. This means that early boot msgs could never be delivered via syslog... I'd really prefer a scheme where this syslog broadcaster can be run relatively late at boot and where it tries to repeatedly send the messages, until sendmsg() actually succeeds. i.e. using the journal cursor logic it would not send a log message until the point where the previous message was delivered with a successful sendmsg(). Wth such a scheme all early boot msgs would be dumped on the network the moment the network is up. Zbigniew, do you have more ideas about this? Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH] journald: Introduce RFC 5424 syslog
On Thu, 19 Feb 2015 13:53:42 +0530, Umut Tezduyar Lindskog
u...@tezduyar.com wrote:
Hi Susant,
Hi Umut,
Thanks for reviewing.
On Thu, Feb 19, 2015 at 8:58 AM, Susant Sahani sus...@redhat.com wrote:
This patch adds support for RFC 5424 syslog format to journald. Journald
can now forward logs to a multicast UDP group.
RFC 5424 format:
PRIVERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP
[SD-ID]s SP MSG
Example conf:
file: journald.conf
SysLogAddress=239.0.0.1:6000
---
Makefile.am | 1 +
man/journald.conf.xml | 12 ++
src/journal/journald-gperf.gperf | 1 +
src/journal/journald-native.c | 3 +
src/journal/journald-server.c | 40 +-
src/journal/journald-server.h | 14 ++
src/journal/journald-stream.c | 4 +
src/journal/journald-syslog-network.c | 246
++
src/journal/journald-syslog.c | 3 +
src/journal/journald-syslog.h | 2 +
10 files changed, 325 insertions(+), 1 deletion(-)
create mode 100644 src/journal/journald-syslog-network.c
diff --git a/Makefile.am b/Makefile.am
index ba63f68..b015f69 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -4487,6 +4487,7 @@ libsystemd_journal_core_la_SOURCES = \
src/journal/journald-kmsg.h \
src/journal/journald-syslog.c \
src/journal/journald-syslog.h \
+ src/journal/journald-syslog-network.c \
src/journal/journald-stream.c \
src/journal/journald-stream.h \
src/journal/journald-server.c \
diff --git a/man/journald.conf.xml b/man/journald.conf.xml
index 364b58f..4fb037b 100644
--- a/man/journald.conf.xml
+++ b/man/journald.conf.xml
@@ -355,6 +355,18 @@
/varlistentry
varlistentry
+termvarnameSysLogAddress=/varname/term
+listitemparaControls whether log messages received by the
+journal daemon shall be forwarded to a multicast UDP network
+group in syslog RFC 5424 format./para
+
+paraThe the address string format is similar to socket
units. See
Double the.
+
citerefentryrefentrytitlesystemd.socket/refentrytitlemanvolnum1/manvolnum/citerefentry
+/para
+/listitem
+ /varlistentry
+
+ varlistentry
termvarnameTTYPath=/varname/term
listitemparaChange the console TTY to use if
diff --git a/src/journal/journald-gperf.gperf
b/src/journal/journald-gperf.gperf
index 74554c1..9cdffbc 100644
--- a/src/journal/journald-gperf.gperf
+++ b/src/journal/journald-gperf.gperf
@@ -40,3 +40,4 @@ Journal.MaxLevelKMsg, config_parse_log_level,
0, offsetof(Server, max_lev
Journal.MaxLevelConsole,config_parse_log_level, 0,
offsetof(Server, max_level_console)
Journal.MaxLevelWall, config_parse_log_level, 0,
offsetof(Server, max_level_wall)
Journal.SplitMode, config_parse_split_mode, 0,
offsetof(Server, split_mode)
+Journal.SysLogAddress, config_parse_syslog_network_address, 0,
offsetof(Server, syslog_addr)
diff --git a/src/journal/journald-native.c
b/src/journal/journald-native.c
index 851625d..9fd370f 100644
--- a/src/journal/journald-native.c
+++ b/src/journal/journald-native.c
@@ -273,6 +273,9 @@ void server_process_native_message(
if (s-forward_to_syslog)
server_forward_syslog(s, priority, identifier,
message, ucred, tv);
+if (s-forward_to_network)
+server_forward_syslog_network(s, priority,
identifier, message, ucred, tv);
+
if (s-forward_to_kmsg)
server_forward_kmsg(s, priority, identifier,
message, ucred);
diff --git a/src/journal/journald-server.c
b/src/journal/journald-server.c
index 7ee8174..de4ef50 100644
--- a/src/journal/journald-server.c
+++ b/src/journal/journald-server.c
@@ -86,7 +86,7 @@ static const char* const split_mode_table[_SPLIT_MAX]
= {
DEFINE_STRING_TABLE_LOOKUP(split_mode, SplitMode);
DEFINE_CONFIG_PARSE_ENUM(config_parse_split_mode, split_mode,
SplitMode, Failed to parse split mode setting);
-static uint64_t available_space(Server *s, bool verbose) {
+uint64_t available_space(Server *s, bool verbose) {
char ids[33];
_cleanup_free_ char *p = NULL;
sd_id128_t machine;
@@ -1356,6 +1356,35 @@ static int server_parse_config_file(Server *s) {
false, s);
}
+int config_parse_syslog_network_address(const char *unit,
+const char *filename,
+unsigned line,
+const char *section,
+unsigned section_line,
+const char *lvalue,
+int ltype,
+const char *rvalue,
+void *data,
+
Re: [systemd-devel] [PATCH] journald: Introduce RFC 5424 syslog
Hi Lennart, On Thu, 19 Feb 2015 16:40:04 +0530, Lennart Poettering lenn...@poettering.net wrote: On Thu, 19.02.15 13:28, Susant Sahani (sus...@redhat.com) wrote: This patch adds support for RFC 5424 syslog format to journald. Journald can now forward logs to a multicast UDP group. RFC 5424 format: PRIVERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP [SD-ID]s SP MSG Hmm, wasn't the last proposal we discussed to do this in an auxiliary daemmon, possibly in systemd-journal-upload or so, but not in journald? ohhh. I missed that part. I see two problems with journald: first of all, for security reasons I am conservative about making it deal with the network directly. Opening up such a basic daemon to the network is a something i'd prefer to avoid. The other thing is that journald runs really really early during boot, at a time where the network is unlikely to be up. This means that early boot msgs could never be delivered via syslog... Makes sense . I agree with it. I'd really prefer a scheme where this syslog broadcaster can be run relatively late at boot and where it tries to repeatedly send the messages, until sendmsg() actually succeeds. i.e. using the journal cursor logic it would not send a log message until the point where the previous message was delivered with a successful sendmsg(). Wth such a scheme all early boot msgs would be dumped on the network the moment the network is up. So do we want to write down another daemon or integrate with journad with cursor logic ? I am ok with any of this. Zbigniew, do you have more ideas about this? Lennart Susant ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] [PATCH] journald: Introduce RFC 5424 syslog
This patch adds support for RFC 5424 syslog format to journald. Journald
can now forward logs to a multicast UDP group.
RFC 5424 format:
PRIVERSION SP TIMESTAMP SP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP
[SD-ID]s SP MSG
Example conf:
file: journald.conf
SysLogAddress=239.0.0.1:6000
---
Makefile.am | 1 +
man/journald.conf.xml | 12 ++
src/journal/journald-gperf.gperf | 1 +
src/journal/journald-native.c | 3 +
src/journal/journald-server.c | 40 +-
src/journal/journald-server.h | 14 ++
src/journal/journald-stream.c | 4 +
src/journal/journald-syslog-network.c | 246 ++
src/journal/journald-syslog.c | 3 +
src/journal/journald-syslog.h | 2 +
10 files changed, 325 insertions(+), 1 deletion(-)
create mode 100644 src/journal/journald-syslog-network.c
diff --git a/Makefile.am b/Makefile.am
index ba63f68..b015f69 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -4487,6 +4487,7 @@ libsystemd_journal_core_la_SOURCES = \
src/journal/journald-kmsg.h \
src/journal/journald-syslog.c \
src/journal/journald-syslog.h \
+ src/journal/journald-syslog-network.c \
src/journal/journald-stream.c \
src/journal/journald-stream.h \
src/journal/journald-server.c \
diff --git a/man/journald.conf.xml b/man/journald.conf.xml
index 364b58f..4fb037b 100644
--- a/man/journald.conf.xml
+++ b/man/journald.conf.xml
@@ -355,6 +355,18 @@
/varlistentry
varlistentry
+termvarnameSysLogAddress=/varname/term
+listitemparaControls whether log messages received by the
+journal daemon shall be forwarded to a multicast UDP network
+group in syslog RFC 5424 format./para
+
+paraThe the address string format is similar to socket units. See
+
citerefentryrefentrytitlesystemd.socket/refentrytitlemanvolnum1/manvolnum/citerefentry
+/para
+/listitem
+ /varlistentry
+
+ varlistentry
termvarnameTTYPath=/varname/term
listitemparaChange the console TTY to use if
diff --git a/src/journal/journald-gperf.gperf b/src/journal/journald-gperf.gperf
index 74554c1..9cdffbc 100644
--- a/src/journal/journald-gperf.gperf
+++ b/src/journal/journald-gperf.gperf
@@ -40,3 +40,4 @@ Journal.MaxLevelKMsg, config_parse_log_level, 0,
offsetof(Server, max_lev
Journal.MaxLevelConsole,config_parse_log_level, 0, offsetof(Server,
max_level_console)
Journal.MaxLevelWall, config_parse_log_level, 0, offsetof(Server,
max_level_wall)
Journal.SplitMode, config_parse_split_mode, 0, offsetof(Server,
split_mode)
+Journal.SysLogAddress, config_parse_syslog_network_address, 0,
offsetof(Server, syslog_addr)
diff --git a/src/journal/journald-native.c b/src/journal/journald-native.c
index 851625d..9fd370f 100644
--- a/src/journal/journald-native.c
+++ b/src/journal/journald-native.c
@@ -273,6 +273,9 @@ void server_process_native_message(
if (s-forward_to_syslog)
server_forward_syslog(s, priority, identifier,
message, ucred, tv);
+if (s-forward_to_network)
+server_forward_syslog_network(s, priority, identifier,
message, ucred, tv);
+
if (s-forward_to_kmsg)
server_forward_kmsg(s, priority, identifier, message,
ucred);
diff --git a/src/journal/journald-server.c b/src/journal/journald-server.c
index 7ee8174..de4ef50 100644
--- a/src/journal/journald-server.c
+++ b/src/journal/journald-server.c
@@ -86,7 +86,7 @@ static const char* const split_mode_table[_SPLIT_MAX] = {
DEFINE_STRING_TABLE_LOOKUP(split_mode, SplitMode);
DEFINE_CONFIG_PARSE_ENUM(config_parse_split_mode, split_mode, SplitMode,
Failed to parse split mode setting);
-static uint64_t available_space(Server *s, bool verbose) {
+uint64_t available_space(Server *s, bool verbose) {
char ids[33];
_cleanup_free_ char *p = NULL;
sd_id128_t machine;
@@ -1356,6 +1356,35 @@ static int server_parse_config_file(Server *s) {
false, s);
}
+int config_parse_syslog_network_address(const char *unit,
+const char *filename,
+unsigned line,
+const char *section,
+unsigned section_line,
+const char *lvalue,
+int ltype,
+const char *rvalue,
+void *data,
+void *userdata) {
+Server *s = userdata;
+int r;
+
+assert(filename);
+assert(lvalue);
+assert(rvalue);
+assert(data);
+
+r =
