Re: py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
Hi Michael, Marek, 2018-06-22 22:48 GMT+02:00 Marek Zarychta : > Patch compiles fine and I can confirm, that it resolves the issue. Thanks for testing to both of you. Really appreciated. I've just committed this patch as r335565. As it is a bit more intrusive than the previous patch I wrote, let's give this a week to settle in HEAD before merging it to 11-STABLE. I'll close PR 229236 once it's been merged. -- Ed Schouten Nuxi, 's-Hertogenbosch, the Netherlands ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
On Fri, Jun 22, 2018 at 09:11:06PM +0200, Ed Schouten wrote:
> Hi Marek,
>
> [ +glebius ]
>
> Thanks for reporting this!
>
> 2018-06-22 18:54 GMT+02:00 Michael Grimm :
> >> Failed to parse TIMESTAMP from x.x.x.x: 12403: Jun 22 17:31:38 CEST:
> >> %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17,
> >> changed state to down
> >
> > Ah, yes! Haven't thought about running syslogd in debugging mode:
> >
> > Failed to parse TIMESTAMP from x.x.x.x: fail2ban.filter [79598]:
> > INFO […]
>
> This is interesting. As fail2ban uses Python's logging framework, I
> managed to reproduce this with the following script:
>
> #!/usr/bin/env python3
> import logging.handlers
> logging.basicConfig(handlers=[
> logging.handlers.SysLogHandler(
> '/var/run/log', facility=logging.handlers.SysLogHandler.LOG_LOCAL7)
> ])
> logging.warning('Hi')
>
> This will write the following message to syslogd:
>
> sendto(3,"<188>WARNING:root:Hi\0",21,0,NULL,0) = 21 (0x15)
>
> This message gets rejected by syslogd, due to the change made in
> r326573, which later got adjusted by me and subsequently MFCed:
>
> https://svnweb.freebsd.org/base?view=revision&revision=326573
>
> Gleb, what are your thoughts on the attached patch? It alters syslogd
> to let the 'legacy' RFC 3164 parser also accept messages without a
> timestamp. The time on the syslogd server will be used instead.
>
> Michael, Marek, could you please give this patch a try? Thanks!
>
Hi Ed,
Thank you for expedited effort.
Patch compiles fine and I can confirm, that it resolves the issue.
--
Marek Zarychta
signature.asc
Description: PGP signature
Re: py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
On 22. Jun 2018, at 22:28, Ed Schouten wrote: > 2018-06-22 22:06 GMT+02:00 Michael Grimm : >> After applying your patch: >>Jun 22 21:22:01 HOSTNAME [31033]: NOTICE [JAILNAME] >> Unban x.x.x.x >> >> Watch: 'fail2ban.actions' -the service- is missing. > > That's likely due to the fact that it now interprets the first word in > the message as the remote hostname, which gets discarded. > > Attached is a somewhat refined patch that only tries to parse the > hostname in remote messages if they are preceded by a timestamp. If > the timestamp is missing, it assumes the entire payload is the > message. Can you give this one a try? Thanks! Great, you fixed it again in very short time, and I really do appreciate this! Now with patch v2: Jun 22 22:39:59 HOSTNAME fail2ban.actions [72605]: NOTICE [JAILNAME] Restore Ban x.x.x.x Thank you very, very much, Michael ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
Hi Michael, 2018-06-22 22:06 GMT+02:00 Michael Grimm : > After applying your patch: > Jun 22 21:22:01 HOSTNAME [31033]: NOTICE [JAILNAME] > Unban x.x.x.x > > Watch: 'fail2ban.actions' -the service- is missing. That's likely due to the fact that it now interprets the first word in the message as the remote hostname, which gets discarded. Attached is a somewhat refined patch that only tries to parse the hostname in remote messages if they are preceded by a timestamp. If the timestamp is missing, it assumes the entire payload is the message. Can you give this one a try? Thanks! -- Ed Schouten Nuxi, 's-Hertogenbosch, the Netherlands syslogd-optional-timestamp-v2.diff Description: Binary data ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
Hi Ed,
On Fri, Jun 22, 2018 at 09:11:06PM +0200, Ed Schouten wrote:
E> > Ah, yes! Haven't thought about running syslogd in debugging mode:
E> >
E> > Failed to parse TIMESTAMP from x.x.x.x: fail2ban.filter [79598]:
INFO […]
E>
E> This is interesting. As fail2ban uses Python's logging framework, I
E> managed to reproduce this with the following script:
E>
E> #!/usr/bin/env python3
E> import logging.handlers
E> logging.basicConfig(handlers=[
E> logging.handlers.SysLogHandler(
E> '/var/run/log', facility=logging.handlers.SysLogHandler.LOG_LOCAL7)
E> ])
E> logging.warning('Hi')
E>
E> This will write the following message to syslogd:
E>
E> sendto(3,"<188>WARNING:root:Hi\0",21,0,NULL,0) = 21 (0x15)
E>
E> This message gets rejected by syslogd, due to the change made in
E> r326573, which later got adjusted by me and subsequently MFCed:
E>
E> https://svnweb.freebsd.org/base?view=revision&revision=326573
E>
E> Gleb, what are your thoughts on the attached patch? It alters syslogd
E> to let the 'legacy' RFC 3164 parser also accept messages without a
E> timestamp. The time on the syslogd server will be used instead.
E>
E> Michael, Marek, could you please give this patch a try? Thanks!
I didn't examine the patch thoroughly, but I agree that looks like
we have no other choice as to support the legacy and normal messages
at the same time.
--
Gleb Smirnoff
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
On 22. Jun 2018, at 21:26, Michael Grimm wrote: > On 22. Jun 2018, at 21:11, Ed Schouten wrote: >> Michael, Marek, could you please give this patch a try? Thanks! > > Recompiled world (FreeBSD 11.2-STABLE r335532), substituted syslogd with the > re-compiled one, and: > > Thank you! Your patch is working w.r.t. fail2ban logging to SYSLOG. Perfect! Now I realised that there is a minor glitch: logfile+logger: Jun 22 19:01:48 HOSTNAME fail2ban.filter: 2018-06-22 19:01:48,637 fail2ban.actions[85544]: NOTICE [JAILNAME] Unban x.x.x.x Old syslogd before MFC: May 30 15:39:41 HOSTNAME fail2ban.actions [929]: NOTICE [JAILNAME] Unban x.x.x.x After applying your patch: Jun 22 21:22:01 HOSTNAME [31033]: NOTICE [JAILNAME] Unban x.x.x.x Watch: 'fail2ban.actions' -the service- is missing. Regards, Michael ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
On 22. Jun 2018, at 21:11, Ed Schouten wrote: > Gleb, what are your thoughts on the attached patch? It alters syslogd > to let the 'legacy' RFC 3164 parser also accept messages without a > timestamp. The time on the syslogd server will be used instead. > > Michael, Marek, could you please give this patch a try? Thanks! Recompiled world (FreeBSD 11.2-STABLE r335532), substituted syslogd with the re-compiled one, and: Thank you! Your patch is working w.r.t. fail2ban logging to SYSLOG. Perfect! Thank you very much for this fast fix, and regards, Michael ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
Hi Marek,
[ +glebius ]
Thanks for reporting this!
2018-06-22 18:54 GMT+02:00 Michael Grimm :
>> Failed to parse TIMESTAMP from x.x.x.x: 12403: Jun 22 17:31:38 CEST:
>> %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17,
>> changed state to down
>
> Ah, yes! Haven't thought about running syslogd in debugging mode:
>
> Failed to parse TIMESTAMP from x.x.x.x: fail2ban.filter [79598]: INFO
> […]
This is interesting. As fail2ban uses Python's logging framework, I
managed to reproduce this with the following script:
#!/usr/bin/env python3
import logging.handlers
logging.basicConfig(handlers=[
logging.handlers.SysLogHandler(
'/var/run/log', facility=logging.handlers.SysLogHandler.LOG_LOCAL7)
])
logging.warning('Hi')
This will write the following message to syslogd:
sendto(3,"<188>WARNING:root:Hi\0",21,0,NULL,0) = 21 (0x15)
This message gets rejected by syslogd, due to the change made in
r326573, which later got adjusted by me and subsequently MFCed:
https://svnweb.freebsd.org/base?view=revision&revision=326573
Gleb, what are your thoughts on the attached patch? It alters syslogd
to let the 'legacy' RFC 3164 parser also accept messages without a
timestamp. The time on the syslogd server will be used instead.
Michael, Marek, could you please give this patch a try? Thanks!
--
Ed Schouten
Nuxi, 's-Hertogenbosch, the Netherlands
Index: usr.sbin/syslogd/syslogd.c
===
--- usr.sbin/syslogd/syslogd.c (revision 335314)
+++ usr.sbin/syslogd/syslogd.c (working copy)
@@ -1172,45 +1172,43 @@
size_t i, msglen;
char line[MAXLINE + 1];
- /* Parse the timestamp provided by the remote side. */
- if (strptime(msg, RFC3164_DATEFMT, &tm_parsed) !=
- msg + RFC3164_DATELEN || msg[RFC3164_DATELEN] != ' ') {
- dprintf("Failed to parse TIMESTAMP from %s: %s\n", from, msg);
- return;
- }
- msg += RFC3164_DATELEN + 1;
+ /* Parse the timestamp provided by the remote side, if any. */
+ timestamp = NULL;
+ if (strptime(msg, RFC3164_DATEFMT, &tm_parsed) ==
+ msg + RFC3164_DATELEN && msg[RFC3164_DATELEN] == ' ') {
+ msg += RFC3164_DATELEN + 1;
+ if (!RemoteAddDate) {
+ struct tm tm_now;
+ time_t t_now;
+ int year;
- if (!RemoteAddDate) {
- struct tm tm_now;
- time_t t_now;
- int year;
-
- /*
- * As the timestamp does not contain the year number,
- * daylight saving time information, nor a time zone,
- * attempt to infer it. Due to clock skews, the
- * timestamp may even be part of the next year. Use the
- * last year for which the timestamp is at most one week
- * in the future.
- *
- * This loop can only run for at most three iterations
- * before terminating.
- */
- t_now = time(NULL);
- localtime_r(&t_now, &tm_now);
- for (year = tm_now.tm_year + 1;; --year) {
- assert(year >= tm_now.tm_year - 1);
- timestamp_remote.tm = tm_parsed;
- timestamp_remote.tm.tm_year = year;
- timestamp_remote.tm.tm_isdst = -1;
- timestamp_remote.usec = 0;
- if (mktime(×tamp_remote.tm) <
- t_now + 7 * 24 * 60 * 60)
-break;
+ /*
+ * As the timestamp does not contain the year
+ * number, daylight saving time information, nor
+ * a time zone, attempt to infer it. Due to
+ * clock skews, the timestamp may even be part
+ * of the next year. Use the last year for which
+ * the timestamp is at most one week in the
+ * future.
+ *
+ * This loop can only run for at most three
+ * iterations before terminating.
+ */
+ t_now = time(NULL);
+ localtime_r(&t_now, &tm_now);
+ for (year = tm_now.tm_year + 1;; --year) {
+assert(year >= tm_now.tm_year - 1);
+timestamp_remote.tm = tm_parsed;
+timestamp_remote.tm.tm_year = year;
+timestamp_remote.tm.tm_isdst = -1;
+timestamp_remote.usec = 0;
+if (mktime(×tamp_remote.tm) <
+t_now + 7 * 24 * 60 * 60)
+ break;
+ }
+ timestamp = ×tamp_remote;
}
- timestamp = ×tamp_remote;
- } else
- timestamp = NULL;
+ }
/*
* A single space character MUST also follow the HOSTNAME field.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
On Fri, Jun 22, 2018 at 08:13:59PM +0200, Kurt Jaeger wrote: > > I have rolled back to r334835. The issue has gone. Should a PR be > > created about this regression ? > > Yes, please. > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=229236 -- Marek Zarychta signature.asc Description: PGP signature
Re: py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
Hi! > >> Could you please give any advice or workaround for this issue? > > I switched to a workaround for the time being which you might use as well > > in a similar way: > > > > #) configure fail2ban to use /var/log/faillog > > #) run something like that in the background: > > > > nohup tail -q -F /var/log/fail2ban.log | logger -t fail2ban.filter -p > > daemon.notice & > > > > #) to let this workaround survive a reboot you need to use a script fired > > up from /etc/rc.d > > > > > > I have rolled back to r334835. The issue has gone. Should a PR be > created about this regression ? Yes, please. > BTW a few decent syslog daemons are available in /usr/ports/sysutils. -- p...@opsec.eu+49 171 31013722 years to go ! ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
W dniu 2018.06.22 o 19:14, Michael Grimm pisze: > On 22. Jun 2018, at 17:59, Marek Zarychta > wrote: > >> Could you please give any advice or workaround for this issue? > I switched to a workaround for the time being which you might use as well in > a similar way: > > #) configure fail2ban to use /var/log/faillog > #) run something like that in the background: > > nohup tail -q -F /var/log/fail2ban.log | logger -t fail2ban.filter -p > daemon.notice & > > #) to let this workaround survive a reboot you need to use a script fired up > from /etc/rc.d > > I have rolled back to r334835. The issue has gone. Should a PR be created about this regression ? BTW a few decent syslog daemons are available in /usr/ports/sysutils. -- Marek Zarychta signature.asc Description: OpenPGP digital signature
Re: py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
On 22. Jun 2018, at 17:59, Marek Zarychta wrote: > Could you please give any advice or workaround for this issue? I switched to a workaround for the time being which you might use as well in a similar way: #) configure fail2ban to use /var/log/faillog #) run something like that in the background: nohup tail -q -F /var/log/fail2ban.log | logger -t fail2ban.filter -p daemon.notice & #) to let this workaround survive a reboot you need to use a script fired up from /etc/rc.d Regards, Michael ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
Marek Zarychta wrote: > On Fri, Jun 22, 2018 at 03:12:05PM +0200, Michael Grimm wrote: >> Hi, >> >> this is 11.2-STABLE (r335532), and I am referring to the recent MFC of >> syslogd modifications [1]. >> >> Because I cannot judge whether fail2ban lacks support for the renewed >> syslogd or syslogd has an issue in receiving fail2ban messages I do >> crosspost this mail to ports and stable. >> >> I do have fail2ban configured to report to SYSLOG: >> >> logtarget = SYSLOG >> syslogsocket = auto >> >> But now, after upgrading to the new syslogd fail2ban refuses to report to >> syslogd; no single message gets recorded [2]. >> >> I did try to modify the syslogsocket setting to /var/run/log without >> success. Pointing logtarget to a regular files tells me that fail2ban is >> running as expected, it only lacks reporting to SYSLOG. >> >> #) Does anyone else has running py-fail2ban at >= r335059 and can confirm my >> observations? >> #) Any ideas how to debug this issue? >> >> Thank you in advance and regards, >> Michael >> >> >> [1] >> https://svnweb.freebsd.org/base/stable/11/usr.sbin/syslogd/Makefile?revision=335059&view=markup&sortby=file >> [2] both syslogd and fail2ban are running at the host, thus another issue >> with syslogd fixed in >>https://svnweb.freebsd.org/base?view=revision&sortby=file&revision=335314 >> does not apply >> > > This is probably connected with the lack of handling of non-RFC > compliant timestamps. > > My syslog server also suffers from this issue. It stopped logging > messages from old Cisco equipment and some newer Netgear switches. > Running it in debug mode gives some clue: > > Failed to parse TIMESTAMP from x.x.x.x: 12403: Jun 22 17:31:38 CEST: > %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, > changed state to down Ah, yes! Haven't thought about running syslogd in debugging mode: Failed to parse TIMESTAMP from x.x.x.x: fail2ban.filter [79598]: INFO […] > Could you please give any advice or workaround for this issue? I cannot answer whether it might be possible to either tell syslogd to accept legacy timestamps [1] or configure fail2ban (or your applications) to switch to using RFC5424 compliant timestamps. [1] I did try to set '-O rfc3164' starting syslogd to no avail Anyone? Regards, Michael ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
On Fri, Jun 22, 2018 at 03:12:05PM +0200, Michael Grimm wrote: > Hi, > > this is 11.2-STABLE (r335532), and I am referring to the recent MFC of > syslogd modifications [1]. > > Because I cannot judge whether fail2ban lacks support for the renewed syslogd > or syslogd has an issue in receiving fail2ban messages I do crosspost this > mail to ports and stable. > > I do have fail2ban configured to report to SYSLOG: > > logtarget = SYSLOG > syslogsocket = auto > > But now, after upgrading to the new syslogd fail2ban refuses to report to > syslogd; no single message gets recorded [2]. > > I did try to modify the syslogsocket setting to /var/run/log without success. > Pointing logtarget to a regular files tells me that fail2ban is running as > expected, it only lacks reporting to SYSLOG. > > #) Does anyone else has running py-fail2ban at >= r335059 and can confirm my > observations? > #) Any ideas how to debug this issue? > > Thank you in advance and regards, > Michael > > > [1] > https://svnweb.freebsd.org/base/stable/11/usr.sbin/syslogd/Makefile?revision=335059&view=markup&sortby=file > [2] both syslogd and fail2ban are running at the host, thus another issue > with syslogd fixed in > https://svnweb.freebsd.org/base?view=revision&sortby=file&revision=335314 > does not apply > This is probably connected with the lack of handling of non-RFC compliant timestamps. My syslog server also suffers from this issue. It stopped logging messages from old Cisco equipment and some newer Netgear switches. Running it in debug mode gives some clue: Failed to parse TIMESTAMP from x.x.x.x: 12403: Jun 22 17:31:38 CEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/17, changed state to down Could you please give any advice or workaround for this issue? -- Marek Zarychta signature.asc Description: PGP signature
py-fail2ban turned silent after syslogd rollout (r335059, stable/11)
Hi, this is 11.2-STABLE (r335532), and I am referring to the recent MFC of syslogd modifications [1]. Because I cannot judge whether fail2ban lacks support for the renewed syslogd or syslogd has an issue in receiving fail2ban messages I do crosspost this mail to ports and stable. I do have fail2ban configured to report to SYSLOG: logtarget = SYSLOG syslogsocket = auto But now, after upgrading to the new syslogd fail2ban refuses to report to syslogd; no single message gets recorded [2]. I did try to modify the syslogsocket setting to /var/run/log without success. Pointing logtarget to a regular files tells me that fail2ban is running as expected, it only lacks reporting to SYSLOG. #) Does anyone else has running py-fail2ban at >= r335059 and can confirm my observations? #) Any ideas how to debug this issue? Thank you in advance and regards, Michael [1] https://svnweb.freebsd.org/base/stable/11/usr.sbin/syslogd/Makefile?revision=335059&view=markup&sortby=file [2] both syslogd and fail2ban are running at the host, thus another issue with syslogd fixed in https://svnweb.freebsd.org/base?view=revision&sortby=file&revision=335314 does not apply ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
