Strace of what? Rsyslog?
-- Florian Seifer Berater secunet Anwendungen, Managed Security Services Division Operational Services secunet Security Networks AG Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259 [email protected] Kurfürstenstraße 58, 45138 Essen, Germany www.secunet.com ----------------------------------------------------------------------- Sitz: Kurfürstenstraße 58, 45138 Essen Amtsgericht Essen HRB 13615 Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas Pleines Aufsichtsratsvorsitzender: Ralf Wintergerst -----Ursprüngliche Nachricht----- Von: rsyslog <[email protected]> Im Auftrag von Benoit DOLEZ via rsyslog Gesendet: Mittwoch, 11. November 2020 10:54 An: [email protected] Cc: Benoit DOLEZ <[email protected]> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages Hi, I think an strace (-s 8192) on the receiver host can be useful. Benoit Le 11/11/2020 à 10:20, Rainer Gerhards via rsyslog a écrit : > looks good! Looking forward to the real sample. You can also check > yourself if the frame is terminated with LF (hex 0a). In the sample it > is. > > Rainer > > El mié., 11 nov. 2020 a las 10:18, Seifer, Florian > (<[email protected]>) escribió: >> >> Ok I got it: >> >> 0x0000: 4500 0130 ad1e 4000 3d06 8571 0a35 cd0d E..0..@.=..q.5.. >> 0x0010: 0a35 28c1 c26c 0202 acec aacd c2a8 f13b .5(..l.........; >> 0x0020: 8018 001d 92c6 0000 0101 080a d80f e141 ...............A >> 0x0030: 8125 ea9a 3230 3230 2d31 312d 3131 5430 .%..2020-11-11T0 >> 0x0040: 393a 3134 3a35 362e 3436 365a 2031 302e 9:14:56.466Z.10. >> 0x0050: 3533 2e32 3035 2e32 202d 202d 202d 202d 53.205.2.-.-.-.- >> 0x0060: 2044 6965 7320 6973 7420 6569 6e65 2076 .Dies.ist.eine.v >> 0x0070: 6f6e 204c 6f67 2049 6e73 6967 6874 2065 on.Log.Insight.e >> 0x0080: 7273 7465 6c6c 7465 2075 6e64 2061 6e20 rstellte.und.an. >> 0x0090: 6c6f 6777 6174 6368 2e73 6563 756e 6574 logwatch.secunet >> 0x00a0: 2e64 653a 3531 3420 6d69 7420 5241 5720 .de:514.mit.RAW. >> 0x00b0: 5443 5020 2875 6e64 206d 6974 2054 6167 TCP.(und.mit.Tag >> 0x00c0: 7320 686f 7374 3d7a 6b7a 2920 6765 7365 s.host=zkz).gese >> 0x00d0: 6e64 6574 6520 5465 7374 6e61 6368 7269 ndete.Testnachri >> 0x00e0: 6368 7420 66c3 bc72 2064 6965 2045 7265 cht.f..r.die.Ere >> 0x00f0: 6967 6e69 7377 6569 7465 726c 6569 7475 ignisweiterleitu >> 0x0100: 6e67 206d 6974 2064 656d 204e 616d 656e ng.mit.dem.Namen >> 0x0110: 2027 416c 6172 6d61 6e6c 6167 6520 5363 .'Alarmanlage.Sc >> 0x0120: 6861 7266 2f55 6e73 6368 6172 6627 2e0a harf/Unscharf'.. >> >> Above is a test-packet I fired to test my new tcpdump options. >> >> Now I have to wait for another log to disappear >> >> -- >> Florian Seifer >> Berater secunet Anwendungen, Managed Security Services >> Division Operational Services >> secunet Security Networks AG >> >> Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259 >> [email protected] >> Kurfürstenstraße 58, 45138 Essen, Germany >> www.secunet.com >> >> ----------------------------------------------------------------------- >> Sitz: Kurfürstenstraße 58, 45138 Essen >> Amtsgericht Essen HRB 13615 >> Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas >> Pleines >> Aufsichtsratsvorsitzender: Ralf Wintergerst >> >> -----Ursprüngliche Nachricht----- >> Von: Rainer Gerhards <[email protected]> >> Gesendet: Mittwoch, 11. November 2020 09:34 >> An: Seifer, Florian <[email protected]> >> Cc: rsyslog-users <[email protected]> >> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages >> >> Maybe you can just use wireshark -- a screen shot from the gui is >> probably sufficient. >> >> Rainer >> >> El mié., 11 nov. 2020 a las 9:31, Seifer, Florian >> (<[email protected]>) escribió: >>> >>> Hello Rainer, >>> >>> I am not terribly firm with tcpdump, how would I go about doing that? >>> >>> Also I forgot to mention, the logs are always coming from the same Host >>> with exactly the same format. >>> ( A VmWare logInsight-Server if that’s relevant) so I doubt the messages >>> are suddenly changing form. >>> >>> >>> -----Ursprüngliche Nachricht----- >>> Von: Rainer Gerhards <[email protected]> >>> Gesendet: Mittwoch, 11. November 2020 09:26 >>> An: Seifer, Florian <[email protected]> >>> Cc: rsyslog-users <[email protected]> >>> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages >>> >>> looks good. my gut feeling is that some senders do not properly >>> terminate the syslog frame with '\n'. Can you check you captures for a >>> case where a message is missing and post all bytes of the last seen >>> and missing messages in the correct sequence? >>> >>> Rainer >>> >>> El mié., 11 nov. 2020 a las 9:16, Seifer, Florian >>> (<[email protected]>) escribió: >>>> >>>> # rsyslog configuration file >>>> >>>> # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html >>>> # or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html >>>> # If you experience problems, see >>>> http://www.rsyslog.com/doc/troubleshoot.html >>>> >>>> #### MODULES #### >>>> >>>> module(load="imuxsock" # provides support for local system logging >>>> (e.g. via >>>> >>>> logger command) >>>> SysSock.Use="off") # Turn off message reception via local log >>>> socket; >>>> # local messages are retrieved through imjournal >>>> now. >>>> module(load="imjournal" # provides access to the systemd >>>> journal >>>> StateFile="imjournal.state") # File to store the position in the >>>> journal >>>> #module(load="imklog") # reads kernel messages (the same are read from >>>> journald) >>>> #module(load"immark") # provides --MARK-- message capability >>>> >>>> # Provides UDP syslog reception >>>> # for parameters see http://www.rsyslog.com/doc/imudp.html >>>> module(load="imudp") # needs to be done just once >>>> input(type="imudp" port="514") >>>> >>>> # Provides TCP syslog reception >>>> # for parameters see http://www.rsyslog.com/doc/imtcp.html >>>> module(load="imtcp") # needs to be done just once >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> #### GLOBAL DIRECTIVES #### >>>> >>>> # Where to place auxiliary files >>>> global(workDirectory="/var/lib/rsyslog") >>>> >>>> # Use default timestamp format >>>> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat") >>>> >>>> # Include all config files in /etc/rsyslog.d/ >>>> include(file="/etc/rsyslog.d/*.conf" mode="optional") >>>> >>>> #### RULES #### >>>> >>>> # Log all kernel messages to the console. >>>> # Logging much else clutters up the screen. >>>> #kern.* /dev/console >>>> >>>> # Log anything (except mail) of level info or higher. >>>> # Don't log private authentication messages! >>>> *.info;mail.none;authpriv.none;cron.none /var/log/messages >>>> >>>> # The authpriv file has restricted access. >>>> authpriv.* /var/log/secure >>>> >>>> # Log all the mail messages in one place. >>>> mail.* -/var/log/maillog >>>> >>>> >>>> # Log cron stuff >>>> cron.* /var/log/cron >>>> >>>> # Everybody gets emergency messages >>>> *.emerg :omusrmsg:* >>>> >>>> # Save news errors of level crit and higher in a special file. >>>> uucp,news.crit /var/log/spooler >>>> >>>> # Save boot messages also to boot.log >>>> local7.* /var/log/boot.log >>>> >>>> >>>> >>>> >>>> >>>> >>>> # ### sample forwarding rule ### >>>> #action(type="omfwd" >>>> # An on-disk queue is created for this action. If the remote host is >>>> # down, messages are spooled to disk and sent when it is up again. >>>> #queue.filename="fwdRule1" # unique name prefix for spool files >>>> #queue.maxdiskspace="1g" # 1gb space limit (use as much as >>>> possible) >>>> #queue.saveonshutdown="on" # save messages to disk on shutdown >>>> #queue.type="LinkedList" # run asynchronously >>>> #action.resumeRetryCount="-1" # infinite retries if host is down >>>> # Remote Logging (we use TCP for reliable delivery) >>>> # remote_host is: name/ip, e.g. 192.168.0.1, port optional e.g. 10514 >>>> #Target="remote_host" Port="XXX" Protocol="tcp") >>>> >>>> >>>> # Diese Regel erstellt pro Hostname ein eigenes Verzeichnis >>>> $template RemoteHost,"/var/log/remote/%hostname%/%programname%.log" >>>> >>>> # Der Regelsatz remote wendet auf alle eingehenden Logs das Template >>>> RemoteHost an: >>>> >>>> $RuleSet remote >>>> *.* ?RemoteHost >>>> >>>> # Nun binden wir den Regelsatz "remote" an alle syslog Meldungen von Außen. >>>> >>>> input(type="imtcp" port="514" ruleset="remote" >>>> supportoctetcountedframing="off") >>>> >>>> $EscapeControlCharactersOnReceive on >>>> >>>> $RepeatedMsgReduction off >>>> >>>> $SystemLogRateLimitInterval 0 >>>> >>>> $DebugLevel 2 >>>> >>>> $DebugFile /var/log/rsyslog/debug.log >>>> >>>> -- >>>> Florian Seifer >>>> Berater secunet Anwendungen, Managed Security Services >>>> Division Operational Services >>>> secunet Security Networks AG >>>> >>>> Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259 >>>> [email protected] >>>> Kurfürstenstraße 58, 45138 Essen, Germany >>>> www.secunet.com >>>> >>>> ----------------------------------------------------------------------- >>>> Sitz: Kurfürstenstraße 58, 45138 Essen >>>> Amtsgericht Essen HRB 13615 >>>> Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas >>>> Pleines >>>> Aufsichtsratsvorsitzender: Ralf Wintergerst >>>> >>>> -----Ursprüngliche Nachricht----- >>>> Von: Rainer Gerhards <[email protected]> >>>> Gesendet: Mittwoch, 11. November 2020 09:13 >>>> An: rsyslog-users <[email protected]> >>>> Cc: Seifer, Florian <[email protected]> >>>> Betreff: Re: [rsyslog] rsyslog ignoring random incoming tcp-messages >>>> >>>> config? >>>> >>>> El mié., 11 nov. 2020 a las 9:11, Seifer, Florian via rsyslog >>>> (<[email protected]>) escribió: >>>>> >>>>> Hello, >>>>> >>>>> I have a rather strange problem with an rsyslog-machine. >>>>> >>>>> I configured it to process incoming logs over network via tcp port 514 >>>>> using imtcp. >>>>> >>>>> The incoming logs on that connection are simply stored in a specific >>>>> folder in a log-file. >>>>> This setup works 90% of the time but sometimes logs get "lost". >>>>> >>>>> I have no idea what causes it. I set up a tcpdump on the receiving >>>>> machine and I can confirm that the packages are indeed being delivered. >>>>> But for some reason rsyslog does not notice them. I can find all other >>>>> log-messages mentioned in the rsyslog-debug log, where they are noticed >>>>> and processed correctly. >>>>> >>>>> The ones that don't make it to the logfiles are no where to be found. I >>>>> cannot find any difference in these files, so I would expect them to be >>>>> handled identically. >>>>> >>>>> Has anyone of you ever had similar problems and found a solution? Its >>>>> not a firewall or network problem as the packages are definitely reaching >>>>> the client. >>>>> >>>>> I am grateful for any support you guys can provide >>>>> >>>>> With kind regards >>>>> >>>>> -- >>>>> Florian Seifer >>>>> Berater secunet Anwendungen, Managed Security Services >>>>> Division Operational Services >>>>> secunet Security Networks AG >>>>> >>>>> Tel.: +49 201 5454-2297, Fax: +49 201 5454-1259 >>>>> [email protected] >>>>> Kurfürstenstraße 58, 45138 Essen, Germany >>>>> www.secunet.com >>>>> >>>>> ----------------------------------------------------------------------- >>>>> Sitz: Kurfürstenstraße 58, 45138 Essen >>>>> Amtsgericht Essen HRB 13615 >>>>> Vorstand: Axel Deininger (Vors.), Torsten Henn, Dr. Kai Martius, Thomas >>>>> Pleines >>>>> Aufsichtsratsvorsitzender: Ralf Wintergerst >>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > -- Benoit DOLEZ GSM: +33 6 21 05 91 69 mailto:[email protected] _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
