Hey folks, Hoping someone has the expertise to help me out here. We have a Syslog server running CentOS 7.9, kernel 5.18.5. It's acting as the centralized point for Syslog (TCP + UDP) ingestion for 100+ Syslog devices. Something is causing a what I think is the kernel (udp_queue_rcv_one_skb) to drop 10k+ packets in a single go, intermittently. I expect given the nature of UDP in general to lose some packets, but 5-10% of inbound packets is excessive, and I think, avoidable. This is our only server experiencing issues - though it's also the highest processing one by a significant margin. Dropwatch's output occurred in the span of 1-2 seconds. The security-config-omsagent.conf file is the rsyslog sub configuration for Forwarding to Microsoft Sentinel. That config has been slightly modified to remove actual subnet ranges. I've removed all local disk logging.
>From my research online, this appears to be a queue issue. I've tried fixing >it using the rsyslog documentation, but evidently haven't made it very far. >Any thoughts or obvious mistakes? Cheers, Mike System Config CPUs: 8 RAM: 16GB Disk Space: 2x 128GB disks (OS + DATA) net.core.rmem_default = 33554432 net.core.rmem_max = 268435456 net.ipv4.tcp_rmem = 4096 131072 6291456 net.ipv4.udp_rmem_min = 4096 Dropwatch -l kas [cid:[email protected]] /etc/rsyslog.conf # rsyslog configuration file # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html global(net.enableDNS="off") #### MODULES #### module( load="impstats" interval="1" severity="7" resetCounters="on" log.file="/var/syslog/impstats.log" log.syslog="off" ) # The imjournal module bellow is now used as a message source instead of imuxsock. $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imjournal # provides access to the systemd journal #$ModLoad imklog # reads kernel messages (the same are read from journald) #$ModLoad immark # provides --MARK-- message capability # Provides UDP syslog reception #$ModLoad imudp #$UDPServerRun 514 module(load="imudp" threads="16") input(type="imudp" port="514" rcvbufSize="256m") # Provides TCP syslog reception $ModLoad imtcp $InputTCPServerRun 514 #### Templates #### $template RemoteIP,"/var/syslog/%FROMHOST-IP%.log" #$template NetApp,"%timestamp% %fromhost-ip% %msg%\n" #$template Ubiquiti,"%msg% %fromhost-ip% Mystic-Ubiquiti\n" #### GLOBAL DIRECTIVES #### # Where to place auxiliary files $WorkDirectory /var/lib/rsyslog # Use default timestamp format #$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat $ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format # File syncing capability is disabled by default. This feature is usually not required, # not useful and an extreme performance hit #$ActionFileEnableSync on # Include all config files in /etc/rsyslog.d/ $IncludeConfig /etc/rsyslog.d/*.conf # Turn off message reception via local log socket; # local messages are retrieved through imjournal now. $OmitLocalLogging on # File to store the position in the journal $IMJournalStateFile imjournal.state # Performance Tuning # $ActionQueueWorkerThreads 2000 $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark 600000 #### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail authpriv, cron) # Dont log private authentication messages! #*.*;mail.none;authpriv.none;cron.none ?RemoteIP # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg :omusrmsg:* # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log # local7.* /var/syslog/boot.log /etc/rsyslog.d/security-config-omsagent.conf # [Firewall Log Filtering] # :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)" stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex, "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop :msg, ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP multicast routing failed)" stop :msg, ereregex, "(TCP_7680)" stop if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr" then @127.0.0.1:25224 & stop local0.info @127.0.0.1:25224 & stop local1.info @127.0.0.1:25224 & stop local2.info @127.0.0.1:25224 & stop local3.info @127.0.0.1:25224 & stop local4.info @127.0.0.1:25224 & stop local5.info @127.0.0.1:25224 & stop local6.info @127.0.0.1:25224 & stop local7.info @127.0.0.1:25224 & stop auth.* @127.0.0.1:25224 & stop authpriv.* @127.0.0.1:25224 & stop daemon.info @127.0.0.1:25224 & stop syslog.* @127.0.0.1:25224 & stop ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop user.* @127.0.0.1:25224 & stop [cid:[email protected]] Michael Redbourne (BCS) Pronouns: he/him Senior Security Analyst Main (C): 1-506-230-3071 VOIP : 1-506-606-0384 Service Desk: 1-877-274-2349 www.bulletproofsi.com<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2Fwww.bulletproofsi.com%2F__%3B!!NFWRZ6kECLqu!pxjMi1iqf_xCGX9L_kgynzgYHwxIacBtBbFWe63ZKWM7Mwo1M0-12T9CrPB_dAqlKGcyX6wHnrGaPpRliSwsShf6YtO2onamhERr%24&data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7C9ebf1cab2a5041e45f2108da8ffc7cd0%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C637980611840534929%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=k5y21GBCF2DAfVgRxUUfVsTifx7G036ytfI0%2BPolIgw%3D&reserved=0> ________________________________________ This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated. Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie ?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions ou des pr?occupations, veuillez contacter notre centre de service ? la client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration. ________________________________________
_______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
