For a udp message, I see this in the debug log: https://gist.github.com/dmeiser/30ba9ef626ab3fb6fd9cf865a88a7d1e
For a tcp message, I see this in debug: https://gist.github.com/dmeiser/24360ae40cf81573b833241c622b3b6a Here is the entire debug session: https://gist.github.com/dmeiser/11cf54f26834a7c09965c5bac098a727 Also, while I am using 7.4.7 in this session, I have tried the latest release from the official repo, as well. Thanks for the help! On Wed, Apr 6, 2016 at 8:42 PM David Lang <[email protected]> wrote: > On Thu, 7 Apr 2016, David Meiser wrote: > > > The log is coming from the debug messages by running rsyslog -d -n as > root > > on the generic forwarder machine. This line comes from the debug output: > > 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15: <5>Apr 6 > > 15:56:34 user.name: hello world > > > > I snipped the udp configuration from the config. It is identical except > > for the fact it uses imudp, not imptcp. > > > > I did open port 514. I also had the thought that it could be an selinux > > problem so I disabled selinux. Then I moved it to 10514 and 1514 and > > opened those ports. And then I disabled the firewall altogether. Each > > permutation had the same result. > > > > Since I can use ncat to copy/paste a message which is received and > > processed, I'm pretty sure there's something non-networking related. > > Ok, so if you are looking at the debug log and it's seeing the message > arrive, > then rsyslog is getting and processing the message. What does the debug > log show > happens after the message is recevied? it should show you everything > that's done > to the message, and you can look at the same thing for the UDP message to > see > what's different about them. > > David Lang > > > On Wed, Apr 6, 2016 at 8:08 PM David Lang <[email protected]> wrote: > > > >> On Wed, 6 Apr 2016, David Meiser wrote: > >> > >>> I've tried it from 3 different locations, including a server on that > >> vlan. > >>> Here's why I think it's an rsyslog configuration issue: the logger > >> messages > >>> I send via TCP do show up in the debug log. Here's an example: > >>> > >>> 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15: <5>Apr > 6 > >>> 15:56:34 user.name: hello world > >> > >> I still don't understand what this is an example of. Where are you > getting > >> this > >> from? > >> > >>> That isn't processed. Change it to UDP, it goes through no problem. > >>> > >>> In regards to the APC UPS message, I pulled the message from the > rsyslog > >>> debug log. > >> > >> so if you go to server A and send a message to server B via UDP it shows > >> up, but > >> if you send it by TCP it doesn't? > >> > >> The config you show below has no UDP configured. > >> > >> There are many distros (including RedHat) that block TCP 514 by default, > >> but > >> allow UDP 514, so check your iptables rules > >> > >> David Lang > >> > >>> On Wed, Apr 6, 2016, 5:06 PM David Lang <[email protected]> wrote: > >>> > >>>> On Wed, 6 Apr 2016, David Meiser wrote: > >>>> > >>>>> As I continue to try to troubleshoot this, I tried using a direct > ncat > >>>>> connection. If I send "1.1.1.1 hello world" (simulating an IP > address > >>>> and > >>>>> message), the output to the log file is: > >>>> > >>>> output to what logfile with what config? Is this the config shown > below? > >>>> > >>>> note, workerthreads=4 is almost certinly wrong, you need a very > complex > >>>> config > >>>> to need even two threads. Having a queue on a write to a file is also > >>>> almost > >>>> always the wrong thing to do. > >>>> > >>>>> Apr 6 16:25:41 1 > >>>>> Apr 6 16:25:41 .1.1 hello world > >>>> > >>>> not a valid syslog message > >>>> > >>>>> If I send "1\.1\.1\.1 hello world" the output is this: > >>>>> Apr 6 16:26:16 . > >>>>> Apr 6 16:26:16 . > >>>>> Apr 6 16:26:16 . > >>>>> Apr 6 16:26:16 h > >>>>> Apr 6 16:26:16 ello world > >>>> > >>>> also not a valid syslog message > >>>> > >>>>> If I send "myserver.domain.com hello world" the output is this: > >>>>> Apr 6 16:28:28 myserver.domain.com hello world > >>>> > >>>> also not a valid syslog message > >>>> > >>>>> If I send a raw syslog message "<13>Apr 6 14:37:38 1.1.1.1 Test > >> Syslog." > >>>> I > >>>>> get: > >>>>> Apr 6 14:37:38 1.1.1.1 Test Syslog. > >>>> > >>>> This is a valid syslog message, and is the output I would expect if > you > >> are > >>>> doing a simple write with the traditional format. > >>>> > >>>>> So, I go back to the original device that sent the TCP syslog message > >> (an > >>>>> APC UPS) and send another test directly to the server and I see this > in > >>>> the > >>>>> logs: > >>>>> 5015.675372226:7f6ce2981700: imptcp: data(131072) on socket 19: > >> <13>Apr 6 > >>>>> 16:36:54 1.1.1.1 APC: Test Syslog.rom o world > >>>> > >>>> how are you seeing this. > >>>> > >>>>> And in the log I get nothing. > >>>> > >>>> so if it works from some places and not others, the first thing to do > is > >>>> to look > >>>> at the network and see if there are any network things that would > block > >> the > >>>> communication from the place that doesn't work, but don't block it > from > >> the > >>>> place that does work. > >>>> > >>>> David Lang > >>>> > >>>>> On Wed, Apr 6, 2016 at 4:03 PM David Meiser <[email protected]> > wrote: > >>>>> > >>>>>> I am trying to setup a generic syslog forwarder that accepts > messages > >> on > >>>>>> tcp & udp. Using imptcp, I see messages come in, but no rulesets > are > >>>>>> processed. My ruleset, right now, is set to take tcp messages and > >> just > >>>>>> output them to file (for troubleshooting). UDP works fine. > >>>>>> > >>>>>> Here is what I see in debug mode: > >>>>>> 2594.333580185:7fd29ab2b700: imptcp: epoll returned 1 events > >>>>>> 2594.333594886:7fd29ab2b700: imptcp: new connection on listen > socket 9 > >>>>>> 2594.334657804:7fd29ab2b700: imptcp: added socket 15 to epoll[8] set > >>>>>> 2594.334670004:7fd29ab2b700: imptcp going on epoll_wait > >>>>>> 2594.334673904:7fd29ab2b700: imptcp: epoll returned 1 events > >>>>>> 2594.334686204:7fd29ab2b700: imptcp: new activity on session socket > 15 > >>>>>> 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15: > >> <5>Apr 6 > >>>>>> 15:56:34 user.name: hello world 2594.334701804:7fd29ab2b700: > imptcp: > >>>>>> removing socket 15 from epoll[8] set 2594.335235814:7fd29ab2b700: > >>>> imptcp: > >>>>>> session on socket 15 closed with iRet 0. > 2594.335240814:7fd29ab2b700: > >>>>>> imptcp going on epoll_wait > >>>>>> > >>>>>> Here is my config: > >>>>>> module(load="imptcp") > >>>>>> > >>>>>> input( > >>>>>> type="imptcp" > >>>>>> port="514" > >>>>>> ruleset="remote" > >>>>>> ) > >>>>>> > >>>>>> ruleset( > >>>>>> name="remote" queue.workerthreads="4" > >>>>>> queue.filename="srvrfwd" > >>>>>> queue.type="LinkedList" > >>>>>> queue.syncqueuefiles="on" > >>>>>> queue.maxdiskspace="10g" > >>>>>> queue.saveonshutdown="on" > >>>>>> queue.size="10000000" > >>>>>> ) { > >>>>>> action( > >>>>>> type="omfile" > >>>>>> file="/var/log/debug" > >>>>>> ) > >>>>>> } > >>>>>> > >>>>>> I've also tried the sample config from the website: > >>>>>> input(type="imptcp" port="10514" ruleset="writeRemoteData") > >>>>>> ruleset(name="writeRemoteData" queue.type="fixedArray" > >>>> queue.size="250000" > >>>>>> queue.dequeueBatchSize="4096" queue.workerThreads="4" > >>>>>> queue.workerThreadMinimumMessages="60000" ) { action(type="omfile" > >>>>>> file="/var/log/remote.log" ioBufferSize="64k" flushOnTXEnd="off" > >>>>>> asyncWriting="on") } > >>>>>> > >>>>>> Same issue. > >>>>>> > >>>>>> Any thoughts? > >>>>>> > >>>>>> Thank you, > >>>>>> Dave > >>>>>> > >>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com/professional-services/ > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > >>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com/professional-services/ > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
