The problem when sending to logger was a problem with framing (NUL frame line ending). However, I'm still having a problem with my test device - an APC UPS.
Here's what is going on below: I logged into the device with an incorrect password. I then logged in and used the builtin test connection function of the APC to send this message: "APC: Test Syslog." When I sent the "APC: Test Syslog" message, it looks like it comes through, but includes the tail end of unauthorized access message. Here is what I see in the debug log: 3663.579327690:7f0bce158700: imptcp: epoll returned 1 events 3663.579333890:7f0bce158700: imptcp: new activity on session socket 16 3663.579339390:7f0bce158700: imptcp: removing socket 16 from epoll[8] set 3663.579360490:7f0bce158700: imptcp: session on socket 16 closed with iRet 0. 3663.579363490:7f0bce158700: imptcp going on epoll_wait 3663.904090302:7f0bce158700: imptcp: epoll returned 1 events 3663.904104602:7f0bce158700: imptcp: new connection on listen socket 9 3663.904131703:7f0bce158700: imptcp: added socket 16 to epoll[8] set 3663.904150003:7f0bce158700: imptcp going on epoll_wait 3663.905069517:7f0bce158700: imptcp: epoll returned 1 events 3663.905075917:7f0bce158700: imptcp: new activity on session socket 16 3663.905081817:7f0bce158700: imptcp: data(131072) on socket 16: <13>Apr 7 08:54:23 192.168.1.2 APC: Test Syslog.horized user attempting to access the Web interface from 192.168.2.2. 0x0006 3663.905090218:7f0bce158700: imptcp going on epoll_wait 3663.914878269:7f0bce158700: imptcp: epoll returned 1 events 3663.914883469:7f0bce158700: imptcp: new activity on session socket 16 3663.914888369:7f0bce158700: imptcp: removing socket 16 from epoll[8] set 3663.914913669:7f0bce158700: imptcp: session on socket 16 closed with iRet 0. 3663.914916669:7f0bce158700: imptcp going on epoll_wait On Thu, Apr 7, 2016 at 8:27 AM David Meiser <[email protected]> wrote: > For a udp message, I see this in the debug log: > https://gist.github.com/dmeiser/30ba9ef626ab3fb6fd9cf865a88a7d1e > > For a tcp message, I see this in debug: > https://gist.github.com/dmeiser/24360ae40cf81573b833241c622b3b6a > > Here is the entire debug session: > https://gist.github.com/dmeiser/11cf54f26834a7c09965c5bac098a727 > > Also, while I am using 7.4.7 in this session, I have tried the latest > release from the official repo, as well. > > Thanks for the help! > > On Wed, Apr 6, 2016 at 8:42 PM David Lang <[email protected]> wrote: > >> On Thu, 7 Apr 2016, David Meiser wrote: >> >> > The log is coming from the debug messages by running rsyslog -d -n as >> root >> > on the generic forwarder machine. This line comes from the debug >> output: >> > 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15: <5>Apr 6 >> > 15:56:34 user.name: hello world >> > >> > I snipped the udp configuration from the config. It is identical except >> > for the fact it uses imudp, not imptcp. >> > >> > I did open port 514. I also had the thought that it could be an selinux >> > problem so I disabled selinux. Then I moved it to 10514 and 1514 and >> > opened those ports. And then I disabled the firewall altogether. Each >> > permutation had the same result. >> > >> > Since I can use ncat to copy/paste a message which is received and >> > processed, I'm pretty sure there's something non-networking related. >> >> Ok, so if you are looking at the debug log and it's seeing the message >> arrive, >> then rsyslog is getting and processing the message. What does the debug >> log show >> happens after the message is recevied? it should show you everything >> that's done >> to the message, and you can look at the same thing for the UDP message to >> see >> what's different about them. >> >> David Lang >> >> > On Wed, Apr 6, 2016 at 8:08 PM David Lang <[email protected]> wrote: >> > >> >> On Wed, 6 Apr 2016, David Meiser wrote: >> >> >> >>> I've tried it from 3 different locations, including a server on that >> >> vlan. >> >>> Here's why I think it's an rsyslog configuration issue: the logger >> >> messages >> >>> I send via TCP do show up in the debug log. Here's an example: >> >>> >> >>> 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15: >> <5>Apr 6 >> >>> 15:56:34 user.name: hello world >> >> >> >> I still don't understand what this is an example of. Where are you >> getting >> >> this >> >> from? >> >> >> >>> That isn't processed. Change it to UDP, it goes through no problem. >> >>> >> >>> In regards to the APC UPS message, I pulled the message from the >> rsyslog >> >>> debug log. >> >> >> >> so if you go to server A and send a message to server B via UDP it >> shows >> >> up, but >> >> if you send it by TCP it doesn't? >> >> >> >> The config you show below has no UDP configured. >> >> >> >> There are many distros (including RedHat) that block TCP 514 by >> default, >> >> but >> >> allow UDP 514, so check your iptables rules >> >> >> >> David Lang >> >> >> >>> On Wed, Apr 6, 2016, 5:06 PM David Lang <[email protected]> wrote: >> >>> >> >>>> On Wed, 6 Apr 2016, David Meiser wrote: >> >>>> >> >>>>> As I continue to try to troubleshoot this, I tried using a direct >> ncat >> >>>>> connection. If I send "1.1.1.1 hello world" (simulating an IP >> address >> >>>> and >> >>>>> message), the output to the log file is: >> >>>> >> >>>> output to what logfile with what config? Is this the config shown >> below? >> >>>> >> >>>> note, workerthreads=4 is almost certinly wrong, you need a very >> complex >> >>>> config >> >>>> to need even two threads. Having a queue on a write to a file is also >> >>>> almost >> >>>> always the wrong thing to do. >> >>>> >> >>>>> Apr 6 16:25:41 1 >> >>>>> Apr 6 16:25:41 .1.1 hello world >> >>>> >> >>>> not a valid syslog message >> >>>> >> >>>>> If I send "1\.1\.1\.1 hello world" the output is this: >> >>>>> Apr 6 16:26:16 . >> >>>>> Apr 6 16:26:16 . >> >>>>> Apr 6 16:26:16 . >> >>>>> Apr 6 16:26:16 h >> >>>>> Apr 6 16:26:16 ello world >> >>>> >> >>>> also not a valid syslog message >> >>>> >> >>>>> If I send "myserver.domain.com hello world" the output is this: >> >>>>> Apr 6 16:28:28 myserver.domain.com hello world >> >>>> >> >>>> also not a valid syslog message >> >>>> >> >>>>> If I send a raw syslog message "<13>Apr 6 14:37:38 1.1.1.1 Test >> >> Syslog." >> >>>> I >> >>>>> get: >> >>>>> Apr 6 14:37:38 1.1.1.1 Test Syslog. >> >>>> >> >>>> This is a valid syslog message, and is the output I would expect if >> you >> >> are >> >>>> doing a simple write with the traditional format. >> >>>> >> >>>>> So, I go back to the original device that sent the TCP syslog >> message >> >> (an >> >>>>> APC UPS) and send another test directly to the server and I see >> this in >> >>>> the >> >>>>> logs: >> >>>>> 5015.675372226:7f6ce2981700: imptcp: data(131072) on socket 19: >> >> <13>Apr 6 >> >>>>> 16:36:54 1.1.1.1 APC: Test Syslog.rom o world >> >>>> >> >>>> how are you seeing this. >> >>>> >> >>>>> And in the log I get nothing. >> >>>> >> >>>> so if it works from some places and not others, the first thing to >> do is >> >>>> to look >> >>>> at the network and see if there are any network things that would >> block >> >> the >> >>>> communication from the place that doesn't work, but don't block it >> from >> >> the >> >>>> place that does work. >> >>>> >> >>>> David Lang >> >>>> >> >>>>> On Wed, Apr 6, 2016 at 4:03 PM David Meiser <[email protected]> >> wrote: >> >>>>> >> >>>>>> I am trying to setup a generic syslog forwarder that accepts >> messages >> >> on >> >>>>>> tcp & udp. Using imptcp, I see messages come in, but no rulesets >> are >> >>>>>> processed. My ruleset, right now, is set to take tcp messages and >> >> just >> >>>>>> output them to file (for troubleshooting). UDP works fine. >> >>>>>> >> >>>>>> Here is what I see in debug mode: >> >>>>>> 2594.333580185:7fd29ab2b700: imptcp: epoll returned 1 events >> >>>>>> 2594.333594886:7fd29ab2b700: imptcp: new connection on listen >> socket 9 >> >>>>>> 2594.334657804:7fd29ab2b700: imptcp: added socket 15 to epoll[8] >> set >> >>>>>> 2594.334670004:7fd29ab2b700: imptcp going on epoll_wait >> >>>>>> 2594.334673904:7fd29ab2b700: imptcp: epoll returned 1 events >> >>>>>> 2594.334686204:7fd29ab2b700: imptcp: new activity on session >> socket 15 >> >>>>>> 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15: >> >> <5>Apr 6 >> >>>>>> 15:56:34 user.name: hello world 2594.334701804:7fd29ab2b700: >> imptcp: >> >>>>>> removing socket 15 from epoll[8] set 2594.335235814:7fd29ab2b700: >> >>>> imptcp: >> >>>>>> session on socket 15 closed with iRet 0. >> 2594.335240814:7fd29ab2b700: >> >>>>>> imptcp going on epoll_wait >> >>>>>> >> >>>>>> Here is my config: >> >>>>>> module(load="imptcp") >> >>>>>> >> >>>>>> input( >> >>>>>> type="imptcp" >> >>>>>> port="514" >> >>>>>> ruleset="remote" >> >>>>>> ) >> >>>>>> >> >>>>>> ruleset( >> >>>>>> name="remote" queue.workerthreads="4" >> >>>>>> queue.filename="srvrfwd" >> >>>>>> queue.type="LinkedList" >> >>>>>> queue.syncqueuefiles="on" >> >>>>>> queue.maxdiskspace="10g" >> >>>>>> queue.saveonshutdown="on" >> >>>>>> queue.size="10000000" >> >>>>>> ) { >> >>>>>> action( >> >>>>>> type="omfile" >> >>>>>> file="/var/log/debug" >> >>>>>> ) >> >>>>>> } >> >>>>>> >> >>>>>> I've also tried the sample config from the website: >> >>>>>> input(type="imptcp" port="10514" ruleset="writeRemoteData") >> >>>>>> ruleset(name="writeRemoteData" queue.type="fixedArray" >> >>>> queue.size="250000" >> >>>>>> queue.dequeueBatchSize="4096" queue.workerThreads="4" >> >>>>>> queue.workerThreadMinimumMessages="60000" ) { action(type="omfile" >> >>>>>> file="/var/log/remote.log" ioBufferSize="64k" flushOnTXEnd="off" >> >>>>>> asyncWriting="on") } >> >>>>>> >> >>>>>> Same issue. >> >>>>>> >> >>>>>> Any thoughts? >> >>>>>> >> >>>>>> Thank you, >> >>>>>> Dave >> >>>>>> >> >>>>> _______________________________________________ >> >>>>> rsyslog mailing list >> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>>> http://www.rsyslog.com/professional-services/ >> >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >> myriad >> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >>>> DON'T LIKE THAT. >> >>>>> >> >>>> _______________________________________________ >> >>>> rsyslog mailing list >> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>>> http://www.rsyslog.com/professional-services/ >> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >>>> DON'T LIKE THAT. >> >>>> >> >>> _______________________________________________ >> >>> rsyslog mailing list >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>> http://www.rsyslog.com/professional-services/ >> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T LIKE THAT. >> >>> >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com/professional-services/ >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T LIKE THAT. >> >> >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you DON'T LIKE THAT. >> > >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
