2016-04-07 15:42 GMT+02:00 David Meiser <[email protected]>: > adding addtlframedelimiter="0" works-ish. I can send messages via logger, > but not from an APC UPS. I'll check to see if I can receive messages from > a Cisco device. > > What's strange is that Logstash - which isn't at all what I'd prefer to use > - have no issue with APC syslog messages. They choke on others, but not on > APC.
Well, that's a typcial problem if you have something that does not comply to the standards. NUL is not a frame delimitor, so different projects have different ideas on how to treat anomalies. Can you trace exactly what APC sends (e.g. via wireshark). Maybe they don't use any frame delimiter at all... (I have seen this once in a long while for real broken senders). Rainer > > On Thu, Apr 7, 2016 at 9:37 AM Rainer Gerhards <[email protected]> > wrote: > >> 2016-04-07 15:00 GMT+02:00 David Meiser <[email protected]>: >> > The problem when sending to logger was a problem with framing (NUL frame >> > line ending). >> >> I have checked the debug log. The NUL seems to be less of a problem, >> the real problem is that the frame terminator LF is missing, so imtcp >> waits for the message to be completed. Maybe that device incorrectly >> uses NUL as frame terminator (an exceptionally bad choice). If so, you >> may get along by using addtlframedelimitor="0" in the input parameter. >> I admit I don't know if it works for NUL, which has assigned many >> special meanings. >> >> HTH >> Rainer >> >> >However, I'm still having a problem with my test device - an >> > APC UPS. >> > >> > Here's what is going on below: I logged into the device with an >> incorrect >> > password. I then logged in and used the builtin test connection function >> > of the APC to send this message: "APC: Test Syslog." When I sent the >> > "APC: Test Syslog" message, it looks like it comes through, but includes >> > the tail end of unauthorized access message. >> > >> > Here is what I see in the debug log: >> > >> > 3663.579327690:7f0bce158700: imptcp: epoll returned 1 events >> > 3663.579333890:7f0bce158700: imptcp: new activity on session socket 16 >> > 3663.579339390:7f0bce158700: imptcp: removing socket 16 from epoll[8] set >> > 3663.579360490:7f0bce158700: imptcp: session on socket 16 closed with >> iRet >> > 0. 3663.579363490:7f0bce158700: imptcp going on epoll_wait >> > 3663.904090302:7f0bce158700: imptcp: epoll returned 1 events >> > 3663.904104602:7f0bce158700: imptcp: new connection on listen socket 9 >> > 3663.904131703:7f0bce158700: imptcp: added socket 16 to epoll[8] set >> > 3663.904150003:7f0bce158700: imptcp going on epoll_wait >> > 3663.905069517:7f0bce158700: imptcp: epoll returned 1 events >> > 3663.905075917:7f0bce158700: imptcp: new activity on session socket 16 >> > 3663.905081817:7f0bce158700: imptcp: data(131072) on socket 16: <13>Apr 7 >> > 08:54:23 192.168.1.2 APC: Test Syslog.horized user attempting to access >> the >> > Web interface from 192.168.2.2. 0x0006 3663.905090218:7f0bce158700: >> imptcp >> > going on epoll_wait 3663.914878269:7f0bce158700: imptcp: epoll returned 1 >> > events 3663.914883469:7f0bce158700: imptcp: new activity on session >> socket >> > 16 3663.914888369:7f0bce158700: imptcp: removing socket 16 from epoll[8] >> > set 3663.914913669:7f0bce158700: imptcp: session on socket 16 closed with >> > iRet 0. 3663.914916669:7f0bce158700: imptcp going on epoll_wait >> > >> > >> > On Thu, Apr 7, 2016 at 8:27 AM David Meiser <[email protected]> wrote: >> > >> >> For a udp message, I see this in the debug log: >> >> https://gist.github.com/dmeiser/30ba9ef626ab3fb6fd9cf865a88a7d1e >> >> >> >> For a tcp message, I see this in debug: >> >> https://gist.github.com/dmeiser/24360ae40cf81573b833241c622b3b6a >> >> >> >> Here is the entire debug session: >> >> https://gist.github.com/dmeiser/11cf54f26834a7c09965c5bac098a727 >> >> >> >> Also, while I am using 7.4.7 in this session, I have tried the latest >> >> release from the official repo, as well. >> >> >> >> Thanks for the help! >> >> >> >> On Wed, Apr 6, 2016 at 8:42 PM David Lang <[email protected]> wrote: >> >> >> >>> On Thu, 7 Apr 2016, David Meiser wrote: >> >>> >> >>> > The log is coming from the debug messages by running rsyslog -d -n as >> >>> root >> >>> > on the generic forwarder machine. This line comes from the debug >> >>> output: >> >>> > 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15: >> <5>Apr 6 >> >>> > 15:56:34 user.name: hello world >> >>> > >> >>> > I snipped the udp configuration from the config. It is identical >> except >> >>> > for the fact it uses imudp, not imptcp. >> >>> > >> >>> > I did open port 514. I also had the thought that it could be an >> selinux >> >>> > problem so I disabled selinux. Then I moved it to 10514 and 1514 and >> >>> > opened those ports. And then I disabled the firewall altogether. >> Each >> >>> > permutation had the same result. >> >>> > >> >>> > Since I can use ncat to copy/paste a message which is received and >> >>> > processed, I'm pretty sure there's something non-networking related. >> >>> >> >>> Ok, so if you are looking at the debug log and it's seeing the message >> >>> arrive, >> >>> then rsyslog is getting and processing the message. What does the debug >> >>> log show >> >>> happens after the message is recevied? it should show you everything >> >>> that's done >> >>> to the message, and you can look at the same thing for the UDP message >> to >> >>> see >> >>> what's different about them. >> >>> >> >>> David Lang >> >>> >> >>> > On Wed, Apr 6, 2016 at 8:08 PM David Lang <[email protected]> wrote: >> >>> > >> >>> >> On Wed, 6 Apr 2016, David Meiser wrote: >> >>> >> >> >>> >>> I've tried it from 3 different locations, including a server on >> that >> >>> >> vlan. >> >>> >>> Here's why I think it's an rsyslog configuration issue: the logger >> >>> >> messages >> >>> >>> I send via TCP do show up in the debug log. Here's an example: >> >>> >>> >> >>> >>> 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15: >> >>> <5>Apr 6 >> >>> >>> 15:56:34 user.name: hello world >> >>> >> >> >>> >> I still don't understand what this is an example of. Where are you >> >>> getting >> >>> >> this >> >>> >> from? >> >>> >> >> >>> >>> That isn't processed. Change it to UDP, it goes through no >> problem. >> >>> >>> >> >>> >>> In regards to the APC UPS message, I pulled the message from the >> >>> rsyslog >> >>> >>> debug log. >> >>> >> >> >>> >> so if you go to server A and send a message to server B via UDP it >> >>> shows >> >>> >> up, but >> >>> >> if you send it by TCP it doesn't? >> >>> >> >> >>> >> The config you show below has no UDP configured. >> >>> >> >> >>> >> There are many distros (including RedHat) that block TCP 514 by >> >>> default, >> >>> >> but >> >>> >> allow UDP 514, so check your iptables rules >> >>> >> >> >>> >> David Lang >> >>> >> >> >>> >>> On Wed, Apr 6, 2016, 5:06 PM David Lang <[email protected]> wrote: >> >>> >>> >> >>> >>>> On Wed, 6 Apr 2016, David Meiser wrote: >> >>> >>>> >> >>> >>>>> As I continue to try to troubleshoot this, I tried using a direct >> >>> ncat >> >>> >>>>> connection. If I send "1.1.1.1 hello world" (simulating an IP >> >>> address >> >>> >>>> and >> >>> >>>>> message), the output to the log file is: >> >>> >>>> >> >>> >>>> output to what logfile with what config? Is this the config shown >> >>> below? >> >>> >>>> >> >>> >>>> note, workerthreads=4 is almost certinly wrong, you need a very >> >>> complex >> >>> >>>> config >> >>> >>>> to need even two threads. Having a queue on a write to a file is >> also >> >>> >>>> almost >> >>> >>>> always the wrong thing to do. >> >>> >>>> >> >>> >>>>> Apr 6 16:25:41 1 >> >>> >>>>> Apr 6 16:25:41 .1.1 hello world >> >>> >>>> >> >>> >>>> not a valid syslog message >> >>> >>>> >> >>> >>>>> If I send "1\.1\.1\.1 hello world" the output is this: >> >>> >>>>> Apr 6 16:26:16 . >> >>> >>>>> Apr 6 16:26:16 . >> >>> >>>>> Apr 6 16:26:16 . >> >>> >>>>> Apr 6 16:26:16 h >> >>> >>>>> Apr 6 16:26:16 ello world >> >>> >>>> >> >>> >>>> also not a valid syslog message >> >>> >>>> >> >>> >>>>> If I send "myserver.domain.com hello world" the output is this: >> >>> >>>>> Apr 6 16:28:28 myserver.domain.com hello world >> >>> >>>> >> >>> >>>> also not a valid syslog message >> >>> >>>> >> >>> >>>>> If I send a raw syslog message "<13>Apr 6 14:37:38 1.1.1.1 Test >> >>> >> Syslog." >> >>> >>>> I >> >>> >>>>> get: >> >>> >>>>> Apr 6 14:37:38 1.1.1.1 Test Syslog. >> >>> >>>> >> >>> >>>> This is a valid syslog message, and is the output I would expect >> if >> >>> you >> >>> >> are >> >>> >>>> doing a simple write with the traditional format. >> >>> >>>> >> >>> >>>>> So, I go back to the original device that sent the TCP syslog >> >>> message >> >>> >> (an >> >>> >>>>> APC UPS) and send another test directly to the server and I see >> >>> this in >> >>> >>>> the >> >>> >>>>> logs: >> >>> >>>>> 5015.675372226:7f6ce2981700: imptcp: data(131072) on socket 19: >> >>> >> <13>Apr 6 >> >>> >>>>> 16:36:54 1.1.1.1 APC: Test Syslog.rom o world >> >>> >>>> >> >>> >>>> how are you seeing this. >> >>> >>>> >> >>> >>>>> And in the log I get nothing. >> >>> >>>> >> >>> >>>> so if it works from some places and not others, the first thing to >> >>> do is >> >>> >>>> to look >> >>> >>>> at the network and see if there are any network things that would >> >>> block >> >>> >> the >> >>> >>>> communication from the place that doesn't work, but don't block it >> >>> from >> >>> >> the >> >>> >>>> place that does work. >> >>> >>>> >> >>> >>>> David Lang >> >>> >>>> >> >>> >>>>> On Wed, Apr 6, 2016 at 4:03 PM David Meiser <[email protected]> >> >>> wrote: >> >>> >>>>> >> >>> >>>>>> I am trying to setup a generic syslog forwarder that accepts >> >>> messages >> >>> >> on >> >>> >>>>>> tcp & udp. Using imptcp, I see messages come in, but no >> rulesets >> >>> are >> >>> >>>>>> processed. My ruleset, right now, is set to take tcp messages >> and >> >>> >> just >> >>> >>>>>> output them to file (for troubleshooting). UDP works fine. >> >>> >>>>>> >> >>> >>>>>> Here is what I see in debug mode: >> >>> >>>>>> 2594.333580185:7fd29ab2b700: imptcp: epoll returned 1 events >> >>> >>>>>> 2594.333594886:7fd29ab2b700: imptcp: new connection on listen >> >>> socket 9 >> >>> >>>>>> 2594.334657804:7fd29ab2b700: imptcp: added socket 15 to epoll[8] >> >>> set >> >>> >>>>>> 2594.334670004:7fd29ab2b700: imptcp going on epoll_wait >> >>> >>>>>> 2594.334673904:7fd29ab2b700: imptcp: epoll returned 1 events >> >>> >>>>>> 2594.334686204:7fd29ab2b700: imptcp: new activity on session >> >>> socket 15 >> >>> >>>>>> 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15: >> >>> >> <5>Apr 6 >> >>> >>>>>> 15:56:34 user.name: hello world 2594.334701804:7fd29ab2b700: >> >>> imptcp: >> >>> >>>>>> removing socket 15 from epoll[8] set >> 2594.335235814:7fd29ab2b700: >> >>> >>>> imptcp: >> >>> >>>>>> session on socket 15 closed with iRet 0. >> >>> 2594.335240814:7fd29ab2b700: >> >>> >>>>>> imptcp going on epoll_wait >> >>> >>>>>> >> >>> >>>>>> Here is my config: >> >>> >>>>>> module(load="imptcp") >> >>> >>>>>> >> >>> >>>>>> input( >> >>> >>>>>> type="imptcp" >> >>> >>>>>> port="514" >> >>> >>>>>> ruleset="remote" >> >>> >>>>>> ) >> >>> >>>>>> >> >>> >>>>>> ruleset( >> >>> >>>>>> name="remote" queue.workerthreads="4" >> >>> >>>>>> queue.filename="srvrfwd" >> >>> >>>>>> queue.type="LinkedList" >> >>> >>>>>> queue.syncqueuefiles="on" >> >>> >>>>>> queue.maxdiskspace="10g" >> >>> >>>>>> queue.saveonshutdown="on" >> >>> >>>>>> queue.size="10000000" >> >>> >>>>>> ) { >> >>> >>>>>> action( >> >>> >>>>>> type="omfile" >> >>> >>>>>> file="/var/log/debug" >> >>> >>>>>> ) >> >>> >>>>>> } >> >>> >>>>>> >> >>> >>>>>> I've also tried the sample config from the website: >> >>> >>>>>> input(type="imptcp" port="10514" ruleset="writeRemoteData") >> >>> >>>>>> ruleset(name="writeRemoteData" queue.type="fixedArray" >> >>> >>>> queue.size="250000" >> >>> >>>>>> queue.dequeueBatchSize="4096" queue.workerThreads="4" >> >>> >>>>>> queue.workerThreadMinimumMessages="60000" ) { >> action(type="omfile" >> >>> >>>>>> file="/var/log/remote.log" ioBufferSize="64k" flushOnTXEnd="off" >> >>> >>>>>> asyncWriting="on") } >> >>> >>>>>> >> >>> >>>>>> Same issue. >> >>> >>>>>> >> >>> >>>>>> Any thoughts? >> >>> >>>>>> >> >>> >>>>>> Thank you, >> >>> >>>>>> Dave >> >>> >>>>>> >> >>> >>>>> _______________________________________________ >> >>> >>>>> rsyslog mailing list >> >>> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>> >>>>> http://www.rsyslog.com/professional-services/ >> >>> >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>> >> myriad >> >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> >>> you >> >>> >>>> DON'T LIKE THAT. >> >>> >>>>> >> >>> >>>> _______________________________________________ >> >>> >>>> rsyslog mailing list >> >>> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>> >>>> http://www.rsyslog.com/professional-services/ >> >>> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>> myriad >> >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> >>> you >> >>> >>>> DON'T LIKE THAT. >> >>> >>>> >> >>> >>> _______________________________________________ >> >>> >>> rsyslog mailing list >> >>> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>> >>> http://www.rsyslog.com/professional-services/ >> >>> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>> myriad >> >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >>> >> DON'T LIKE THAT. >> >>> >>> >> >>> >> _______________________________________________ >> >>> >> rsyslog mailing list >> >>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>> >> http://www.rsyslog.com/professional-services/ >> >>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>> myriad >> >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >>> >> DON'T LIKE THAT. >> >>> >> >> >>> > _______________________________________________ >> >>> > rsyslog mailing list >> >>> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>> > http://www.rsyslog.com/professional-services/ >> >>> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >> if >> >>> you DON'T LIKE THAT. >> >>> > >> >>> _______________________________________________ >> >>> rsyslog mailing list >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >>> http://www.rsyslog.com/professional-services/ >> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >>> DON'T LIKE THAT. >> >>> >> >> >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
