2016-04-07 17:05 GMT+02:00 David Meiser <[email protected]>: > APC sends no delimiter at all. > > I'd like to get this working with rsyslog, but I was able to get syslog-ng > running in the interim. It seems they don't care at all about broken > formatting.
I have no idea of how they guess when the frame ends. Maybe they use "<" as indication of the next frame, but that's not reliable (try sending a message with "<" inside it, then we know if that's the case). IAW: I don't know any decent way to handle such a broken sender. sorry I have no better answer... Rainer > > Really, my goal is to put this in front of our alerting system, which can't > handle high-volume syslog, so that the alerting system can alert on > specific messages. > > On Thu, Apr 7, 2016 at 9:45 AM Rainer Gerhards <[email protected]> > wrote: > >> 2016-04-07 15:42 GMT+02:00 David Meiser <[email protected]>: >> > adding addtlframedelimiter="0" works-ish. I can send messages via >> logger, >> > but not from an APC UPS. I'll check to see if I can receive messages >> from >> > a Cisco device. >> > >> > What's strange is that Logstash - which isn't at all what I'd prefer to >> use >> > - have no issue with APC syslog messages. They choke on others, but not >> on >> > APC. >> >> Well, that's a typcial problem if you have something that does not >> comply to the standards. NUL is not a frame delimitor, so different >> projects have different ideas on how to treat anomalies. >> >> Can you trace exactly what APC sends (e.g. via wireshark). Maybe they >> don't use any frame delimiter at all... (I have seen this once in a >> long while for real broken senders). >> >> Rainer >> > >> > On Thu, Apr 7, 2016 at 9:37 AM Rainer Gerhards <[email protected] >> > >> > wrote: >> > >> >> 2016-04-07 15:00 GMT+02:00 David Meiser <[email protected]>: >> >> > The problem when sending to logger was a problem with framing (NUL >> frame >> >> > line ending). >> >> >> >> I have checked the debug log. The NUL seems to be less of a problem, >> >> the real problem is that the frame terminator LF is missing, so imtcp >> >> waits for the message to be completed. Maybe that device incorrectly >> >> uses NUL as frame terminator (an exceptionally bad choice). If so, you >> >> may get along by using addtlframedelimitor="0" in the input parameter. >> >> I admit I don't know if it works for NUL, which has assigned many >> >> special meanings. >> >> >> >> HTH >> >> Rainer >> >> >> >> >However, I'm still having a problem with my test device - an >> >> > APC UPS. >> >> > >> >> > Here's what is going on below: I logged into the device with an >> >> incorrect >> >> > password. I then logged in and used the builtin test connection >> function >> >> > of the APC to send this message: "APC: Test Syslog." When I sent the >> >> > "APC: Test Syslog" message, it looks like it comes through, but >> includes >> >> > the tail end of unauthorized access message. >> >> > >> >> > Here is what I see in the debug log: >> >> > >> >> > 3663.579327690:7f0bce158700: imptcp: epoll returned 1 events >> >> > 3663.579333890:7f0bce158700: imptcp: new activity on session socket 16 >> >> > 3663.579339390:7f0bce158700: imptcp: removing socket 16 from epoll[8] >> set >> >> > 3663.579360490:7f0bce158700: imptcp: session on socket 16 closed with >> >> iRet >> >> > 0. 3663.579363490:7f0bce158700: imptcp going on epoll_wait >> >> > 3663.904090302:7f0bce158700: imptcp: epoll returned 1 events >> >> > 3663.904104602:7f0bce158700: imptcp: new connection on listen socket 9 >> >> > 3663.904131703:7f0bce158700: imptcp: added socket 16 to epoll[8] set >> >> > 3663.904150003:7f0bce158700: imptcp going on epoll_wait >> >> > 3663.905069517:7f0bce158700: imptcp: epoll returned 1 events >> >> > 3663.905075917:7f0bce158700: imptcp: new activity on session socket 16 >> >> > 3663.905081817:7f0bce158700: imptcp: data(131072) on socket 16: >> <13>Apr 7 >> >> > 08:54:23 192.168.1.2 APC: Test Syslog.horized user attempting to >> access >> >> the >> >> > Web interface from 192.168.2.2. 0x0006 3663.905090218:7f0bce158700: >> >> imptcp >> >> > going on epoll_wait 3663.914878269:7f0bce158700: imptcp: epoll >> returned 1 >> >> > events 3663.914883469:7f0bce158700: imptcp: new activity on session >> >> socket >> >> > 16 3663.914888369:7f0bce158700: imptcp: removing socket 16 from >> epoll[8] >> >> > set 3663.914913669:7f0bce158700: imptcp: session on socket 16 closed >> with >> >> > iRet 0. 3663.914916669:7f0bce158700: imptcp going on epoll_wait >> >> > >> >> > >> >> > On Thu, Apr 7, 2016 at 8:27 AM David Meiser <[email protected]> >> wrote: >> >> > >> >> >> For a udp message, I see this in the debug log: >> >> >> https://gist.github.com/dmeiser/30ba9ef626ab3fb6fd9cf865a88a7d1e >> >> >> >> >> >> For a tcp message, I see this in debug: >> >> >> https://gist.github.com/dmeiser/24360ae40cf81573b833241c622b3b6a >> >> >> >> >> >> Here is the entire debug session: >> >> >> https://gist.github.com/dmeiser/11cf54f26834a7c09965c5bac098a727 >> >> >> >> >> >> Also, while I am using 7.4.7 in this session, I have tried the latest >> >> >> release from the official repo, as well. >> >> >> >> >> >> Thanks for the help! >> >> >> >> >> >> On Wed, Apr 6, 2016 at 8:42 PM David Lang <[email protected]> wrote: >> >> >> >> >> >>> On Thu, 7 Apr 2016, David Meiser wrote: >> >> >>> >> >> >>> > The log is coming from the debug messages by running rsyslog -d >> -n as >> >> >>> root >> >> >>> > on the generic forwarder machine. This line comes from the debug >> >> >>> output: >> >> >>> > 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15: >> >> <5>Apr 6 >> >> >>> > 15:56:34 user.name: hello world >> >> >>> > >> >> >>> > I snipped the udp configuration from the config. It is identical >> >> except >> >> >>> > for the fact it uses imudp, not imptcp. >> >> >>> > >> >> >>> > I did open port 514. I also had the thought that it could be an >> >> selinux >> >> >>> > problem so I disabled selinux. Then I moved it to 10514 and 1514 >> and >> >> >>> > opened those ports. And then I disabled the firewall altogether. >> >> Each >> >> >>> > permutation had the same result. >> >> >>> > >> >> >>> > Since I can use ncat to copy/paste a message which is received and >> >> >>> > processed, I'm pretty sure there's something non-networking >> related. >> >> >>> >> >> >>> Ok, so if you are looking at the debug log and it's seeing the >> message >> >> >>> arrive, >> >> >>> then rsyslog is getting and processing the message. What does the >> debug >> >> >>> log show >> >> >>> happens after the message is recevied? it should show you everything >> >> >>> that's done >> >> >>> to the message, and you can look at the same thing for the UDP >> message >> >> to >> >> >>> see >> >> >>> what's different about them. >> >> >>> >> >> >>> David Lang >> >> >>> >> >> >>> > On Wed, Apr 6, 2016 at 8:08 PM David Lang <[email protected]> wrote: >> >> >>> > >> >> >>> >> On Wed, 6 Apr 2016, David Meiser wrote: >> >> >>> >> >> >> >>> >>> I've tried it from 3 different locations, including a server on >> >> that >> >> >>> >> vlan. >> >> >>> >>> Here's why I think it's an rsyslog configuration issue: the >> logger >> >> >>> >> messages >> >> >>> >>> I send via TCP do show up in the debug log. Here's an example: >> >> >>> >>> >> >> >>> >>> 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15: >> >> >>> <5>Apr 6 >> >> >>> >>> 15:56:34 user.name: hello world >> >> >>> >> >> >> >>> >> I still don't understand what this is an example of. Where are >> you >> >> >>> getting >> >> >>> >> this >> >> >>> >> from? >> >> >>> >> >> >> >>> >>> That isn't processed. Change it to UDP, it goes through no >> >> problem. >> >> >>> >>> >> >> >>> >>> In regards to the APC UPS message, I pulled the message from the >> >> >>> rsyslog >> >> >>> >>> debug log. >> >> >>> >> >> >> >>> >> so if you go to server A and send a message to server B via UDP >> it >> >> >>> shows >> >> >>> >> up, but >> >> >>> >> if you send it by TCP it doesn't? >> >> >>> >> >> >> >>> >> The config you show below has no UDP configured. >> >> >>> >> >> >> >>> >> There are many distros (including RedHat) that block TCP 514 by >> >> >>> default, >> >> >>> >> but >> >> >>> >> allow UDP 514, so check your iptables rules >> >> >>> >> >> >> >>> >> David Lang >> >> >>> >> >> >> >>> >>> On Wed, Apr 6, 2016, 5:06 PM David Lang <[email protected]> wrote: >> >> >>> >>> >> >> >>> >>>> On Wed, 6 Apr 2016, David Meiser wrote: >> >> >>> >>>> >> >> >>> >>>>> As I continue to try to troubleshoot this, I tried using a >> direct >> >> >>> ncat >> >> >>> >>>>> connection. If I send "1.1.1.1 hello world" (simulating an IP >> >> >>> address >> >> >>> >>>> and >> >> >>> >>>>> message), the output to the log file is: >> >> >>> >>>> >> >> >>> >>>> output to what logfile with what config? Is this the config >> shown >> >> >>> below? >> >> >>> >>>> >> >> >>> >>>> note, workerthreads=4 is almost certinly wrong, you need a very >> >> >>> complex >> >> >>> >>>> config >> >> >>> >>>> to need even two threads. Having a queue on a write to a file >> is >> >> also >> >> >>> >>>> almost >> >> >>> >>>> always the wrong thing to do. >> >> >>> >>>> >> >> >>> >>>>> Apr 6 16:25:41 1 >> >> >>> >>>>> Apr 6 16:25:41 .1.1 hello world >> >> >>> >>>> >> >> >>> >>>> not a valid syslog message >> >> >>> >>>> >> >> >>> >>>>> If I send "1\.1\.1\.1 hello world" the output is this: >> >> >>> >>>>> Apr 6 16:26:16 . >> >> >>> >>>>> Apr 6 16:26:16 . >> >> >>> >>>>> Apr 6 16:26:16 . >> >> >>> >>>>> Apr 6 16:26:16 h >> >> >>> >>>>> Apr 6 16:26:16 ello world >> >> >>> >>>> >> >> >>> >>>> also not a valid syslog message >> >> >>> >>>> >> >> >>> >>>>> If I send "myserver.domain.com hello world" the output is >> this: >> >> >>> >>>>> Apr 6 16:28:28 myserver.domain.com hello world >> >> >>> >>>> >> >> >>> >>>> also not a valid syslog message >> >> >>> >>>> >> >> >>> >>>>> If I send a raw syslog message "<13>Apr 6 14:37:38 1.1.1.1 >> Test >> >> >>> >> Syslog." >> >> >>> >>>> I >> >> >>> >>>>> get: >> >> >>> >>>>> Apr 6 14:37:38 1.1.1.1 Test Syslog. >> >> >>> >>>> >> >> >>> >>>> This is a valid syslog message, and is the output I would >> expect >> >> if >> >> >>> you >> >> >>> >> are >> >> >>> >>>> doing a simple write with the traditional format. >> >> >>> >>>> >> >> >>> >>>>> So, I go back to the original device that sent the TCP syslog >> >> >>> message >> >> >>> >> (an >> >> >>> >>>>> APC UPS) and send another test directly to the server and I >> see >> >> >>> this in >> >> >>> >>>> the >> >> >>> >>>>> logs: >> >> >>> >>>>> 5015.675372226:7f6ce2981700: imptcp: data(131072) on socket >> 19: >> >> >>> >> <13>Apr 6 >> >> >>> >>>>> 16:36:54 1.1.1.1 APC: Test Syslog.rom o world >> >> >>> >>>> >> >> >>> >>>> how are you seeing this. >> >> >>> >>>> >> >> >>> >>>>> And in the log I get nothing. >> >> >>> >>>> >> >> >>> >>>> so if it works from some places and not others, the first >> thing to >> >> >>> do is >> >> >>> >>>> to look >> >> >>> >>>> at the network and see if there are any network things that >> would >> >> >>> block >> >> >>> >> the >> >> >>> >>>> communication from the place that doesn't work, but don't >> block it >> >> >>> from >> >> >>> >> the >> >> >>> >>>> place that does work. >> >> >>> >>>> >> >> >>> >>>> David Lang >> >> >>> >>>> >> >> >>> >>>>> On Wed, Apr 6, 2016 at 4:03 PM David Meiser < >> [email protected]> >> >> >>> wrote: >> >> >>> >>>>> >> >> >>> >>>>>> I am trying to setup a generic syslog forwarder that accepts >> >> >>> messages >> >> >>> >> on >> >> >>> >>>>>> tcp & udp. Using imptcp, I see messages come in, but no >> >> rulesets >> >> >>> are >> >> >>> >>>>>> processed. My ruleset, right now, is set to take tcp >> messages >> >> and >> >> >>> >> just >> >> >>> >>>>>> output them to file (for troubleshooting). UDP works fine. >> >> >>> >>>>>> >> >> >>> >>>>>> Here is what I see in debug mode: >> >> >>> >>>>>> 2594.333580185:7fd29ab2b700: imptcp: epoll returned 1 events >> >> >>> >>>>>> 2594.333594886:7fd29ab2b700: imptcp: new connection on listen >> >> >>> socket 9 >> >> >>> >>>>>> 2594.334657804:7fd29ab2b700: imptcp: added socket 15 to >> epoll[8] >> >> >>> set >> >> >>> >>>>>> 2594.334670004:7fd29ab2b700: imptcp going on epoll_wait >> >> >>> >>>>>> 2594.334673904:7fd29ab2b700: imptcp: epoll returned 1 events >> >> >>> >>>>>> 2594.334686204:7fd29ab2b700: imptcp: new activity on session >> >> >>> socket 15 >> >> >>> >>>>>> 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket >> 15: >> >> >>> >> <5>Apr 6 >> >> >>> >>>>>> 15:56:34 user.name: hello world 2594.334701804:7fd29ab2b700: >> >> >>> imptcp: >> >> >>> >>>>>> removing socket 15 from epoll[8] set >> >> 2594.335235814:7fd29ab2b700: >> >> >>> >>>> imptcp: >> >> >>> >>>>>> session on socket 15 closed with iRet 0. >> >> >>> 2594.335240814:7fd29ab2b700: >> >> >>> >>>>>> imptcp going on epoll_wait >> >> >>> >>>>>> >> >> >>> >>>>>> Here is my config: >> >> >>> >>>>>> module(load="imptcp") >> >> >>> >>>>>> >> >> >>> >>>>>> input( >> >> >>> >>>>>> type="imptcp" >> >> >>> >>>>>> port="514" >> >> >>> >>>>>> ruleset="remote" >> >> >>> >>>>>> ) >> >> >>> >>>>>> >> >> >>> >>>>>> ruleset( >> >> >>> >>>>>> name="remote" queue.workerthreads="4" >> >> >>> >>>>>> queue.filename="srvrfwd" >> >> >>> >>>>>> queue.type="LinkedList" >> >> >>> >>>>>> queue.syncqueuefiles="on" >> >> >>> >>>>>> queue.maxdiskspace="10g" >> >> >>> >>>>>> queue.saveonshutdown="on" >> >> >>> >>>>>> queue.size="10000000" >> >> >>> >>>>>> ) { >> >> >>> >>>>>> action( >> >> >>> >>>>>> type="omfile" >> >> >>> >>>>>> file="/var/log/debug" >> >> >>> >>>>>> ) >> >> >>> >>>>>> } >> >> >>> >>>>>> >> >> >>> >>>>>> I've also tried the sample config from the website: >> >> >>> >>>>>> input(type="imptcp" port="10514" ruleset="writeRemoteData") >> >> >>> >>>>>> ruleset(name="writeRemoteData" queue.type="fixedArray" >> >> >>> >>>> queue.size="250000" >> >> >>> >>>>>> queue.dequeueBatchSize="4096" queue.workerThreads="4" >> >> >>> >>>>>> queue.workerThreadMinimumMessages="60000" ) { >> >> action(type="omfile" >> >> >>> >>>>>> file="/var/log/remote.log" ioBufferSize="64k" >> flushOnTXEnd="off" >> >> >>> >>>>>> asyncWriting="on") } >> >> >>> >>>>>> >> >> >>> >>>>>> Same issue. >> >> >>> >>>>>> >> >> >>> >>>>>> Any thoughts? >> >> >>> >>>>>> >> >> >>> >>>>>> Thank you, >> >> >>> >>>>>> Dave >> >> >>> >>>>>> >> >> >>> >>>>> _______________________________________________ >> >> >>> >>>>> rsyslog mailing list >> >> >>> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >>> >>>>> http://www.rsyslog.com/professional-services/ >> >> >>> >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >>> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >> by a >> >> >>> >> myriad >> >> >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> POST if >> >> >>> you >> >> >>> >>>> DON'T LIKE THAT. >> >> >>> >>>>> >> >> >>> >>>> _______________________________________________ >> >> >>> >>>> rsyslog mailing list >> >> >>> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >>> >>>> http://www.rsyslog.com/professional-services/ >> >> >>> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >>> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >> by a >> >> >>> myriad >> >> >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> POST if >> >> >>> you >> >> >>> >>>> DON'T LIKE THAT. >> >> >>> >>>> >> >> >>> >>> _______________________________________________ >> >> >>> >>> rsyslog mailing list >> >> >>> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >>> >>> http://www.rsyslog.com/professional-services/ >> >> >>> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >>> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >> a >> >> >>> myriad >> >> >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >> if >> >> you >> >> >>> >> DON'T LIKE THAT. >> >> >>> >>> >> >> >>> >> _______________________________________________ >> >> >>> >> rsyslog mailing list >> >> >>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >>> >> http://www.rsyslog.com/professional-services/ >> >> >>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >> >>> myriad >> >> >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST >> if >> >> you >> >> >>> >> DON'T LIKE THAT. >> >> >>> >> >> >> >>> > _______________________________________________ >> >> >>> > rsyslog mailing list >> >> >>> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >>> > http://www.rsyslog.com/professional-services/ >> >> >>> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >> >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> POST >> >> if >> >> >>> you DON'T LIKE THAT. >> >> >>> > >> >> >>> _______________________________________________ >> >> >>> rsyslog mailing list >> >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >>> http://www.rsyslog.com/professional-services/ >> >> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >> myriad >> >> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >> >>> DON'T LIKE THAT. >> >> >>> >> >> >> >> >> > _______________________________________________ >> >> > rsyslog mailing list >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> > http://www.rsyslog.com/professional-services/ >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T LIKE THAT. >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com/professional-services/ >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T LIKE THAT. >> >> >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
