APC sends no delimiter at all. I'd like to get this working with rsyslog, but I was able to get syslog-ng running in the interim. It seems they don't care at all about broken formatting.
Really, my goal is to put this in front of our alerting system, which can't handle high-volume syslog, so that the alerting system can alert on specific messages. On Thu, Apr 7, 2016 at 9:45 AM Rainer Gerhards <[email protected]> wrote: > 2016-04-07 15:42 GMT+02:00 David Meiser <[email protected]>: > > adding addtlframedelimiter="0" works-ish. I can send messages via > logger, > > but not from an APC UPS. I'll check to see if I can receive messages > from > > a Cisco device. > > > > What's strange is that Logstash - which isn't at all what I'd prefer to > use > > - have no issue with APC syslog messages. They choke on others, but not > on > > APC. > > Well, that's a typcial problem if you have something that does not > comply to the standards. NUL is not a frame delimitor, so different > projects have different ideas on how to treat anomalies. > > Can you trace exactly what APC sends (e.g. via wireshark). Maybe they > don't use any frame delimiter at all... (I have seen this once in a > long while for real broken senders). > > Rainer > > > > On Thu, Apr 7, 2016 at 9:37 AM Rainer Gerhards <[email protected] > > > > wrote: > > > >> 2016-04-07 15:00 GMT+02:00 David Meiser <[email protected]>: > >> > The problem when sending to logger was a problem with framing (NUL > frame > >> > line ending). > >> > >> I have checked the debug log. The NUL seems to be less of a problem, > >> the real problem is that the frame terminator LF is missing, so imtcp > >> waits for the message to be completed. Maybe that device incorrectly > >> uses NUL as frame terminator (an exceptionally bad choice). If so, you > >> may get along by using addtlframedelimitor="0" in the input parameter. > >> I admit I don't know if it works for NUL, which has assigned many > >> special meanings. > >> > >> HTH > >> Rainer > >> > >> >However, I'm still having a problem with my test device - an > >> > APC UPS. > >> > > >> > Here's what is going on below: I logged into the device with an > >> incorrect > >> > password. I then logged in and used the builtin test connection > function > >> > of the APC to send this message: "APC: Test Syslog." When I sent the > >> > "APC: Test Syslog" message, it looks like it comes through, but > includes > >> > the tail end of unauthorized access message. > >> > > >> > Here is what I see in the debug log: > >> > > >> > 3663.579327690:7f0bce158700: imptcp: epoll returned 1 events > >> > 3663.579333890:7f0bce158700: imptcp: new activity on session socket 16 > >> > 3663.579339390:7f0bce158700: imptcp: removing socket 16 from epoll[8] > set > >> > 3663.579360490:7f0bce158700: imptcp: session on socket 16 closed with > >> iRet > >> > 0. 3663.579363490:7f0bce158700: imptcp going on epoll_wait > >> > 3663.904090302:7f0bce158700: imptcp: epoll returned 1 events > >> > 3663.904104602:7f0bce158700: imptcp: new connection on listen socket 9 > >> > 3663.904131703:7f0bce158700: imptcp: added socket 16 to epoll[8] set > >> > 3663.904150003:7f0bce158700: imptcp going on epoll_wait > >> > 3663.905069517:7f0bce158700: imptcp: epoll returned 1 events > >> > 3663.905075917:7f0bce158700: imptcp: new activity on session socket 16 > >> > 3663.905081817:7f0bce158700: imptcp: data(131072) on socket 16: > <13>Apr 7 > >> > 08:54:23 192.168.1.2 APC: Test Syslog.horized user attempting to > access > >> the > >> > Web interface from 192.168.2.2. 0x0006 3663.905090218:7f0bce158700: > >> imptcp > >> > going on epoll_wait 3663.914878269:7f0bce158700: imptcp: epoll > returned 1 > >> > events 3663.914883469:7f0bce158700: imptcp: new activity on session > >> socket > >> > 16 3663.914888369:7f0bce158700: imptcp: removing socket 16 from > epoll[8] > >> > set 3663.914913669:7f0bce158700: imptcp: session on socket 16 closed > with > >> > iRet 0. 3663.914916669:7f0bce158700: imptcp going on epoll_wait > >> > > >> > > >> > On Thu, Apr 7, 2016 at 8:27 AM David Meiser <[email protected]> > wrote: > >> > > >> >> For a udp message, I see this in the debug log: > >> >> https://gist.github.com/dmeiser/30ba9ef626ab3fb6fd9cf865a88a7d1e > >> >> > >> >> For a tcp message, I see this in debug: > >> >> https://gist.github.com/dmeiser/24360ae40cf81573b833241c622b3b6a > >> >> > >> >> Here is the entire debug session: > >> >> https://gist.github.com/dmeiser/11cf54f26834a7c09965c5bac098a727 > >> >> > >> >> Also, while I am using 7.4.7 in this session, I have tried the latest > >> >> release from the official repo, as well. > >> >> > >> >> Thanks for the help! > >> >> > >> >> On Wed, Apr 6, 2016 at 8:42 PM David Lang <[email protected]> wrote: > >> >> > >> >>> On Thu, 7 Apr 2016, David Meiser wrote: > >> >>> > >> >>> > The log is coming from the debug messages by running rsyslog -d > -n as > >> >>> root > >> >>> > on the generic forwarder machine. This line comes from the debug > >> >>> output: > >> >>> > 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15: > >> <5>Apr 6 > >> >>> > 15:56:34 user.name: hello world > >> >>> > > >> >>> > I snipped the udp configuration from the config. It is identical > >> except > >> >>> > for the fact it uses imudp, not imptcp. > >> >>> > > >> >>> > I did open port 514. I also had the thought that it could be an > >> selinux > >> >>> > problem so I disabled selinux. Then I moved it to 10514 and 1514 > and > >> >>> > opened those ports. And then I disabled the firewall altogether. > >> Each > >> >>> > permutation had the same result. > >> >>> > > >> >>> > Since I can use ncat to copy/paste a message which is received and > >> >>> > processed, I'm pretty sure there's something non-networking > related. > >> >>> > >> >>> Ok, so if you are looking at the debug log and it's seeing the > message > >> >>> arrive, > >> >>> then rsyslog is getting and processing the message. What does the > debug > >> >>> log show > >> >>> happens after the message is recevied? it should show you everything > >> >>> that's done > >> >>> to the message, and you can look at the same thing for the UDP > message > >> to > >> >>> see > >> >>> what's different about them. > >> >>> > >> >>> David Lang > >> >>> > >> >>> > On Wed, Apr 6, 2016 at 8:08 PM David Lang <[email protected]> wrote: > >> >>> > > >> >>> >> On Wed, 6 Apr 2016, David Meiser wrote: > >> >>> >> > >> >>> >>> I've tried it from 3 different locations, including a server on > >> that > >> >>> >> vlan. > >> >>> >>> Here's why I think it's an rsyslog configuration issue: the > logger > >> >>> >> messages > >> >>> >>> I send via TCP do show up in the debug log. Here's an example: > >> >>> >>> > >> >>> >>> 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15: > >> >>> <5>Apr 6 > >> >>> >>> 15:56:34 user.name: hello world > >> >>> >> > >> >>> >> I still don't understand what this is an example of. Where are > you > >> >>> getting > >> >>> >> this > >> >>> >> from? > >> >>> >> > >> >>> >>> That isn't processed. Change it to UDP, it goes through no > >> problem. > >> >>> >>> > >> >>> >>> In regards to the APC UPS message, I pulled the message from the > >> >>> rsyslog > >> >>> >>> debug log. > >> >>> >> > >> >>> >> so if you go to server A and send a message to server B via UDP > it > >> >>> shows > >> >>> >> up, but > >> >>> >> if you send it by TCP it doesn't? > >> >>> >> > >> >>> >> The config you show below has no UDP configured. > >> >>> >> > >> >>> >> There are many distros (including RedHat) that block TCP 514 by > >> >>> default, > >> >>> >> but > >> >>> >> allow UDP 514, so check your iptables rules > >> >>> >> > >> >>> >> David Lang > >> >>> >> > >> >>> >>> On Wed, Apr 6, 2016, 5:06 PM David Lang <[email protected]> wrote: > >> >>> >>> > >> >>> >>>> On Wed, 6 Apr 2016, David Meiser wrote: > >> >>> >>>> > >> >>> >>>>> As I continue to try to troubleshoot this, I tried using a > direct > >> >>> ncat > >> >>> >>>>> connection. If I send "1.1.1.1 hello world" (simulating an IP > >> >>> address > >> >>> >>>> and > >> >>> >>>>> message), the output to the log file is: > >> >>> >>>> > >> >>> >>>> output to what logfile with what config? Is this the config > shown > >> >>> below? > >> >>> >>>> > >> >>> >>>> note, workerthreads=4 is almost certinly wrong, you need a very > >> >>> complex > >> >>> >>>> config > >> >>> >>>> to need even two threads. Having a queue on a write to a file > is > >> also > >> >>> >>>> almost > >> >>> >>>> always the wrong thing to do. > >> >>> >>>> > >> >>> >>>>> Apr 6 16:25:41 1 > >> >>> >>>>> Apr 6 16:25:41 .1.1 hello world > >> >>> >>>> > >> >>> >>>> not a valid syslog message > >> >>> >>>> > >> >>> >>>>> If I send "1\.1\.1\.1 hello world" the output is this: > >> >>> >>>>> Apr 6 16:26:16 . > >> >>> >>>>> Apr 6 16:26:16 . > >> >>> >>>>> Apr 6 16:26:16 . > >> >>> >>>>> Apr 6 16:26:16 h > >> >>> >>>>> Apr 6 16:26:16 ello world > >> >>> >>>> > >> >>> >>>> also not a valid syslog message > >> >>> >>>> > >> >>> >>>>> If I send "myserver.domain.com hello world" the output is > this: > >> >>> >>>>> Apr 6 16:28:28 myserver.domain.com hello world > >> >>> >>>> > >> >>> >>>> also not a valid syslog message > >> >>> >>>> > >> >>> >>>>> If I send a raw syslog message "<13>Apr 6 14:37:38 1.1.1.1 > Test > >> >>> >> Syslog." > >> >>> >>>> I > >> >>> >>>>> get: > >> >>> >>>>> Apr 6 14:37:38 1.1.1.1 Test Syslog. > >> >>> >>>> > >> >>> >>>> This is a valid syslog message, and is the output I would > expect > >> if > >> >>> you > >> >>> >> are > >> >>> >>>> doing a simple write with the traditional format. > >> >>> >>>> > >> >>> >>>>> So, I go back to the original device that sent the TCP syslog > >> >>> message > >> >>> >> (an > >> >>> >>>>> APC UPS) and send another test directly to the server and I > see > >> >>> this in > >> >>> >>>> the > >> >>> >>>>> logs: > >> >>> >>>>> 5015.675372226:7f6ce2981700: imptcp: data(131072) on socket > 19: > >> >>> >> <13>Apr 6 > >> >>> >>>>> 16:36:54 1.1.1.1 APC: Test Syslog.rom o world > >> >>> >>>> > >> >>> >>>> how are you seeing this. > >> >>> >>>> > >> >>> >>>>> And in the log I get nothing. > >> >>> >>>> > >> >>> >>>> so if it works from some places and not others, the first > thing to > >> >>> do is > >> >>> >>>> to look > >> >>> >>>> at the network and see if there are any network things that > would > >> >>> block > >> >>> >> the > >> >>> >>>> communication from the place that doesn't work, but don't > block it > >> >>> from > >> >>> >> the > >> >>> >>>> place that does work. > >> >>> >>>> > >> >>> >>>> David Lang > >> >>> >>>> > >> >>> >>>>> On Wed, Apr 6, 2016 at 4:03 PM David Meiser < > [email protected]> > >> >>> wrote: > >> >>> >>>>> > >> >>> >>>>>> I am trying to setup a generic syslog forwarder that accepts > >> >>> messages > >> >>> >> on > >> >>> >>>>>> tcp & udp. Using imptcp, I see messages come in, but no > >> rulesets > >> >>> are > >> >>> >>>>>> processed. My ruleset, right now, is set to take tcp > messages > >> and > >> >>> >> just > >> >>> >>>>>> output them to file (for troubleshooting). UDP works fine. > >> >>> >>>>>> > >> >>> >>>>>> Here is what I see in debug mode: > >> >>> >>>>>> 2594.333580185:7fd29ab2b700: imptcp: epoll returned 1 events > >> >>> >>>>>> 2594.333594886:7fd29ab2b700: imptcp: new connection on listen > >> >>> socket 9 > >> >>> >>>>>> 2594.334657804:7fd29ab2b700: imptcp: added socket 15 to > epoll[8] > >> >>> set > >> >>> >>>>>> 2594.334670004:7fd29ab2b700: imptcp going on epoll_wait > >> >>> >>>>>> 2594.334673904:7fd29ab2b700: imptcp: epoll returned 1 events > >> >>> >>>>>> 2594.334686204:7fd29ab2b700: imptcp: new activity on session > >> >>> socket 15 > >> >>> >>>>>> 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket > 15: > >> >>> >> <5>Apr 6 > >> >>> >>>>>> 15:56:34 user.name: hello world 2594.334701804:7fd29ab2b700: > >> >>> imptcp: > >> >>> >>>>>> removing socket 15 from epoll[8] set > >> 2594.335235814:7fd29ab2b700: > >> >>> >>>> imptcp: > >> >>> >>>>>> session on socket 15 closed with iRet 0. > >> >>> 2594.335240814:7fd29ab2b700: > >> >>> >>>>>> imptcp going on epoll_wait > >> >>> >>>>>> > >> >>> >>>>>> Here is my config: > >> >>> >>>>>> module(load="imptcp") > >> >>> >>>>>> > >> >>> >>>>>> input( > >> >>> >>>>>> type="imptcp" > >> >>> >>>>>> port="514" > >> >>> >>>>>> ruleset="remote" > >> >>> >>>>>> ) > >> >>> >>>>>> > >> >>> >>>>>> ruleset( > >> >>> >>>>>> name="remote" queue.workerthreads="4" > >> >>> >>>>>> queue.filename="srvrfwd" > >> >>> >>>>>> queue.type="LinkedList" > >> >>> >>>>>> queue.syncqueuefiles="on" > >> >>> >>>>>> queue.maxdiskspace="10g" > >> >>> >>>>>> queue.saveonshutdown="on" > >> >>> >>>>>> queue.size="10000000" > >> >>> >>>>>> ) { > >> >>> >>>>>> action( > >> >>> >>>>>> type="omfile" > >> >>> >>>>>> file="/var/log/debug" > >> >>> >>>>>> ) > >> >>> >>>>>> } > >> >>> >>>>>> > >> >>> >>>>>> I've also tried the sample config from the website: > >> >>> >>>>>> input(type="imptcp" port="10514" ruleset="writeRemoteData") > >> >>> >>>>>> ruleset(name="writeRemoteData" queue.type="fixedArray" > >> >>> >>>> queue.size="250000" > >> >>> >>>>>> queue.dequeueBatchSize="4096" queue.workerThreads="4" > >> >>> >>>>>> queue.workerThreadMinimumMessages="60000" ) { > >> action(type="omfile" > >> >>> >>>>>> file="/var/log/remote.log" ioBufferSize="64k" > flushOnTXEnd="off" > >> >>> >>>>>> asyncWriting="on") } > >> >>> >>>>>> > >> >>> >>>>>> Same issue. > >> >>> >>>>>> > >> >>> >>>>>> Any thoughts? > >> >>> >>>>>> > >> >>> >>>>>> Thank you, > >> >>> >>>>>> Dave > >> >>> >>>>>> > >> >>> >>>>> _______________________________________________ > >> >>> >>>>> rsyslog mailing list > >> >>> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >>> >>>>> http://www.rsyslog.com/professional-services/ > >> >>> >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >>> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > by a > >> >>> >> myriad > >> >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST if > >> >>> you > >> >>> >>>> DON'T LIKE THAT. > >> >>> >>>>> > >> >>> >>>> _______________________________________________ > >> >>> >>>> rsyslog mailing list > >> >>> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >>> >>>> http://www.rsyslog.com/professional-services/ > >> >>> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >>> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > by a > >> >>> myriad > >> >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST if > >> >>> you > >> >>> >>>> DON'T LIKE THAT. > >> >>> >>>> > >> >>> >>> _______________________________________________ > >> >>> >>> rsyslog mailing list > >> >>> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >>> >>> http://www.rsyslog.com/professional-services/ > >> >>> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >>> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a > >> >>> myriad > >> >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > >> you > >> >>> >> DON'T LIKE THAT. > >> >>> >>> > >> >>> >> _______________________________________________ > >> >>> >> rsyslog mailing list > >> >>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >>> >> http://www.rsyslog.com/professional-services/ > >> >>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> >>> myriad > >> >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > >> you > >> >>> >> DON'T LIKE THAT. > >> >>> >> > >> >>> > _______________________________________________ > >> >>> > rsyslog mailing list > >> >>> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >>> > http://www.rsyslog.com/professional-services/ > >> >>> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > >> if > >> >>> you DON'T LIKE THAT. > >> >>> > > >> >>> _______________________________________________ > >> >>> rsyslog mailing list > >> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >>> http://www.rsyslog.com/professional-services/ > >> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad > >> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >> >>> DON'T LIKE THAT. > >> >>> > >> >> > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
