2016-04-07 17:16 GMT+02:00 David Lang <[email protected]>:
> On Thu, 7 Apr 2016, David Meiser wrote:
>
>> APC sends no delimiter at all.
>>
>> I'd like to get this working with rsyslog, but I was able to get syslog-ng
>> running in the interim.  It seems they don't care at all about broken
>> formatting.
>
>
> oh, they care, they just care about different parts and break in different
> ways on malformed messages :-)

nit-picking here: it's not just a malformed *message* (protocol
payload), it's actually wrong/missing framing. That's the same thing
like you have a text file and remove all LF from it. So they actually
break the transport protocol, not just the payload. It's far more
severe.

Rainer
>
>> Really, my goal is to put this in front of our alerting system, which
>> can't
>> handle high-volume syslog, so that the alerting system can alert on
>> specific messages.
>
>
> I'd suggest just using UDP from the APC to rsyslog. With UDP the packet
> boundry is the delimiter. If you have a rsyslog server reasonably close
> (network wise) to your logging source, UDP logging can be very reliable and
> then rsyslog can queue/filter the messages before going to your alerting
> system.
>
> David Lang
>
>
>> On Thu, Apr 7, 2016 at 9:45 AM Rainer Gerhards <[email protected]>
>> wrote:
>>
>>> 2016-04-07 15:42 GMT+02:00 David Meiser <[email protected]>:
>>>>
>>>> adding addtlframedelimiter="0" works-ish.  I can send messages via
>>>
>>> logger,
>>>>
>>>> but not from an APC UPS.  I'll check to see if I can receive messages
>>>
>>> from
>>>>
>>>> a Cisco device.
>>>>
>>>> What's strange is that Logstash - which isn't at all what I'd prefer to
>>>
>>> use
>>>>
>>>> - have no issue with APC syslog messages.  They choke on others, but not
>>>
>>> on
>>>>
>>>> APC.
>>>
>>>
>>> Well, that's a typcial problem if you have something that does not
>>> comply to the standards. NUL is not a frame delimitor, so different
>>> projects have different ideas on how to treat anomalies.
>>>
>>> Can you trace exactly what APC sends (e.g. via wireshark). Maybe they
>>> don't use any frame delimiter at all... (I have seen this once in a
>>> long while for real broken senders).
>>>
>>> Rainer
>>>>
>>>>
>>>> On Thu, Apr 7, 2016 at 9:37 AM Rainer Gerhards <[email protected]
>>>>
>>>> wrote:
>>>>
>>>>> 2016-04-07 15:00 GMT+02:00 David Meiser <[email protected]>:
>>>>>>
>>>>>> The problem when sending to logger was a problem with framing (NUL
>>>
>>> frame
>>>>>>
>>>>>> line ending).
>>>>>
>>>>>
>>>>> I have checked the debug log. The NUL seems to be less of a problem,
>>>>> the real problem is that the frame terminator LF is missing, so imtcp
>>>>> waits for the message to be completed. Maybe that device incorrectly
>>>>> uses NUL as frame terminator (an exceptionally bad choice). If so, you
>>>>> may get along by using addtlframedelimitor="0" in the input parameter.
>>>>> I admit I don't know if it works for NUL, which has assigned many
>>>>> special meanings.
>>>>>
>>>>> HTH
>>>>> Rainer
>>>>>
>>>>>> However, I'm still having a problem with my test device - an
>>>>>> APC UPS.
>>>>>>
>>>>>> Here's what is going on below:  I logged into the device with an
>>>>>
>>>>> incorrect
>>>>>>
>>>>>> password.  I then logged in and used the builtin test connection
>>>
>>> function
>>>>>>
>>>>>> of the APC to send this message:  "APC: Test Syslog."  When I sent the
>>>>>> "APC: Test Syslog" message, it looks like it comes through, but
>>>
>>> includes
>>>>>>
>>>>>> the tail end of unauthorized access message.
>>>>>>
>>>>>> Here is what I see in the debug log:
>>>>>>
>>>>>> 3663.579327690:7f0bce158700: imptcp: epoll returned 1 events
>>>>>> 3663.579333890:7f0bce158700: imptcp: new activity on session socket 16
>>>>>> 3663.579339390:7f0bce158700: imptcp: removing socket 16 from epoll[8]
>>>
>>> set
>>>>>>
>>>>>> 3663.579360490:7f0bce158700: imptcp: session on socket 16 closed with
>>>>>
>>>>> iRet
>>>>>>
>>>>>> 0. 3663.579363490:7f0bce158700: imptcp going on epoll_wait
>>>>>> 3663.904090302:7f0bce158700: imptcp: epoll returned 1 events
>>>>>> 3663.904104602:7f0bce158700: imptcp: new connection on listen socket 9
>>>>>> 3663.904131703:7f0bce158700: imptcp: added socket 16 to epoll[8] set
>>>>>> 3663.904150003:7f0bce158700: imptcp going on epoll_wait
>>>>>> 3663.905069517:7f0bce158700: imptcp: epoll returned 1 events
>>>>>> 3663.905075917:7f0bce158700: imptcp: new activity on session socket 16
>>>>>> 3663.905081817:7f0bce158700: imptcp: data(131072) on socket 16:
>>>
>>> <13>Apr 7
>>>>>>
>>>>>> 08:54:23 192.168.1.2 APC: Test Syslog.horized user attempting to
>>>
>>> access
>>>>>
>>>>> the
>>>>>>
>>>>>> Web interface from 192.168.2.2. 0x0006 3663.905090218:7f0bce158700:
>>>>>
>>>>> imptcp
>>>>>>
>>>>>> going on epoll_wait 3663.914878269:7f0bce158700: imptcp: epoll
>>>
>>> returned 1
>>>>>>
>>>>>> events 3663.914883469:7f0bce158700: imptcp: new activity on session
>>>>>
>>>>> socket
>>>>>>
>>>>>> 16 3663.914888369:7f0bce158700: imptcp: removing socket 16 from
>>>
>>> epoll[8]
>>>>>>
>>>>>> set 3663.914913669:7f0bce158700: imptcp: session on socket 16 closed
>>>
>>> with
>>>>>>
>>>>>> iRet 0. 3663.914916669:7f0bce158700: imptcp going on epoll_wait
>>>>>>
>>>>>>
>>>>>> On Thu, Apr 7, 2016 at 8:27 AM David Meiser <[email protected]>
>>>
>>> wrote:
>>>>>>
>>>>>>
>>>>>>> For a udp message, I see this in the debug log:
>>>>>>> https://gist.github.com/dmeiser/30ba9ef626ab3fb6fd9cf865a88a7d1e
>>>>>>>
>>>>>>> For a tcp message, I see this in debug:
>>>>>>> https://gist.github.com/dmeiser/24360ae40cf81573b833241c622b3b6a
>>>>>>>
>>>>>>> Here is the entire debug session:
>>>>>>> https://gist.github.com/dmeiser/11cf54f26834a7c09965c5bac098a727
>>>>>>>
>>>>>>> Also, while I am using 7.4.7 in this session, I have tried the latest
>>>>>>> release from the official repo, as well.
>>>>>>>
>>>>>>> Thanks for the help!
>>>>>>>
>>>>>>> On Wed, Apr 6, 2016 at 8:42 PM David Lang <[email protected]> wrote:
>>>>>>>
>>>>>>>> On Thu, 7 Apr 2016, David Meiser wrote:
>>>>>>>>
>>>>>>>>> The log is coming from the debug messages by running rsyslog -d
>>>
>>> -n as
>>>>>>>>
>>>>>>>> root
>>>>>>>>>
>>>>>>>>> on the generic forwarder machine.  This line comes from the debug
>>>>>>>>
>>>>>>>> output:
>>>>>>>>>
>>>>>>>>> 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15:
>>>>>
>>>>> <5>Apr 6
>>>>>>>>>
>>>>>>>>> 15:56:34 user.name: hello world
>>>>>>>>>
>>>>>>>>> I snipped the udp configuration from the config.  It is identical
>>>>>
>>>>> except
>>>>>>>>>
>>>>>>>>> for the fact it uses imudp, not imptcp.
>>>>>>>>>
>>>>>>>>> I did open port 514.  I also had the thought that it could be an
>>>>>
>>>>> selinux
>>>>>>>>>
>>>>>>>>> problem so I disabled selinux.  Then I moved it to 10514 and 1514
>>>
>>> and
>>>>>>>>>
>>>>>>>>> opened those ports.  And then I disabled the firewall altogether.
>>>>>
>>>>> Each
>>>>>>>>>
>>>>>>>>> permutation had the same result.
>>>>>>>>>
>>>>>>>>> Since I can use ncat to copy/paste a message which is received and
>>>>>>>>> processed, I'm pretty sure there's something non-networking
>>>
>>> related.
>>>>>>>>
>>>>>>>>
>>>>>>>> Ok, so if you are looking at the debug log and it's seeing the
>>>
>>> message
>>>>>>>>
>>>>>>>> arrive,
>>>>>>>> then rsyslog is getting and processing the message. What does the
>>>
>>> debug
>>>>>>>>
>>>>>>>> log show
>>>>>>>> happens after the message is recevied? it should show you everything
>>>>>>>> that's done
>>>>>>>> to the message, and you can look at the same thing for the UDP
>>>
>>> message
>>>>>
>>>>> to
>>>>>>>>
>>>>>>>> see
>>>>>>>> what's different about them.
>>>>>>>>
>>>>>>>> David Lang
>>>>>>>>
>>>>>>>>> On Wed, Apr 6, 2016 at 8:08 PM David Lang <[email protected]> wrote:
>>>>>>>>>
>>>>>>>>>> On Wed, 6 Apr 2016, David Meiser wrote:
>>>>>>>>>>
>>>>>>>>>>> I've tried it from 3 different locations, including a server on
>>>>>
>>>>> that
>>>>>>>>>>
>>>>>>>>>> vlan.
>>>>>>>>>>>
>>>>>>>>>>> Here's why I think it's an rsyslog configuration issue: the
>>>
>>> logger
>>>>>>>>>>
>>>>>>>>>> messages
>>>>>>>>>>>
>>>>>>>>>>> I send via TCP do show up in the debug log.  Here's an example:
>>>>>>>>>>>
>>>>>>>>>>> 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15:
>>>>>>>>
>>>>>>>> <5>Apr 6
>>>>>>>>>>>
>>>>>>>>>>> 15:56:34 user.name: hello world
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I still don't understand what this is an example of. Where are
>>>
>>> you
>>>>>>>>
>>>>>>>> getting
>>>>>>>>>>
>>>>>>>>>> this
>>>>>>>>>> from?
>>>>>>>>>>
>>>>>>>>>>> That isn't processed.  Change it to UDP, it goes through no
>>>>>
>>>>> problem.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> In regards to the APC UPS message, I pulled the message from the
>>>>>>>>
>>>>>>>> rsyslog
>>>>>>>>>>>
>>>>>>>>>>> debug log.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> so if you go to server A and send a message to server B via UDP
>>>
>>> it
>>>>>>>>
>>>>>>>> shows
>>>>>>>>>>
>>>>>>>>>> up, but
>>>>>>>>>> if you send it by TCP it doesn't?
>>>>>>>>>>
>>>>>>>>>> The config you show below has no UDP configured.
>>>>>>>>>>
>>>>>>>>>> There are many distros (including RedHat) that block TCP 514 by
>>>>>>>>
>>>>>>>> default,
>>>>>>>>>>
>>>>>>>>>> but
>>>>>>>>>> allow UDP 514, so check your iptables rules
>>>>>>>>>>
>>>>>>>>>> David Lang
>>>>>>>>>>
>>>>>>>>>>> On Wed, Apr 6, 2016, 5:06 PM David Lang <[email protected]> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> On Wed, 6 Apr 2016, David Meiser wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> As I continue to try to troubleshoot this, I tried using a
>>>
>>> direct
>>>>>>>>
>>>>>>>> ncat
>>>>>>>>>>>>>
>>>>>>>>>>>>> connection.  If I send "1.1.1.1 hello world" (simulating an IP
>>>>>>>>
>>>>>>>> address
>>>>>>>>>>>>
>>>>>>>>>>>> and
>>>>>>>>>>>>>
>>>>>>>>>>>>> message), the output to the log file is:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> output to what logfile with what config? Is this the config
>>>
>>> shown
>>>>>>>>
>>>>>>>> below?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> note, workerthreads=4 is almost certinly wrong, you need a very
>>>>>>>>
>>>>>>>> complex
>>>>>>>>>>>>
>>>>>>>>>>>> config
>>>>>>>>>>>> to need even two threads. Having a queue on a write to a file
>>>
>>> is
>>>>>
>>>>> also
>>>>>>>>>>>>
>>>>>>>>>>>> almost
>>>>>>>>>>>> always the wrong thing to do.
>>>>>>>>>>>>
>>>>>>>>>>>>> Apr 6 16:25:41 1
>>>>>>>>>>>>> Apr 6 16:25:41 .1.1 hello world
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> not a valid syslog message
>>>>>>>>>>>>
>>>>>>>>>>>>> If I send "1\.1\.1\.1 hello world" the output is this:
>>>>>>>>>>>>> Apr 6 16:26:16 .
>>>>>>>>>>>>> Apr 6 16:26:16 .
>>>>>>>>>>>>> Apr 6 16:26:16 .
>>>>>>>>>>>>> Apr 6 16:26:16 h
>>>>>>>>>>>>> Apr 6 16:26:16 ello world
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> also not a valid syslog message
>>>>>>>>>>>>
>>>>>>>>>>>>> If I send "myserver.domain.com hello world" the output is
>>>
>>> this:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Apr 6 16:28:28 myserver.domain.com hello world
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> also not a valid syslog message
>>>>>>>>>>>>
>>>>>>>>>>>>> If I send a raw syslog message "<13>Apr 6 14:37:38 1.1.1.1
>>>
>>> Test
>>>>>>>>>>
>>>>>>>>>> Syslog."
>>>>>>>>>>>>
>>>>>>>>>>>> I
>>>>>>>>>>>>>
>>>>>>>>>>>>> get:
>>>>>>>>>>>>> Apr 6 14:37:38 1.1.1.1 Test Syslog.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This is a valid syslog message, and is the output I would
>>>
>>> expect
>>>>>
>>>>> if
>>>>>>>>
>>>>>>>> you
>>>>>>>>>>
>>>>>>>>>> are
>>>>>>>>>>>>
>>>>>>>>>>>> doing a simple write with the traditional format.
>>>>>>>>>>>>
>>>>>>>>>>>>> So, I go back to the original device that sent the TCP syslog
>>>>>>>>
>>>>>>>> message
>>>>>>>>>>
>>>>>>>>>> (an
>>>>>>>>>>>>>
>>>>>>>>>>>>> APC UPS) and send another test directly to the server and I
>>>
>>> see
>>>>>>>>
>>>>>>>> this in
>>>>>>>>>>>>
>>>>>>>>>>>> the
>>>>>>>>>>>>>
>>>>>>>>>>>>> logs:
>>>>>>>>>>>>> 5015.675372226:7f6ce2981700: imptcp: data(131072) on socket
>>>
>>> 19:
>>>>>>>>>>
>>>>>>>>>> <13>Apr 6
>>>>>>>>>>>>>
>>>>>>>>>>>>> 16:36:54 1.1.1.1 APC: Test Syslog.rom o world
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> how are you seeing this.
>>>>>>>>>>>>
>>>>>>>>>>>>> And in the log I get nothing.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> so if it works from some places and not others, the first
>>>
>>> thing to
>>>>>>>>
>>>>>>>> do is
>>>>>>>>>>>>
>>>>>>>>>>>> to look
>>>>>>>>>>>> at the network and see if there are any network things that
>>>
>>> would
>>>>>>>>
>>>>>>>> block
>>>>>>>>>>
>>>>>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>> communication from the place that doesn't work, but don't
>>>
>>> block it
>>>>>>>>
>>>>>>>> from
>>>>>>>>>>
>>>>>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>> place that does work.
>>>>>>>>>>>>
>>>>>>>>>>>> David Lang
>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Apr 6, 2016 at 4:03 PM David Meiser <
>>>
>>> [email protected]>
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I am trying to setup a generic syslog forwarder that accepts
>>>>>>>>
>>>>>>>> messages
>>>>>>>>>>
>>>>>>>>>> on
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> tcp & udp.  Using imptcp, I see messages come in, but no
>>>>>
>>>>> rulesets
>>>>>>>>
>>>>>>>> are
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> processed.  My ruleset, right now, is set to take tcp
>>>
>>> messages
>>>>>
>>>>> and
>>>>>>>>>>
>>>>>>>>>> just
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> output them to file (for troubleshooting).  UDP works fine.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here is what I see in debug mode:
>>>>>>>>>>>>>> 2594.333580185:7fd29ab2b700: imptcp: epoll returned 1 events
>>>>>>>>>>>>>> 2594.333594886:7fd29ab2b700: imptcp: new connection on listen
>>>>>>>>
>>>>>>>> socket 9
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2594.334657804:7fd29ab2b700: imptcp: added socket 15 to
>>>
>>> epoll[8]
>>>>>>>>
>>>>>>>> set
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2594.334670004:7fd29ab2b700: imptcp going on epoll_wait
>>>>>>>>>>>>>> 2594.334673904:7fd29ab2b700: imptcp: epoll returned 1 events
>>>>>>>>>>>>>> 2594.334686204:7fd29ab2b700: imptcp: new activity on session
>>>>>>>>
>>>>>>>> socket 15
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket
>>>
>>> 15:
>>>>>>>>>>
>>>>>>>>>> <5>Apr 6
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 15:56:34 user.name: hello world 2594.334701804:7fd29ab2b700:
>>>>>>>>
>>>>>>>> imptcp:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> removing socket 15 from epoll[8] set
>>>>>
>>>>> 2594.335235814:7fd29ab2b700:
>>>>>>>>>>>>
>>>>>>>>>>>> imptcp:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> session on socket 15 closed with iRet 0.
>>>>>>>>
>>>>>>>> 2594.335240814:7fd29ab2b700:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> imptcp going on epoll_wait
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here is my config:
>>>>>>>>>>>>>> module(load="imptcp")
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> input(
>>>>>>>>>>>>>>  type="imptcp"
>>>>>>>>>>>>>>  port="514"
>>>>>>>>>>>>>>  ruleset="remote"
>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ruleset(
>>>>>>>>>>>>>>  name="remote" queue.workerthreads="4"
>>>>>>>>>>>>>>  queue.filename="srvrfwd"
>>>>>>>>>>>>>>  queue.type="LinkedList"
>>>>>>>>>>>>>>  queue.syncqueuefiles="on"
>>>>>>>>>>>>>>  queue.maxdiskspace="10g"
>>>>>>>>>>>>>>  queue.saveonshutdown="on"
>>>>>>>>>>>>>>  queue.size="10000000"
>>>>>>>>>>>>>> ) {
>>>>>>>>>>>>>>   action(
>>>>>>>>>>>>>>     type="omfile"
>>>>>>>>>>>>>>     file="/var/log/debug"
>>>>>>>>>>>>>>   )
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've also tried the sample config from the website:
>>>>>>>>>>>>>> input(type="imptcp" port="10514" ruleset="writeRemoteData")
>>>>>>>>>>>>>> ruleset(name="writeRemoteData" queue.type="fixedArray"
>>>>>>>>>>>>
>>>>>>>>>>>> queue.size="250000"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> queue.dequeueBatchSize="4096" queue.workerThreads="4"
>>>>>>>>>>>>>> queue.workerThreadMinimumMessages="60000" ) {
>>>>>
>>>>> action(type="omfile"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> file="/var/log/remote.log" ioBufferSize="64k"
>>>
>>> flushOnTXEnd="off"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> asyncWriting="on") }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Same issue.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any thoughts?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>> Dave
>>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>
>>> by a
>>>>>>>>>>
>>>>>>>>>> myriad
>>>>>>>>>>>>
>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>
>>> POST if
>>>>>>>>
>>>>>>>> you
>>>>>>>>>>>>
>>>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>
>>> by a
>>>>>>>>
>>>>>>>> myriad
>>>>>>>>>>>>
>>>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>
>>> POST if
>>>>>>>>
>>>>>>>> you
>>>>>>>>>>>>
>>>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>>>
>>> a
>>>>>>>>
>>>>>>>> myriad
>>>>>>>>>>
>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>
>>> if
>>>>>
>>>>> you
>>>>>>>>>>
>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> rsyslog mailing list
>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>>>>
>>>>>>>> myriad
>>>>>>>>>>
>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>>>
>>> if
>>>>>
>>>>> you
>>>>>>>>>>
>>>>>>>>>> DON'T LIKE THAT.
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> rsyslog mailing list
>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>>>>
>>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>
>>> POST
>>>>>
>>>>> if
>>>>>>>>
>>>>>>>> you DON'T LIKE THAT.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> rsyslog mailing list
>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>
>>>>> myriad
>>>>>>>>
>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>>>
>>> you
>>>>>>>>
>>>>>>>> DON'T LIKE THAT.
>>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>
>>> myriad
>>>>>
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>> myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Reply via email to