On Thu, 7 Apr 2016, David Meiser wrote:
APC sends no delimiter at all.
I'd like to get this working with rsyslog, but I was able to get syslog-ng
running in the interim. It seems they don't care at all about broken
formatting.
oh, they care, they just care about different parts and break in different ways
on malformed messages :-)
Really, my goal is to put this in front of our alerting system, which can't
handle high-volume syslog, so that the alerting system can alert on
specific messages.
I'd suggest just using UDP from the APC to rsyslog. With UDP the packet boundry
is the delimiter. If you have a rsyslog server reasonably close (network wise)
to your logging source, UDP logging can be very reliable and then rsyslog can
queue/filter the messages before going to your alerting system.
David Lang
On Thu, Apr 7, 2016 at 9:45 AM Rainer Gerhards <[email protected]>
wrote:
2016-04-07 15:42 GMT+02:00 David Meiser <[email protected]>:
adding addtlframedelimiter="0" works-ish. I can send messages via
logger,
but not from an APC UPS. I'll check to see if I can receive messages
from
a Cisco device.
What's strange is that Logstash - which isn't at all what I'd prefer to
use
- have no issue with APC syslog messages. They choke on others, but not
on
APC.
Well, that's a typcial problem if you have something that does not
comply to the standards. NUL is not a frame delimitor, so different
projects have different ideas on how to treat anomalies.
Can you trace exactly what APC sends (e.g. via wireshark). Maybe they
don't use any frame delimiter at all... (I have seen this once in a
long while for real broken senders).
Rainer
On Thu, Apr 7, 2016 at 9:37 AM Rainer Gerhards <[email protected]
wrote:
2016-04-07 15:00 GMT+02:00 David Meiser <[email protected]>:
The problem when sending to logger was a problem with framing (NUL
frame
line ending).
I have checked the debug log. The NUL seems to be less of a problem,
the real problem is that the frame terminator LF is missing, so imtcp
waits for the message to be completed. Maybe that device incorrectly
uses NUL as frame terminator (an exceptionally bad choice). If so, you
may get along by using addtlframedelimitor="0" in the input parameter.
I admit I don't know if it works for NUL, which has assigned many
special meanings.
HTH
Rainer
However, I'm still having a problem with my test device - an
APC UPS.
Here's what is going on below: I logged into the device with an
incorrect
password. I then logged in and used the builtin test connection
function
of the APC to send this message: "APC: Test Syslog." When I sent the
"APC: Test Syslog" message, it looks like it comes through, but
includes
the tail end of unauthorized access message.
Here is what I see in the debug log:
3663.579327690:7f0bce158700: imptcp: epoll returned 1 events
3663.579333890:7f0bce158700: imptcp: new activity on session socket 16
3663.579339390:7f0bce158700: imptcp: removing socket 16 from epoll[8]
set
3663.579360490:7f0bce158700: imptcp: session on socket 16 closed with
iRet
0. 3663.579363490:7f0bce158700: imptcp going on epoll_wait
3663.904090302:7f0bce158700: imptcp: epoll returned 1 events
3663.904104602:7f0bce158700: imptcp: new connection on listen socket 9
3663.904131703:7f0bce158700: imptcp: added socket 16 to epoll[8] set
3663.904150003:7f0bce158700: imptcp going on epoll_wait
3663.905069517:7f0bce158700: imptcp: epoll returned 1 events
3663.905075917:7f0bce158700: imptcp: new activity on session socket 16
3663.905081817:7f0bce158700: imptcp: data(131072) on socket 16:
<13>Apr 7
08:54:23 192.168.1.2 APC: Test Syslog.horized user attempting to
access
the
Web interface from 192.168.2.2. 0x0006 3663.905090218:7f0bce158700:
imptcp
going on epoll_wait 3663.914878269:7f0bce158700: imptcp: epoll
returned 1
events 3663.914883469:7f0bce158700: imptcp: new activity on session
socket
16 3663.914888369:7f0bce158700: imptcp: removing socket 16 from
epoll[8]
set 3663.914913669:7f0bce158700: imptcp: session on socket 16 closed
with
iRet 0. 3663.914916669:7f0bce158700: imptcp going on epoll_wait
On Thu, Apr 7, 2016 at 8:27 AM David Meiser <[email protected]>
wrote:
For a udp message, I see this in the debug log:
https://gist.github.com/dmeiser/30ba9ef626ab3fb6fd9cf865a88a7d1e
For a tcp message, I see this in debug:
https://gist.github.com/dmeiser/24360ae40cf81573b833241c622b3b6a
Here is the entire debug session:
https://gist.github.com/dmeiser/11cf54f26834a7c09965c5bac098a727
Also, while I am using 7.4.7 in this session, I have tried the latest
release from the official repo, as well.
Thanks for the help!
On Wed, Apr 6, 2016 at 8:42 PM David Lang <[email protected]> wrote:
On Thu, 7 Apr 2016, David Meiser wrote:
The log is coming from the debug messages by running rsyslog -d
-n as
root
on the generic forwarder machine. This line comes from the debug
output:
2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15:
<5>Apr 6
15:56:34 user.name: hello world
I snipped the udp configuration from the config. It is identical
except
for the fact it uses imudp, not imptcp.
I did open port 514. I also had the thought that it could be an
selinux
problem so I disabled selinux. Then I moved it to 10514 and 1514
and
opened those ports. And then I disabled the firewall altogether.
Each
permutation had the same result.
Since I can use ncat to copy/paste a message which is received and
processed, I'm pretty sure there's something non-networking
related.
Ok, so if you are looking at the debug log and it's seeing the
message
arrive,
then rsyslog is getting and processing the message. What does the
debug
log show
happens after the message is recevied? it should show you everything
that's done
to the message, and you can look at the same thing for the UDP
message
to
see
what's different about them.
David Lang
On Wed, Apr 6, 2016 at 8:08 PM David Lang <[email protected]> wrote:
On Wed, 6 Apr 2016, David Meiser wrote:
I've tried it from 3 different locations, including a server on
that
vlan.
Here's why I think it's an rsyslog configuration issue: the
logger
messages
I send via TCP do show up in the debug log. Here's an example:
2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket 15:
<5>Apr 6
15:56:34 user.name: hello world
I still don't understand what this is an example of. Where are
you
getting
this
from?
That isn't processed. Change it to UDP, it goes through no
problem.
In regards to the APC UPS message, I pulled the message from the
rsyslog
debug log.
so if you go to server A and send a message to server B via UDP
it
shows
up, but
if you send it by TCP it doesn't?
The config you show below has no UDP configured.
There are many distros (including RedHat) that block TCP 514 by
default,
but
allow UDP 514, so check your iptables rules
David Lang
On Wed, Apr 6, 2016, 5:06 PM David Lang <[email protected]> wrote:
On Wed, 6 Apr 2016, David Meiser wrote:
As I continue to try to troubleshoot this, I tried using a
direct
ncat
connection. If I send "1.1.1.1 hello world" (simulating an IP
address
and
message), the output to the log file is:
output to what logfile with what config? Is this the config
shown
below?
note, workerthreads=4 is almost certinly wrong, you need a very
complex
config
to need even two threads. Having a queue on a write to a file
is
also
almost
always the wrong thing to do.
Apr 6 16:25:41 1
Apr 6 16:25:41 .1.1 hello world
not a valid syslog message
If I send "1\.1\.1\.1 hello world" the output is this:
Apr 6 16:26:16 .
Apr 6 16:26:16 .
Apr 6 16:26:16 .
Apr 6 16:26:16 h
Apr 6 16:26:16 ello world
also not a valid syslog message
If I send "myserver.domain.com hello world" the output is
this:
Apr 6 16:28:28 myserver.domain.com hello world
also not a valid syslog message
If I send a raw syslog message "<13>Apr 6 14:37:38 1.1.1.1
Test
Syslog."
I
get:
Apr 6 14:37:38 1.1.1.1 Test Syslog.
This is a valid syslog message, and is the output I would
expect
if
you
are
doing a simple write with the traditional format.
So, I go back to the original device that sent the TCP syslog
message
(an
APC UPS) and send another test directly to the server and I
see
this in
the
logs:
5015.675372226:7f6ce2981700: imptcp: data(131072) on socket
19:
<13>Apr 6
16:36:54 1.1.1.1 APC: Test Syslog.rom o world
how are you seeing this.
And in the log I get nothing.
so if it works from some places and not others, the first
thing to
do is
to look
at the network and see if there are any network things that
would
block
the
communication from the place that doesn't work, but don't
block it
from
the
place that does work.
David Lang
On Wed, Apr 6, 2016 at 4:03 PM David Meiser <
[email protected]>
wrote:
I am trying to setup a generic syslog forwarder that accepts
messages
on
tcp & udp. Using imptcp, I see messages come in, but no
rulesets
are
processed. My ruleset, right now, is set to take tcp
messages
and
just
output them to file (for troubleshooting). UDP works fine.
Here is what I see in debug mode:
2594.333580185:7fd29ab2b700: imptcp: epoll returned 1 events
2594.333594886:7fd29ab2b700: imptcp: new connection on listen
socket 9
2594.334657804:7fd29ab2b700: imptcp: added socket 15 to
epoll[8]
set
2594.334670004:7fd29ab2b700: imptcp going on epoll_wait
2594.334673904:7fd29ab2b700: imptcp: epoll returned 1 events
2594.334686204:7fd29ab2b700: imptcp: new activity on session
socket 15
2594.334692604:7fd29ab2b700: imptcp: data(131072) on socket
15:
<5>Apr 6
15:56:34 user.name: hello world 2594.334701804:7fd29ab2b700:
imptcp:
removing socket 15 from epoll[8] set
2594.335235814:7fd29ab2b700:
imptcp:
session on socket 15 closed with iRet 0.
2594.335240814:7fd29ab2b700:
imptcp going on epoll_wait
Here is my config:
module(load="imptcp")
input(
type="imptcp"
port="514"
ruleset="remote"
)
ruleset(
name="remote" queue.workerthreads="4"
queue.filename="srvrfwd"
queue.type="LinkedList"
queue.syncqueuefiles="on"
queue.maxdiskspace="10g"
queue.saveonshutdown="on"
queue.size="10000000"
) {
action(
type="omfile"
file="/var/log/debug"
)
}
I've also tried the sample config from the website:
input(type="imptcp" port="10514" ruleset="writeRemoteData")
ruleset(name="writeRemoteData" queue.type="fixedArray"
queue.size="250000"
queue.dequeueBatchSize="4096" queue.workerThreads="4"
queue.workerThreadMinimumMessages="60000" ) {
action(type="omfile"
file="/var/log/remote.log" ioBufferSize="64k"
flushOnTXEnd="off"
asyncWriting="on") }
Same issue.
Any thoughts?
Thank you,
Dave
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST
if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
