If you have multiple threads working to process messages, thread 1 will grab
messages 1-10 and start processing them, thread 2 will grab messages 11-20 and
start processing them in parallel, so ordering will be lost.
avoid using multiple threads when processing them, and you avoid that problem.
If logs get written to a disk queue, when new messages arrive they are processed
first, and messages from the queue get read and processed interspersed with the
new messages. The only way to avoid this problem is to not use a disk queue.
David Lang
On Mon, 10 Jul 2017, Scot Kreienkamp wrote:
Yep, understood on the ordering. Log4j is sending to rsyslog@localhost via
UDP, rsyslog is relaying via TCP. For my usage, vast majority of the time is
just fine.
I would have assumed that rsyslog would attempt processing of messages from any
queue in the order received though... No?
Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 | Office: 734-384-6403 | |
Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
-----Original Message-----
From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Monday, July 10, 2017 4:56 PM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Split messages options
note that there is some potential for messages to get out of order (over the
network with UDP, and inside rsyslog if you use disk queues or multiple threads)
UDP messages can be dropped if the network is busy as well (by any
router/firewall or receiving host)
but the vast majority of the time, everything will be in order.
David Lang
On Mon, 10 Jul 2017, Scot Kreienkamp wrote:
Date: Mon, 10 Jul 2017 20:49:42 +0000
From: Scot Kreienkamp <scot.kreienk...@la-z-boy.com>
Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
To: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Split messages options
That makes sense, I wasn't aware of the limitation in log4j. And yes, it is a
very old implementation and getting them to update would likely take years. :-)
In my case I'm transporting the logfile from the prod servers to a common
collector server internally that the devs can have access to; all I need to do
is reconstitute the file exactly as is on the other side. I'll change
syslogappender to a non-default port and use a custom template to forward it on
so I can force the correct tag on all forwarded messages, that way the
splitting won't matter and the logfile will be written on the destination
exactly as the source. That will solve my problem for now.
Thanks for the bit about log4j. I'll have to do some more research there.
Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 | Office: 734-384-6403 | |
Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
-----Original Message-----
From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Monday, July 10, 2017 4:38 PM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Split messages options
ahh, if this is a very old log4j, it will refuse to send UDP messages >1K in
size, so it splits things before they get to rsyslog, and your maxmessagesize
isn't going to help.
normally I am not in favor of writing to disk and then reading them, but if you
are stuck with an old log4j, that may be your best option.
older log4j implementations are UDP only with a 1000 byte max size
slightly newer implementations support TCP with larger message sizes (but I've
seen some that still limit you to 2k)
current implementations are far more flexible.
David Lang
I'm not bringing them in with imfile, log4j is using the builtin syslog
appender to submit it directly to syslog over UDP 514, the default UDP listener
port. I was assuming rsyslog is doing the splitting of the messages due to
size.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
This message is intended only for the individual or entity to which it is
addressed. It may contain privileged, confidential information which is exempt
from disclosure under applicable laws. If you are not the intended recipient,
you are strictly prohibited from disseminating or distributing this information
(other than to the intended recipient) or copying this information. If you
have received this communication in error, please notify us immediately by
e-mail or by telephone at the above number. Thank you.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
