Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

[David Lang]

On Wed, 18 Oct 2017, deoren wrote:


On 10/18/2017 3:15 PM, David Lang wrote:

On Wed, 18 Oct 2017, deoren wrote:


On 10/18/2017 1:36 PM, David Lang wrote:

On Wed, 18 Oct 2017, deoren wrote:
Since the sender and receiver in this are both the latest versions of 
rsyslog (with the plan for the setup to remain that way), can I scale 
the accepted message size values to properly accommodate non-standard 
message sizes (delivered via JSON payloads)?


up to 128K should not be a problem, I believe that to scale the message 
size >128K you need to change a setting in the source.


Do you have experience delivering messages that large? I wonder whether I'm 
going about this the right way.


I wasn't using relp, but I did see logs hit 128k and get truncated a few times.

large maxmessagesize leads to wasted memory in rsyslog, but nothing more severe 
than that.


The particular box I'm having trouble with runs a Redmine instance that 
is heavily used and has lots of activity within its production log and 
development log files (large entries). I'd like to deliver entire log 
entries instead of truncated versions if possible.


how large are the lines?


At present, the development log file has a line with a length of 2997 
characters and the production.log file has a line with a length of 8608 
characters.


those are quite reasonable


I checked again and found a single line of 24405 characters. I've not yet 
checked back through historical log files to see what some of the longer 
lines were.


if your maxmessagesize was 64k, that should not have been a problem.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

[deoren]

On 10/18/2017 3:15 PM, David Lang wrote:

On Wed, 18 Oct 2017, deoren wrote:


On 10/18/2017 1:36 PM, David Lang wrote:

On Wed, 18 Oct 2017, deoren wrote:
Since the sender and receiver in this are both the latest versions 
of rsyslog (with the plan for the setup to remain that way), can I 
scale the accepted message size values to properly accommodate 
non-standard message sizes (delivered via JSON payloads)?


up to 128K should not be a problem, I believe that to scale the 
message size >128K you need to change a setting in the source.


Do you have experience delivering messages that large? I wonder whether 
I'm going about this the right way.


The particular box I'm having trouble with runs a Redmine instance 
that is heavily used and has lots of activity within its production 
log and development log files (large entries). I'd like to deliver 
entire log entries instead of truncated versions if possible.


how large are the lines?


At present, the development log file has a line with a length of 2997 
characters and the production.log file has a line with a length of 
8608 characters.


those are quite reasonable


I checked again and found a single line of 24405 characters. I've not 
yet checked back through historical log files to see what some of the 
longer lines were.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

[deoren]

On 10/18/2017 5:02 PM, deoren wrote:

On 10/18/2017 3:22 PM, Rainer Gerhards wrote:

The queue errors are bad. Anything else in regard to that queue?


After discussing it on this thread, I stopped rsyslog yesterday and 
moved all content from /var/spool/rsyslog to a different directory, 
hoping to have rsyslog come back online with an empty queue. I was ready 
to sacrifice the messages already queued at that point if I could just 
get the box sending log messages in a stable manner to the receiver.


After some time, the queue began filling up again. As I noted earlier, I 
updated the forward action to use omfwd instead of omrelp and the queue 
cleared. Since I had good results with that change, I stopped rsyslog 
and then moved back the files I had previously moved out of 
/var/spool/rsyslog and started the daemon back up.


The queue quickly drained, but the remote receiver crashed a few times 
in the process. Each time systemd would kick rsyslog and get it going 
again.



The thread messages are informational and part of the better status
indicators. They say that the load was so high an additional worker 
thread

was started... And later shut down as it was no longer needed.
Thank you for including the additional info in the new version. It's 
nice to be able to spot at a glance when rsyslog finds itself under load.


I stopped rsyslog and then started it up again and now have these messages:

root@redminebox:/var/spool/rsyslog# tail /var/log/rsyslog.log
2017-10-18T17:17:09.004746-05:00 redminebox rsyslogd: -- MARK --
2017-10-18T17:37:09.028716-05:00 redminebox rsyslogd: -- MARK --
2017-10-18T17:57:09.128810-05:00 redminebox rsyslogd: -- MARK --
2017-10-18T18:17:09.228894-05:00 redminebox rsyslogd: -- MARK --
2017-10-18T18:37:09.329050-05:00 redminebox rsyslogd: -- MARK --
2017-10-18T18:40:01.669581-05:00 redminebox rsyslogd: rsyslogd's groupid 
changed to 103
2017-10-18T18:40:01.669990-05:00 redminebox rsyslogd: rsyslogd's userid 
changed to 101
2017-10-18T18:40:01.670521-05:00 redminebox rsyslogd: ForwardToSawmill1 
queue[DA]: lost 5 messages from diskqueue (invalid .qi file) [v8.30.0]
2017-10-18T18:40:01.670622-05:00 redminebox rsyslogd: problem on disk 
queue 'ForwardToSawmill1 queue[DA]': queue files contain 5 messages 
fewer than specified in .qi file -- we lost those messages. That's all 
we know. [v8.30.0 try http://www.rsyslog.com/e/0 ]
2017-10-18T18:40:01.670727-05:00 redminebox rsyslogd:  [origin 
software="rsyslogd" swVersion="8.30.0" x-pid="1116" 
x-info="http://www.rsyslog.com;] start


Messages are flowing and at the the impstats output shows that no 
messages are held in the forward queue.


Prior to restarting rsyslog the impstats output showed that 6 messages 
were held in the queue. That was the case for the last several hours.


Is omfwd and imptcp better at handling unexpected message sizes than 
omrelp and imrelp?


On a related note, I looked back over the documentation and I found this 
under the imrelp page[2]:


> MaxDataSize  Default is the global message size Sets the 
max message size (in bytes) that can be received. Any messages above 
this size will be rejected causing the relp client to reconnect and retry.


* Does that refer to the maxMessageSize setting for the global() 
configuration object[1]?


* Are rejected messages logged anywhere? If not, would you be open to 
adding that option as an enhancement to a future version? With the 
imptcp module noting messages larger than max msg size (which I guess is 
another reference to the MaxDataSize global() configuration object?), 
that is helpful with detecting a mismatch between the sent message and 
what is allowed. By having the option to enable logging that a message 
was rejected via imrelp it would help with troubleshooting.


refs:

[1] http://www.rsyslog.com/doc/v8-stable/rainerscript/global.html
[2] http://www.rsyslog.com/doc/v8-stable/configuration/modules/imrelp.html

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

[deoren]

On 10/18/2017 12:34 PM, Rainer Gerhards wrote:

2017-10-18 1:14 GMT+02:00 deoren
:

On 10/17/2017 3:45 PM, Rainer Gerhards wrote:


Errno 11 seems to be EAGAIN, more a status than a warning. The full Debug
log may reveal details.



Is the debug on demand log file sufficient or should enabling debug mode at
startup the better route?


debug on startup is the way to go, else we may miss an important
triggering condition.

Rainer


Acknowledged. If I capture a full debug log in the future, where do I 
send it? What is an acceptable file size?

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

[deoren]

On 10/18/2017 3:22 PM, Rainer Gerhards wrote:

The queue errors are bad. Anything else in regard to that queue?


After discussing it on this thread, I stopped rsyslog yesterday and 
moved all content from /var/spool/rsyslog to a different directory, 
hoping to have rsyslog come back online with an empty queue. I was ready 
to sacrifice the messages already queued at that point if I could just 
get the box sending log messages in a stable manner to the receiver.


After some time, the queue began filling up again. As I noted earlier, I 
updated the forward action to use omfwd instead of omrelp and the queue 
cleared. Since I had good results with that change, I stopped rsyslog 
and then moved back the files I had previously moved out of 
/var/spool/rsyslog and started the daemon back up.


The queue quickly drained, but the remote receiver crashed a few times 
in the process. Each time systemd would kick rsyslog and get it going again.



The thread messages are informational and part of the better status
indicators. They say that the load was so high an additional worker thread
was started... And later shut down as it was no longer needed.
Thank you for including the additional info in the new version. It's 
nice to be able to spot at a glance when rsyslog finds itself under load.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Rsyslog Message Queues

[emaleth]

Thank you very much for your answer and explaination :)

Greetings, Sabine

Von meinem iPhone gesendet

> Am 17.10.2017 um 16:42 schrieb David Lang :
> 
>> On Tue, 17 Oct 2017, emaleth wrote:
>> 
>> Because it was still in the $Work Directory, it disappeared only after 
>> restarting rsyslog.
> 
> That doesn't mean that the logs weren't sent.
> 
> Rsyslog creates a file when it first starts to spill to disk, but it doesn't 
> delete the file when empty because that would cause a race condition with 
> another thread trying to write out to the file.
> 
> So once you start to spill to disk, the last file for that queue will remain 
> until rsyslog shuts down and it knows that there is no chance of needing to 
> use the file.
> 
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

[Rainer Gerhards]

The queue errors are bad. Anything else in regard to that queue?

The thread messages are informational and part of the better status
indicators. They say that the load was so high an additional worker thread
was started... And later shut down as it was no longer needed.

Rainer

Sent from phone, thus brief.

Am 18.10.2017 22:15 schrieb "David Lang" :

On Wed, 18 Oct 2017, deoren wrote:

On 10/18/2017 1:36 PM, David Lang wrote:
>
>> On Wed, 18 Oct 2017, deoren wrote:
>>
>>> Since the sender and receiver in this are both the latest versions of
>>> rsyslog (with the plan for the setup to remain that way), can I scale the
>>> accepted message size values to properly accommodate non-standard message
>>> sizes (delivered via JSON payloads)?
>>>
>>
>> up to 128K should not be a problem, I believe that to scale the message
>> size >128K you need to change a setting in the source.
>>
>> The particular box I'm having trouble with runs a Redmine instance that
>>> is heavily used and has lots of activity within its production log and
>>> development log files (large entries). I'd like to deliver entire log
>>> entries instead of truncated versions if possible.
>>>
>>
>> how large are the lines?
>>
>
> At present, the development log file has a line with a length of 2997
> characters and the production.log file has a line with a length of 8608
> characters.
>

those are quite reasonable


Yesterday I muddied the waters a bit when I opted to test using a regex
> with imfile to treat a large batch of content between a marker as part of a
> single message. I just picked two samples and found that one was 25K and
> the other was 97K.
>

that's starting to get large enough to worry.


I should note that I have this setting enabled on all of our Ubuntu boxes:
>
> maxMessageSize="64k"
>

bump this up if you are going to be using the large size.


I forget which article I found that mentioned it, but the idea was that
> JSON message content is larger than standard syslog messages and the
> increased max message size would help accommodate that.
>

yes, json is more verbose than an unformatted text log, it also depends on
if you duplicate data in the json version (do you have both the plain text
message and the data parsed from it?)

one thing you can do is to have your senders check the size of the log (set
$.x = exec_template("template"); test for lenght($.x)), and if the message
is too large, write it locally to a 'oversized' file and then either figure
a way to trim the message before it's sent, or send a placeholder saying
that you had a message that was too large to send, so you are storing it
locally.


It could be completely unrelated, but I found these messages when I checked
> back on the /var/log/rsyslog.log file:
>
> 2017-10-18T13:50:15.134882-05:00 redminebox rsyslogd: main Q:Reg: high
> activity - starting 1 additional worker thread(s), currently 1 active
> worker threads. [v8.30.0 try http://www.rsyslog.com/e/2439 ]
> 2017-10-18T13:50:15.446207-05:00 redminebox rsyslogd: ForwardToSawmill1
> queue[DA]: qDeqDisk error happened at around offset 4152 [v8.30.0 try
> http://www.rsyslog.com/e/2059 ]
> 2017-10-18T13:50:15.446464-05:00 redminebox rsyslogd: ForwardToSawmill1
> queue[DA]: error dequeueing element - ignoring, but strange things may
> happen [v8.30.0 try http://www.rsyslog.com/e/2059 ]
> 2017-10-18T13:50:15.525897-05:00 redminebox rsyslogd: ForwardToSawmill1
> queue[DA]: qDeqDisk error happened at around offset 12289 [v8.30.0 try
> http://www.rsyslog.com/e/2059 ]
> 2017-10-18T13:50:15.526146-05:00 redminebox rsyslogd: ForwardToSawmill1
> queue[DA]: error dequeueing element - ignoring, but strange things may
> happen [v8.30.0 try http://www.rsyslog.com/e/2059 ]
> 2017-10-18T13:51:15.847344-05:00 redminebox rsyslogd: main Q:Reg: worker
> thread 1a8d180 terminated, now 1 active worker threads [v8.30.0 try
> http://www.rsyslog.com/e/2439 ]
>
> This is the first time I've seen those log entries.
>

these sound significant, but Rainer will need to comment on them.

David Lang

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

[David Lang]

On Wed, 18 Oct 2017, deoren wrote:


On 10/18/2017 1:36 PM, David Lang wrote:

On Wed, 18 Oct 2017, deoren wrote:
Since the sender and receiver in this are both the latest versions of 
rsyslog (with the plan for the setup to remain that way), can I scale the 
accepted message size values to properly accommodate non-standard message 
sizes (delivered via JSON payloads)?


up to 128K should not be a problem, I believe that to scale the message 
size >128K you need to change a setting in the source.


The particular box I'm having trouble with runs a Redmine instance that is 
heavily used and has lots of activity within its production log and 
development log files (large entries). I'd like to deliver entire log 
entries instead of truncated versions if possible.


how large are the lines?


At present, the development log file has a line with a length of 2997 
characters and the production.log file has a line with a length of 8608 
characters.


those are quite reasonable

Yesterday I muddied the waters a bit when I opted to test using a regex with 
imfile to treat a large batch of content between a marker as part of a single 
message. I just picked two samples and found that one was 25K and the other 
was 97K.


that's starting to get large enough to worry.


I should note that I have this setting enabled on all of our Ubuntu boxes:

maxMessageSize="64k"


bump this up if you are going to be using the large size.

I forget which article I found that mentioned it, but the idea was that JSON 
message content is larger than standard syslog messages and the increased max 
message size would help accommodate that.


yes, json is more verbose than an unformatted text log, it also depends on if 
you duplicate data in the json version (do you have both the plain text message 
and the data parsed from it?)


one thing you can do is to have your senders check the size of the log (set $.x 
= exec_template("template"); test for lenght($.x)), and if the message is too 
large, write it locally to a 'oversized' file and then either figure a way to 
trim the message before it's sent, or send a placeholder saying that you had a 
message that was too large to send, so you are storing it locally.


It could be completely unrelated, but I found these messages when I checked 
back on the /var/log/rsyslog.log file:


2017-10-18T13:50:15.134882-05:00 redminebox rsyslogd: main Q:Reg: high 
activity - starting 1 additional worker thread(s), currently 1 active worker 
threads. [v8.30.0 try http://www.rsyslog.com/e/2439 ]
2017-10-18T13:50:15.446207-05:00 redminebox rsyslogd: ForwardToSawmill1 
queue[DA]: qDeqDisk error happened at around offset 4152 [v8.30.0 try 
http://www.rsyslog.com/e/2059 ]
2017-10-18T13:50:15.446464-05:00 redminebox rsyslogd: ForwardToSawmill1 
queue[DA]: error dequeueing element - ignoring, but strange things may happen 
[v8.30.0 try http://www.rsyslog.com/e/2059 ]
2017-10-18T13:50:15.525897-05:00 redminebox rsyslogd: ForwardToSawmill1 
queue[DA]: qDeqDisk error happened at around offset 12289 [v8.30.0 try 
http://www.rsyslog.com/e/2059 ]
2017-10-18T13:50:15.526146-05:00 redminebox rsyslogd: ForwardToSawmill1 
queue[DA]: error dequeueing element - ignoring, but strange things may happen 
[v8.30.0 try http://www.rsyslog.com/e/2059 ]
2017-10-18T13:51:15.847344-05:00 redminebox rsyslogd: main Q:Reg: worker 
thread 1a8d180 terminated, now 1 active worker threads [v8.30.0 try 
http://www.rsyslog.com/e/2439 ]


This is the first time I've seen those log entries.


these sound significant, but Rainer will need to comment on them.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

[deoren]

On 10/18/2017 1:36 PM, David Lang wrote:

On Wed, 18 Oct 2017, deoren wrote:



I checked and sawmill1 is having trouble sending the messages on to 
the "downstream" receivers (sawmill2, sawmill3). Based on the "... at 
least 232 byte larger than max msg size ..." log entries, I'd guess 
that message size is causing the issue here.


I'm speaking from ignorance here, but shouldn't omrelp (or I guess 
more appropriately imrelp on the receiver) note the problematic 
message size in the non-debug level logs?


you can run into trouble logging too much about problems like this


Could be a useful troubleshooting flag, but I'm guessing that a debug 
log file already indicates this.




It also appears that rsyslog is crashing when receiving the data via 
imptcp. Is that expected?


no, the message should be truncated and the rest of the message become 
the next message.


Since the sender and receiver in this are both the latest versions of 
rsyslog (with the plan for the setup to remain that way), can I scale 
the accepted message size values to properly accommodate non-standard 
message sizes (delivered via JSON payloads)?


up to 128K should not be a problem, I believe that to scale the message 
size

128K you need to change a setting in the source.


The particular box I'm having trouble with runs a Redmine instance 
that is heavily used and has lots of activity within its production 
log and development log files (large entries). I'd like to deliver 
entire log entries instead of truncated versions if possible.


how large are the lines?


At present, the development log file has a line with a length of 2997 
characters and the production.log file has a line with a length of 8608 
characters.


Yesterday I muddied the waters a bit when I opted to test using a regex 
with imfile to treat a large batch of content between a marker as part 
of a single message. I just picked two samples and found that one was 
25K and the other was 97K.


I should note that I have this setting enabled on all of our Ubuntu boxes:

maxMessageSize="64k"

I forget which article I found that mentioned it, but the idea was that 
JSON message content is larger than standard syslog messages and the 
increased max message size would help accommodate that.


It could be completely unrelated, but I found these messages when I 
checked back on the /var/log/rsyslog.log file:


2017-10-18T13:50:15.134882-05:00 redminebox rsyslogd: main Q:Reg: high 
activity - starting 1 additional worker thread(s), currently 1 active 
worker threads. [v8.30.0 try http://www.rsyslog.com/e/2439 ]
2017-10-18T13:50:15.446207-05:00 redminebox rsyslogd: ForwardToSawmill1 
queue[DA]: qDeqDisk error happened at around offset 4152 [v8.30.0 try 
http://www.rsyslog.com/e/2059 ]
2017-10-18T13:50:15.446464-05:00 redminebox rsyslogd: ForwardToSawmill1 
queue[DA]: error dequeueing element - ignoring, but strange things may 
happen [v8.30.0 try http://www.rsyslog.com/e/2059 ]
2017-10-18T13:50:15.525897-05:00 redminebox rsyslogd: ForwardToSawmill1 
queue[DA]: qDeqDisk error happened at around offset 12289 [v8.30.0 try 
http://www.rsyslog.com/e/2059 ]
2017-10-18T13:50:15.526146-05:00 redminebox rsyslogd: ForwardToSawmill1 
queue[DA]: error dequeueing element - ignoring, but strange things may 
happen [v8.30.0 try http://www.rsyslog.com/e/2059 ]
2017-10-18T13:51:15.847344-05:00 redminebox rsyslogd: main Q:Reg: worker 
thread 1a8d180 terminated, now 1 active worker threads [v8.30.0 try 
http://www.rsyslog.com/e/2439 ]


This is the first time I've seen those log entries.

We're running v8.30.0 from the official Ubuntu PPA on all of our Ubuntu 
nodes.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

[David Lang]

On Wed, 18 Oct 2017, deoren wrote:



I checked and sawmill1 is having trouble sending the messages on to the 
"downstream" receivers (sawmill2, sawmill3). Based on the "... at least 
232 byte larger than max msg size ..." log entries, I'd guess that 
message size is causing the issue here.


I'm speaking from ignorance here, but shouldn't omrelp (or I guess more 
appropriately imrelp on the receiver) note the problematic message size 
in the non-debug level logs?


you can run into trouble logging too much about problems like this

It also appears that rsyslog is crashing when receiving the data via 
imptcp. Is that expected?


no, the message should be truncated and the rest of the message become the next 
message.


Since the sender and receiver in this are both the latest versions of 
rsyslog (with the plan for the setup to remain that way), can I scale 
the accepted message size values to properly accommodate non-standard 
message sizes (delivered via JSON payloads)?


up to 128K should not be a problem, I believe that to scale the message size 

128K you need to change a setting in the source.


The particular box I'm having trouble with runs a Redmine instance that 
is heavily used and has lots of activity within its production log and 
development log files (large entries). I'd like to deliver entire log 
entries instead of truncated versions if possible.


how large are the lines?
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

[Rainer Gerhards]

2017-10-18 1:14 GMT+02:00 deoren
:
> On 10/17/2017 3:45 PM, Rainer Gerhards wrote:
>>
>> Errno 11 seems to be EAGAIN, more a status than a warning. The full Debug
>> log may reveal details.
>
>
> Is the debug on demand log file sufficient or should enabling debug mode at
> startup the better route?

debug on startup is the way to go, else we may miss an important
triggering condition.

Rainer
>
> On a different note, will rsyslog accept a message for remote delivery that
> it is not capable of delivering? I recall reading that rsyslog will truncate
> messages that are too long. Is that the case also for JSON payloads within
> the MSG property?
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

[deoren]

On 10/18/2017 12:02 PM, deoren wrote:

On 10/18/2017 11:51 AM, deoren wrote:

On 10/17/2017 6:57 PM, David Lang wrote:
Yes, rsyslog will accept messages it can't deliver, the accepting of 
messages is decoupled from the delivery.


if a message is too long, it will get ttruncated, even if it's json 
(at that point it's a string of bytes, rsyslog has no way of knowing 
that it's json)


Thanks for the confirmation.

On a related note, I recalled reading that someone else with omrelp 
delivery issues switched to omfwd + TCP and the messages were sent 
without issue. I gave it a try and sure enough, the queue cleared and 
old messages were delivered.


2017-10-18T11:35:58.746189-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' resumed (module 'omrelp') [v8.30.0 try 
http://www.rsyslog.com/e/2359 ]
2017-10-18T11:35:58.746527-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' suspended (module 'omrelp'), retry 0. There should 
be messages before this one giving the reason for suspension. [v8.30.0 
try http://www.rsyslog.com/e/2007 ]
2017-10-18T11:36:29.181334-05:00 redminebox rsyslogd:  [origin 
software="rsyslogd" swVersion="8.30.0" x-pid="7723" 
x-info="http://www.rsyslog.com;] exiting on signal 15.
2017-10-18T11:40:29.381587-05:00 redminebox rsyslogd: rsyslogd's 
groupid changed to 103
2017-10-18T11:40:29.382641-05:00 redminebox rsyslogd: rsyslogd's 
userid changed to 101
2017-10-18T11:40:29.382824-05:00 redminebox rsyslogd:  [origin 
software="rsyslogd" swVersion="8.30.0" x-pid="31636" 
x-info="http://www.rsyslog.com;] start
2017-10-18T11:40:39.386593-05:00 redminebox rsyslogd: omfwd: 
TCPSendBuf error -2027, destruct TCP Connection to 
REMOTE_RECEIVER_IP_HERE:514 [v8.30.0 try http://www.rsyslog.com/e/2027 ]
2017-10-18T11:40:39.386805-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' suspended (module 'builtin:omfwd'), retry 0. There 
should be messages before this one giving the reason for suspension. 
[v8.30.0 try http://www.rsyslog.com/e/2007 ]
2017-10-18T11:40:39.387139-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' resumed (module 'builtin:omfwd') [v8.30.0 try 
http://www.rsyslog.com/e/2359 ]


I wish I had tried doing that before moving out the prior ~ 1 GB of 
held messages. If I want to send those, is the fix as simple as 
shutting down rsyslog and overwriting the contents of 
/var/spool/rsyslog (which currently is pretty sparse)?


I did just that and the queued messages were recognized, and delivered 
to the remote receiver VERY quickly.


Related messages in /var/log/rsyslog.log:

2017-10-18T11:57:07.577559-05:00 redminebox rsyslogd: rsyslogd's groupid 
changed to 103
2017-10-18T11:57:07.577923-05:00 redminebox rsyslogd: rsyslogd's userid 
changed to 101
2017-10-18T11:57:07.578039-05:00 redminebox rsyslogd:  [origin 
software="rsyslogd" swVersion="8.30.0" x-pid="32946" 
x-info="http://www.rsyslog.com;] start
2017-10-18T11:57:09.433779-05:00 redminebox rsyslogd: omfwd: TCPSendBuf 
error -2027, destruct TCP Connection to REMOVE_RECEIVER_IP_ADDRESS:514 
[v8.30.0 try http://www.rsyslog.com/e/2027 ]
2017-10-18T11:57:09.433982-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' suspended (module 'builtin:omfwd'), retry 0. There 
should be messages before this one giving the reason for suspension. 
[v8.30.0 try http://www.rsyslog.com/e/2007 ]
2017-10-18T11:57:09.434136-05:00 redminebox rsyslogd: omfwd: TCPSendBuf 
error -2027, destruct TCP Connection to REMOVE_RECEIVER_IP_ADDRESS:514 
[v8.30.0 try http://www.rsyslog.com/e/2027 ]
2017-10-18T11:57:09.434229-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' suspended (module 'builtin:omfwd'), retry 0. There 
should be messages before this one giving the reason for suspension. 
[v8.30.0 try http://www.rsyslog.com/e/2007 ]
2017-10-18T11:57:09.434389-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' resumed (module 'builtin:omfwd') [v8.30.0 try 
http://www.rsyslog.com/e/2359 ]
2017-10-18T11:57:09.434742-05:00 redminebox rsyslogd: omfwd: TCPSendBuf 
error -2027, destruct TCP Connection to REMOVE_RECEIVER_IP_ADDRESS:514 
[v8.30.0 try http://www.rsyslog.com/e/2027 ]
2017-10-18T11:57:09.434850-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' suspended (module 'builtin:omfwd'), retry 0. There 
should be messages before this one giving the reason for suspension. 
[v8.30.0 try http://www.rsyslog.com/e/2007 ]
2017-10-18T11:57:19.441198-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' resumed (module 'builtin:omfwd') [v8.30.0 try 
http://www.rsyslog.com/e/2359 ]
2017-10-18T11:57:19.444940-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' resumed (module 'builtin:omfwd') [v8.30.0 try 
http://www.rsyslog.com/e/2359 ]
2017-10-18T11:57:30.466879-05:00 redminebox rsyslogd: omfwd: TCPSendBuf 
error -2027, destruct TCP Connection to REMOVE_RECEIVER_IP_ADDRESS:514 
[v8.30.0 try http://www.rsyslog.com/e/2027 ]
2017-10-18T11:57:30.467075-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' suspended (module 

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

[deoren]

On 10/18/2017 11:51 AM, deoren wrote:

On 10/17/2017 6:57 PM, David Lang wrote:
Yes, rsyslog will accept messages it can't deliver, the accepting of 
messages is decoupled from the delivery.


if a message is too long, it will get ttruncated, even if it's json 
(at that point it's a string of bytes, rsyslog has no way of knowing 
that it's json)


Thanks for the confirmation.

On a related note, I recalled reading that someone else with omrelp 
delivery issues switched to omfwd + TCP and the messages were sent 
without issue. I gave it a try and sure enough, the queue cleared and 
old messages were delivered.


2017-10-18T11:35:58.746189-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' resumed (module 'omrelp') [v8.30.0 try 
http://www.rsyslog.com/e/2359 ]
2017-10-18T11:35:58.746527-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' suspended (module 'omrelp'), retry 0. There should 
be messages before this one giving the reason for suspension. [v8.30.0 
try http://www.rsyslog.com/e/2007 ]
2017-10-18T11:36:29.181334-05:00 redminebox rsyslogd:  [origin 
software="rsyslogd" swVersion="8.30.0" x-pid="7723" 
x-info="http://www.rsyslog.com;] exiting on signal 15.
2017-10-18T11:40:29.381587-05:00 redminebox rsyslogd: rsyslogd's groupid 
changed to 103
2017-10-18T11:40:29.382641-05:00 redminebox rsyslogd: rsyslogd's userid 
changed to 101
2017-10-18T11:40:29.382824-05:00 redminebox rsyslogd:  [origin 
software="rsyslogd" swVersion="8.30.0" x-pid="31636" 
x-info="http://www.rsyslog.com;] start
2017-10-18T11:40:39.386593-05:00 redminebox rsyslogd: omfwd: TCPSendBuf 
error -2027, destruct TCP Connection to REMOTE_RECEIVER_IP_HERE:514 
[v8.30.0 try http://www.rsyslog.com/e/2027 ]
2017-10-18T11:40:39.386805-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' suspended (module 'builtin:omfwd'), retry 0. There 
should be messages before this one giving the reason for suspension. 
[v8.30.0 try http://www.rsyslog.com/e/2007 ]
2017-10-18T11:40:39.387139-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' resumed (module 'builtin:omfwd') [v8.30.0 try 
http://www.rsyslog.com/e/2359 ]


I wish I had tried doing that before moving out the prior ~ 1 GB of held 
messages. If I want to send those, is the fix as simple as shutting down 
rsyslog and overwriting the contents of /var/spool/rsyslog (which 
currently is pretty sparse)?


I did just that and the queued messages were recognized, and delivered 
to the remote receiver VERY quickly.


Related messages in /var/log/rsyslog.log:

2017-10-18T11:57:07.577559-05:00 redminebox rsyslogd: rsyslogd's groupid 
changed to 103
2017-10-18T11:57:07.577923-05:00 redminebox rsyslogd: rsyslogd's userid 
changed to 101
2017-10-18T11:57:07.578039-05:00 redminebox rsyslogd:  [origin 
software="rsyslogd" swVersion="8.30.0" x-pid="32946" 
x-info="http://www.rsyslog.com;] start
2017-10-18T11:57:09.433779-05:00 redminebox rsyslogd: omfwd: TCPSendBuf 
error -2027, destruct TCP Connection to REMOVE_RECEIVER_IP_ADDRESS:514 
[v8.30.0 try http://www.rsyslog.com/e/2027 ]
2017-10-18T11:57:09.433982-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' suspended (module 'builtin:omfwd'), retry 0. There 
should be messages before this one giving the reason for suspension. 
[v8.30.0 try http://www.rsyslog.com/e/2007 ]
2017-10-18T11:57:09.434136-05:00 redminebox rsyslogd: omfwd: TCPSendBuf 
error -2027, destruct TCP Connection to REMOVE_RECEIVER_IP_ADDRESS:514 
[v8.30.0 try http://www.rsyslog.com/e/2027 ]
2017-10-18T11:57:09.434229-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' suspended (module 'builtin:omfwd'), retry 0. There 
should be messages before this one giving the reason for suspension. 
[v8.30.0 try http://www.rsyslog.com/e/2007 ]
2017-10-18T11:57:09.434389-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' resumed (module 'builtin:omfwd') [v8.30.0 try 
http://www.rsyslog.com/e/2359 ]
2017-10-18T11:57:09.434742-05:00 redminebox rsyslogd: omfwd: TCPSendBuf 
error -2027, destruct TCP Connection to REMOVE_RECEIVER_IP_ADDRESS:514 
[v8.30.0 try http://www.rsyslog.com/e/2027 ]
2017-10-18T11:57:09.434850-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' suspended (module 'builtin:omfwd'), retry 0. There 
should be messages before this one giving the reason for suspension. 
[v8.30.0 try http://www.rsyslog.com/e/2007 ]
2017-10-18T11:57:19.441198-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' resumed (module 'builtin:omfwd') [v8.30.0 try 
http://www.rsyslog.com/e/2359 ]
2017-10-18T11:57:19.444940-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' resumed (module 'builtin:omfwd') [v8.30.0 try 
http://www.rsyslog.com/e/2359 ]
2017-10-18T11:57:30.466879-05:00 redminebox rsyslogd: omfwd: TCPSendBuf 
error -2027, destruct TCP Connection to REMOVE_RECEIVER_IP_ADDRESS:514 
[v8.30.0 try http://www.rsyslog.com/e/2027 ]
2017-10-18T11:57:30.467075-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' suspended (module 'builtin:omfwd'), retry 0. There 
should be messages 

Re: [rsyslog] If messages are stuck in a queue, do you have any option other than nuking the queue file(s)?

[deoren]

On 10/17/2017 6:57 PM, David Lang wrote:
Yes, rsyslog will accept messages it can't deliver, the accepting of 
messages is decoupled from the delivery.


if a message is too long, it will get ttruncated, even if it's json (at 
that point it's a string of bytes, rsyslog has no way of knowing that 
it's json)


Thanks for the confirmation.

On a related note, I recalled reading that someone else with omrelp 
delivery issues switched to omfwd + TCP and the messages were sent 
without issue. I gave it a try and sure enough, the queue cleared and 
old messages were delivered.


2017-10-18T11:35:58.746189-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' resumed (module 'omrelp') [v8.30.0 try 
http://www.rsyslog.com/e/2359 ]
2017-10-18T11:35:58.746527-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' suspended (module 'omrelp'), retry 0. There should 
be messages before this one giving the reason for suspension. [v8.30.0 
try http://www.rsyslog.com/e/2007 ]
2017-10-18T11:36:29.181334-05:00 redminebox rsyslogd:  [origin 
software="rsyslogd" swVersion="8.30.0" x-pid="7723" 
x-info="http://www.rsyslog.com;] exiting on signal 15.
2017-10-18T11:40:29.381587-05:00 redminebox rsyslogd: rsyslogd's groupid 
changed to 103
2017-10-18T11:40:29.382641-05:00 redminebox rsyslogd: rsyslogd's userid 
changed to 101
2017-10-18T11:40:29.382824-05:00 redminebox rsyslogd:  [origin 
software="rsyslogd" swVersion="8.30.0" x-pid="31636" 
x-info="http://www.rsyslog.com;] start
2017-10-18T11:40:39.386593-05:00 redminebox rsyslogd: omfwd: TCPSendBuf 
error -2027, destruct TCP Connection to REMOTE_RECEIVER_IP_HERE:514 
[v8.30.0 try http://www.rsyslog.com/e/2027 ]
2017-10-18T11:40:39.386805-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' suspended (module 'builtin:omfwd'), retry 0. There 
should be messages before this one giving the reason for suspension. 
[v8.30.0 try http://www.rsyslog.com/e/2007 ]
2017-10-18T11:40:39.387139-05:00 redminebox rsyslogd: action 
'ForwardToSawmill1' resumed (module 'builtin:omfwd') [v8.30.0 try 
http://www.rsyslog.com/e/2359 ]


I wish I had tried doing that before moving out the prior ~ 1 GB of held 
messages. If I want to send those, is the fix as simple as shutting down 
rsyslog and overwriting the contents of /var/spool/rsyslog (which 
currently is pretty sparse)?



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Missing libfastjson 0.99.7 in repository

[Florian Riedl]

We updated the RPMs. The libfastjson4 RPM that we provide should now
replace the libfastjson 0.99.4 that is provided through the base
repository.

Florian

2017-10-18 13:21 GMT+02:00 Cédric Jeanneret via rsyslog
:
> Apparently, a new libfastjson4 has been pushed and it replaces the
> libfastjson package. Just installing it did the trick, and rsyslog is
> running as expected.
>
> Thanks!
>
> --
> Cédric Jeanneret
> Senior Linux System Administrator
> Infrastructure Solutions
>
> Camptocamp SA
> PSE-A / EPFL, 1015 Lausanne
>
> Phone: +41 21 619 10 32
> Office: +41 21 619 10 02
> Email: cedric.jeanne...@camptocamp.com
>
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Updates 8.29 -> 8.30 broke several logs

[Rainer Gerhards]

Do you mean some logs were written to and some not?

If so, I need a Debug log to diagnose what is going on.

Rainer

Sent from phone, thus brief.

Am 18.10.2017 17:36 schrieb "Mike Schleif" :

> # cat /etc/centos-release
> CentOS Linux release 7.4.1708 (Core)
>
>
> After yum updates yesterday (see below,) several logs no longer logged,
> including /var/log/secure
>
> In the last hour, we rolled back that entire yum update, and logging
> appears to be as expected
>
> Please, advise. Thank you.
>
> ~ Mike
>
>
> # yum history info 62
> Loaded plugins: fastestmirror
> Transaction ID : 62
> Begin time : Tue Oct 17 07:42:51 2017
> Begin rpmdb: 597:442a35918ca922c515d3f9bbc38cb3733341358a
> End time   :07:43:00 2017 (9 seconds)
> End rpmdb  : 597:f817c423ae76bafaafaab823cfca6d4030e069f0
> User   : Jeffrey Reed 
> Return-Code: Success
> Command Line   : update
> Transaction performed with:
> Installed rpm-4.11.3-25.el7.x86_64  @base
> Installed yum-3.4.3-154.el7.centos.noarch   @base
> Installed yum-plugin-fastestmirror-1.1.31-42.el7.noarch @base
> Packages Altered:
> Updated epel-release-7-10.noarch   @epel
> Update   7-11.noarch   @epel-testing
> Updated libfastjson4-0.99.5-1.el7.x86_64   @rsyslog_v8
> Update   0.99.7-1.el7.x86_64   @rsyslog_v8
> Updated mysql-community-client-5.6.37-2.el7.x86_64 @mysql56-community
> Update 5.6.38-2.el7.x86_64 @mysql56-community
> Updated mysql-community-common-5.6.37-2.el7.x86_64 @mysql56-community
> Update 5.6.38-2.el7.x86_64 @mysql56-community
> Updated mysql-community-libs-5.6.37-2.el7.x86_64   @mysql56-community
> Update   5.6.38-2.el7.x86_64   @mysql56-community
> Updated rsyslog-8.29.0-2.el7.x86_64@rsyslog_v8
> Update  8.30.0-1.el7.x86_64@rsyslog_v8
> Updated rsyslog-mysql-8.29.0-2.el7.x86_64  @rsyslog_v8
> Update8.30.0-1.el7.x86_64  @rsyslog_v8
> history info
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Updates 8.29 -> 8.30 broke several logs

[Mike Schleif]

# cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)


After yum updates yesterday (see below,) several logs no longer logged,
including /var/log/secure

In the last hour, we rolled back that entire yum update, and logging
appears to be as expected

Please, advise. Thank you.

~ Mike


# yum history info 62
Loaded plugins: fastestmirror
Transaction ID : 62
Begin time : Tue Oct 17 07:42:51 2017
Begin rpmdb: 597:442a35918ca922c515d3f9bbc38cb3733341358a
End time   :07:43:00 2017 (9 seconds)
End rpmdb  : 597:f817c423ae76bafaafaab823cfca6d4030e069f0
User   : Jeffrey Reed 
Return-Code: Success
Command Line   : update
Transaction performed with:
Installed rpm-4.11.3-25.el7.x86_64  @base
Installed yum-3.4.3-154.el7.centos.noarch   @base
Installed yum-plugin-fastestmirror-1.1.31-42.el7.noarch @base
Packages Altered:
Updated epel-release-7-10.noarch   @epel
Update   7-11.noarch   @epel-testing
Updated libfastjson4-0.99.5-1.el7.x86_64   @rsyslog_v8
Update   0.99.7-1.el7.x86_64   @rsyslog_v8
Updated mysql-community-client-5.6.37-2.el7.x86_64 @mysql56-community
Update 5.6.38-2.el7.x86_64 @mysql56-community
Updated mysql-community-common-5.6.37-2.el7.x86_64 @mysql56-community
Update 5.6.38-2.el7.x86_64 @mysql56-community
Updated mysql-community-libs-5.6.37-2.el7.x86_64   @mysql56-community
Update   5.6.38-2.el7.x86_64   @mysql56-community
Updated rsyslog-8.29.0-2.el7.x86_64@rsyslog_v8
Update  8.30.0-1.el7.x86_64@rsyslog_v8
Updated rsyslog-mysql-8.29.0-2.el7.x86_64  @rsyslog_v8
Update8.30.0-1.el7.x86_64  @rsyslog_v8
history info
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Missing libfastjson 0.99.7 in repository

[Cédric Jeanneret via rsyslog]

Apparently, a new libfastjson4 has been pushed and it replaces the
libfastjson package. Just installing it did the trick, and rsyslog is
running as expected.

Thanks!

-- 
Cédric Jeanneret
Senior Linux System Administrator
Infrastructure Solutions

Camptocamp SA
PSE-A / EPFL, 1015 Lausanne

Phone: +41 21 619 10 32
Office: +41 21 619 10 02
Email: cedric.jeanne...@camptocamp.com




signature.asc
Description: OpenPGP digital signature
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Exclude patterns for imfile whith different multiline format

[Rainer Gerhards]

2017-10-18 9:27 GMT+02:00 mostolog--- via rsyslog :
> It doesn't seem possible to exclude specific files when using directory
> wilcards too.
>
> eg: /logs/**/^{file_format_X.log}

** is NOT supported as it would require us to put inotify handles over
a very large directory set

imfile uses this for obtaining the filename:

https://linux.die.net/man/3/glob

Rainer
>
>
> Perhaps http://www.linuxjournal.com/content/bash-extended-globbing could do
> the trick, but i was wondering if there won't make more sense to use REGEX
> or exclude paratemer (or...)
>
>
> Regards
>
> On 17/10/17 16:59, Rainer Gerhards wrote:
>>
>> You cannot exclude, bit you can include those that you want. I guess with
>> ugly globbing that should be possible. Best try with ls first.
>>
>> http://tldp.org/LDP/abs/html/globbingref.html
>>
>> Sent from phone, thus brief.
>>
>> Am 17.10.2017 15:23 schrieb "mostolog--- via rsyslog"
>> >:
>>
>> Hi
>>
>>
>> One of our relay rsyslog servers has 2 kind of logs: apache and
>> java. Sadly, *not all java applications share the same log format.*
>>
>> We have solved this using a bash script to generate our config
>> which has one input for each file.
>>
>> To avoid having to reconfigure rsyslog each time a log is created
>> or an application is added/removed we though about using
>> wildcards. More or less, this would result in:
>>
>>input(
>> addMetadata="on"
>> file="/logs/apache/*/*"
>> persistStateInterval="3"
>> ruleset="ruleset"
>> tag="apache"
>> type="imfile"
>>)
>>input(
>> addMetadata="on"
>> file="/logs/java/*/*"
>> persistStateInterval="3"
>> readTimeout="5"
>> ruleset="ruleset"
>> startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
>> tag="java"
>> type="imfile"
>>)
>>
>> But, as stated before, *this would have conflicts with those files
>> having a different multiline patterns*. For example:
>>
>>/logs/java/foo/foo.log
>> startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
>>/logs/java/bar/bar.log
>> startmsg.regex="^[[:alpha:]]{3} [[:digit:]]{2},
>> [[:digit:]]{4}"
>>/logs/java/other/other.log
>>startmsg.regex="^[[:digit:]]{2}-[[:digit:]]{2}-[[:digit:]]{4}"
>>/logs/java/one/one.log
>> startmsg.regex="^[[:alpha:]]{4,5}
>>[[:digit:]]{4}/[[:digit:]]{2}/[[:digit:]]{2}"
>>/logs/java/two/two.log
>>startmsg.regex="^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}"
>>
>>
>> Is there any option to exclude this files from the wilcard?
>>
>> Couldn't wilcard accept a regex as pattern? Eg: all files except
>> those in foo, or bar, or...
>> file="/logs/java/((?!(foo|bar|other|one|two)).)*/.*"
>>
>> Any other solution?
>>
>> Regards
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> 
>> http://www.rsyslog.com/professional-services/
>> 
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you DON'T LIKE THAT.
>>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Exclude patterns for imfile whith different multiline format

[mostolog--- via rsyslog]

It doesn't seem possible to exclude specific files when using directory 
wilcards too.


eg: /logs/**/^{file_format_X.log}


Perhaps http://www.linuxjournal.com/content/bash-extended-globbing could 
do the trick, but i was wondering if there won't make more sense to use 
REGEX or exclude paratemer (or...)



Regards

On 17/10/17 16:59, Rainer Gerhards wrote:
You cannot exclude, bit you can include those that you want. I guess 
with ugly globbing that should be possible. Best try with ls first.


http://tldp.org/LDP/abs/html/globbingref.html

Sent from phone, thus brief.

Am 17.10.2017 15:23 schrieb "mostolog--- via rsyslog" 
>:


Hi


One of our relay rsyslog servers has 2 kind of logs: apache and
java. Sadly, *not all java applications share the same log format.*

We have solved this using a bash script to generate our config
which has one input for each file.

To avoid having to reconfigure rsyslog each time a log is created
or an application is added/removed we though about using
wildcards. More or less, this would result in:

   input(
        addMetadata="on"
        file="/logs/apache/*/*"
        persistStateInterval="3"
        ruleset="ruleset"
        tag="apache"
        type="imfile"
   )
   input(
        addMetadata="on"
        file="/logs/java/*/*"
        persistStateInterval="3"
        readTimeout="5"
        ruleset="ruleset"
        startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
        tag="java"
        type="imfile"
   )

But, as stated before, *this would have conflicts with those files
having a different multiline patterns*. For example:

   /logs/java/foo/foo.log
        startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
   /logs/java/bar/bar.log
        startmsg.regex="^[[:alpha:]]{3} [[:digit:]]{2},
[[:digit:]]{4}"
   /logs/java/other/other.log
   startmsg.regex="^[[:digit:]]{2}-[[:digit:]]{2}-[[:digit:]]{4}"
   /logs/java/one/one.log
        startmsg.regex="^[[:alpha:]]{4,5}
   [[:digit:]]{4}/[[:digit:]]{2}/[[:digit:]]{2}"
   /logs/java/two/two.log
   startmsg.regex="^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}"


Is there any option to exclude this files from the wilcard?

Couldn't wilcard accept a regex as pattern? Eg: all files except
those in foo, or bar, or...
    file="/logs/java/((?!(foo|bar|other|one|two)).)*/.*"

Any other solution?

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog

http://www.rsyslog.com/professional-services/

What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Missing libfastjson 0.99.7 in repository

[Cédric Jeanneret via rsyslog]

Hello,

After some more searches, apparently the package name has changed to
"libfastjson4".
But we're unable to install it, because it conflict with files installed
with "libfastjson":

Transaction check error:
  file /usr/lib64/libfastjson.so.4 from install of
libfastjson4-0.99.7-1.el7.x86_64 conflicts with file from package
libfastjson-0.99.4-2.el7.x86_64

More over, trying to remove "libfastjson" will also drop rsyslog,
rsyslog-gnutls and rsyslog-relp.

This seems to point to a wrong dependency for those packages.

Cheers,

C.

On 10/18/2017 07:04 AM, Cédric Jeanneret wrote:
> Hello,
>
> Apparently, the libfastjson 0.99.7 is missing in the redhat/centos
> repository, and this prevents rsyslog 8.30 to start when we enable relp
> protocol:
>
> rsyslogd: symbol lookup error: rsyslogd: undefined symbol:
> fjson_global_do_case_sensitive_comparison
>
> Care to add as soon as possible the right libfastjson in the repositories?
>
> Thank you in advance!
>
> Cheers,
>
> C.
>

-- 
Cédric Jeanneret
Senior Linux System Administrator
Infrastructure Solutions

Camptocamp SA
PSE-A / EPFL, 1015 Lausanne

Phone: +41 21 619 10 32
Office: +41 21 619 10 02
Email: cedric.jeanne...@camptocamp.com




signature.asc
Description: OpenPGP digital signature
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.