Re: [rsyslog] preserving metadata on message split

2017-11-09 Thread Scot Kreienkamp 
Yep, Rainer confirmed that I can go into the megabytes in size; I had mine set 
to 128k as I had read somewhere in the mail archive that was the max.  Or maybe 
it was the max for RELP, not sure as that was at least a year ago.  Either way, 
I don't use RELP so I'm at 4 megs now and it shouldn't be a problem anymore.

Thanks!


Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
-Original Message-
From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Thursday, November 9, 2017 3:20 PM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] preserving metadata on message split

increase the max message size so that the messages don't get split?

what is the source of the messages? (TCP, UDP, imfile, etc)

David Lang

On Thu, 9 Nov 2017, Scot Kreienkamp wrote:

> Hi David,
>
> Any ideas on any way to get around this if there's no way to preserve the 
> metadata on an oversized message?
>
>
> Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
> One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
> Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
> -Original Message-
> From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David 
> Lang
> Sent: Monday, November 6, 2017 4:49 PM
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] preserving metadata on message split
>
> No, because when the message hits the max size, the processing of the next
> message (which is a fragment) is completely independent of the first message.
>
> now, if this is a message where the metadata should be available in processing
> the second fragment, I agree that it should be available.
>
> can you give an example of a message where the metadata is lost?
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
>
> This message is intended only for the individual or entity to which it is 
> addressed.  It may contain privileged, confidential information which is 
> exempt from disclosure under applicable laws.  If you are not the intended 
> recipient, you are strictly prohibited from disseminating or distributing 
> this information (other than to the intended recipient) or copying this 
> information.  If you have received this communication in error, please notify 
> us immediately by e-mail or by telephone at the above number. Thank you.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] preserving metadata on message split

2017-11-09 Thread David Lang 

On Thu, 9 Nov 2017, Scot Kreienkamp wrote:

I have no plans to use RELP so I'll try setting it to 2 meg then.  That should 
give it plenty of overhead, more than enough to resolve it for now.  I have no 
control over the message size, it's whatever Oracle spits out.


what you can do is have Oracle deliver the logs to a local copy of rsyslog, and 
have that copy check for max length and 'do something' with it (which could be 
to send it to an external file that splits the logs and feeds the results back 
into syslog)


David Lang


The limits for rsyslog and relp would be good info to add to the docu.

Thanks!


Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
-Original Message-
From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer 
Gerhards
Sent: Thursday, November 9, 2017 11:27 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] preserving metadata on message split

larger 128k currently seems to be a problem when RELP is used.
Otherwise, we have configs which actually have set it to a couple of
MB (and occasionally use it).

Rainer

2017-11-09 17:25 GMT+01:00 deoren
<rsyslog-users-lists.adiscon@whyaskwhy.org>:

On 11/9/2017 10:24 AM, Scot Kreienkamp wrote:


I have it set at 128k now... I thought I read in the list archives that
was the maximium value?



https://github.com/rsyslog/rsyslog/issues/1741

Looks like it (for now).


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

This message is intended only for the individual or entity to which it is 
addressed.  It may contain privileged, confidential information which is exempt 
from disclosure under applicable laws.  If you are not the intended recipient, 
you are strictly prohibited from disseminating or distributing this information 
(other than to the intended recipient) or copying this information.  If you 
have received this communication in error, please notify us immediately by 
e-mail or by telephone at the above number. Thank you.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] preserving metadata on message split

2017-11-09 Thread David Lang 

increase the max message size so that the messages don't get split?

what is the source of the messages? (TCP, UDP, imfile, etc)

David Lang

On Thu, 9 Nov 2017, Scot Kreienkamp wrote:


Hi David,

Any ideas on any way to get around this if there's no way to preserve the 
metadata on an oversized message?


Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
-Original Message-
From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Monday, November 6, 2017 4:49 PM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] preserving metadata on message split

No, because when the message hits the max size, the processing of the next
message (which is a fragment) is completely independent of the first message.

now, if this is a message where the metadata should be available in processing
the second fragment, I agree that it should be available.

can you give an example of a message where the metadata is lost?

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

This message is intended only for the individual or entity to which it is 
addressed.  It may contain privileged, confidential information which is exempt 
from disclosure under applicable laws.  If you are not the intended recipient, 
you are strictly prohibited from disseminating or distributing this information 
(other than to the intended recipient) or copying this information.  If you 
have received this communication in error, please notify us immediately by 
e-mail or by telephone at the above number. Thank you.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] preserving metadata on message split

2017-11-09 Thread deoren 
The current documentation[1] mentions that 8K is the default value for 
the maxmessagesize global parameter.


The documentation[2] for imrelp mentions that the 'MaxDataSize' 
parameter defaults to the global message size, though it isn't 100% 
clear to me whether that is referring to the maxmessagesize global 
parameter.


I suspect that the RELP limits will be better documented once the bugs 
noted here[3] and here[4] are worked out. I have asked for clarification 
re the link between those two parameters (imrelp's MaxDataSize and 
global() maxMessageSize) and plan to offer a PR for doc updates once 
that clarification is given.



[1] http://www.rsyslog.com/doc/v8-stable/rainerscript/global.html

[2] http://www.rsyslog.com/doc/v8-stable/configuration/modules/imrelp.html

[3] https://github.com/rsyslog/rsyslog/issues/1741

[4] https://github.com/rsyslog/rsyslog/issues/1782



On 11/9/2017 10:33 AM, Scot Kreienkamp wrote:

I have no plans to use RELP so I'll try setting it to 2 meg then.  That should 
give it plenty of overhead, more than enough to resolve it for now.  I have no 
control over the message size, it's whatever Oracle spits out.

The limits for rsyslog and relp would be good info to add to the docu.

Thanks!


Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
-Original Message-
From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer 
Gerhards
Sent: Thursday, November 9, 2017 11:27 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] preserving metadata on message split

larger 128k currently seems to be a problem when RELP is used.
Otherwise, we have configs which actually have set it to a couple of
MB (and occasionally use it).

Rainer

2017-11-09 17:25 GMT+01:00 deoren
<rsyslog-users-lists.adiscon@whyaskwhy.org>:

On 11/9/2017 10:24 AM, Scot Kreienkamp wrote:


I have it set at 128k now... I thought I read in the list archives that
was the maximium value?



https://github.com/rsyslog/rsyslog/issues/1741


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] preserving metadata on message split

2017-11-09 Thread Rainer Gerhards 
2017-11-09 17:33 GMT+01:00 Scot Kreienkamp <scot.kreienk...@la-z-boy.com>:
> I have no plans to use RELP so I'll try setting it to 2 meg then.  That 
> should give it plenty of overhead, more than enough to resolve it for now.  I 
> have no control over the message size, it's whatever Oracle spits out.
>
> The limits for rsyslog

rsyslog actually has no limit (albeit I know nobody who tested with
gigabytes of message size ;-)

>and relp would be good info to add to the docu.
relp is probably a bug.

Rainer
>
> Thanks!
>
>
> Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
> One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
> Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
> -Original Message-
> From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer 
> Gerhards
> Sent: Thursday, November 9, 2017 11:27 AM
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] preserving metadata on message split
>
> larger 128k currently seems to be a problem when RELP is used.
> Otherwise, we have configs which actually have set it to a couple of
> MB (and occasionally use it).
>
> Rainer
>
> 2017-11-09 17:25 GMT+01:00 deoren
> <rsyslog-users-lists.adiscon@whyaskwhy.org>:
>> On 11/9/2017 10:24 AM, Scot Kreienkamp wrote:
>>>
>>> I have it set at 128k now... I thought I read in the list archives that
>>> was the maximium value?
>>
>>
>> https://github.com/rsyslog/rsyslog/issues/1741
>>
>> Looks like it (for now).
>>
>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
>
> This message is intended only for the individual or entity to which it is 
> addressed.  It may contain privileged, confidential information which is 
> exempt from disclosure under applicable laws.  If you are not the intended 
> recipient, you are strictly prohibited from disseminating or distributing 
> this information (other than to the intended recipient) or copying this 
> information.  If you have received this communication in error, please notify 
> us immediately by e-mail or by telephone at the above number. Thank you.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] preserving metadata on message split

2017-11-09 Thread Scot Kreienkamp 
I have no plans to use RELP so I'll try setting it to 2 meg then.  That should 
give it plenty of overhead, more than enough to resolve it for now.  I have no 
control over the message size, it's whatever Oracle spits out.

The limits for rsyslog and relp would be good info to add to the docu.

Thanks!


Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
-Original Message-
From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer 
Gerhards
Sent: Thursday, November 9, 2017 11:27 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] preserving metadata on message split

larger 128k currently seems to be a problem when RELP is used.
Otherwise, we have configs which actually have set it to a couple of
MB (and occasionally use it).

Rainer

2017-11-09 17:25 GMT+01:00 deoren
<rsyslog-users-lists.adiscon@whyaskwhy.org>:
> On 11/9/2017 10:24 AM, Scot Kreienkamp wrote:
>>
>> I have it set at 128k now... I thought I read in the list archives that
>> was the maximium value?
>
>
> https://github.com/rsyslog/rsyslog/issues/1741
>
> Looks like it (for now).
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

This message is intended only for the individual or entity to which it is 
addressed.  It may contain privileged, confidential information which is exempt 
from disclosure under applicable laws.  If you are not the intended recipient, 
you are strictly prohibited from disseminating or distributing this information 
(other than to the intended recipient) or copying this information.  If you 
have received this communication in error, please notify us immediately by 
e-mail or by telephone at the above number. Thank you.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] preserving metadata on message split

2017-11-09 Thread Rainer Gerhards 
larger 128k currently seems to be a problem when RELP is used.
Otherwise, we have configs which actually have set it to a couple of
MB (and occasionally use it).

Rainer

2017-11-09 17:25 GMT+01:00 deoren
:
> On 11/9/2017 10:24 AM, Scot Kreienkamp wrote:
>>
>> I have it set at 128k now... I thought I read in the list archives that
>> was the maximium value?
>
>
> https://github.com/rsyslog/rsyslog/issues/1741
>
> Looks like it (for now).
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] preserving metadata on message split

2017-11-09 Thread deoren 

On 11/9/2017 10:24 AM, Scot Kreienkamp wrote:

I have it set at 128k now... I thought I read in the list archives that was the 
maximium value?


https://github.com/rsyslog/rsyslog/issues/1741

Looks like it (for now).

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] preserving metadata on message split

2017-11-09 Thread Scot Kreienkamp 
I have it set at 128k now... I thought I read in the list archives that was the 
maximium value?


Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
-Original Message-
From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of Rainer 
Gerhards
Sent: Thursday, November 9, 2017 10:00 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] preserving metadata on message split

2017-11-09 14:46 GMT+01:00 Scot Kreienkamp <scot.kreienk...@la-z-boy.com>:
> Hi David,
>
> Any ideas on any way to get around this if there's no way to preserve the 
> metadata on an oversized message?

You need to increase the message size. It's a simple config parmater
[global(maxmessagesize="xx")] I mean.

Rainer
>
>
> Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
> One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
> Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
> -Original Message-
> From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David 
> Lang
> Sent: Monday, November 6, 2017 4:49 PM
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] preserving metadata on message split
>
> No, because when the message hits the max size, the processing of the next
> message (which is a fragment) is completely independent of the first message.
>
> now, if this is a message where the metadata should be available in processing
> the second fragment, I agree that it should be available.
>
> can you give an example of a message where the metadata is lost?
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
>
> This message is intended only for the individual or entity to which it is 
> addressed.  It may contain privileged, confidential information which is 
> exempt from disclosure under applicable laws.  If you are not the intended 
> recipient, you are strictly prohibited from disseminating or distributing 
> this information (other than to the intended recipient) or copying this 
> information.  If you have received this communication in error, please notify 
> us immediately by e-mail or by telephone at the above number. Thank you.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] preserving metadata on message split

2017-11-09 Thread Scot Kreienkamp 
Nope, just using TCP as the transport, nothing additional like RELP.


Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
-Original Message-
From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of deoren
Sent: Thursday, November 9, 2017 10:07 AM
To: rsyslog@lists.adiscon.com
Subject: Re: [rsyslog] preserving metadata on message split

On 11/9/2017 9:00 AM, Rainer Gerhards wrote:
> 2017-11-09 14:46 GMT+01:00 Scot Kreienkamp <scot.kreienk...@la-z-boy.com>:
>> Hi David,
>>
>> Any ideas on any way to get around this if there's no way to preserve the 
>> metadata on an oversized message?
>
> You need to increase the message size. It's a simple config parmater
> [global(maxmessagesize="xx")] I mean.
>
> Rainer

Scot,

I didn't see in your previous posts whether you're using RELP for
sending/receiving messages, but just in case you, are please be aware
that there appears to be a bug where imrelp does not use the
maxmessagesize value. Instead, you will have to set imrelp's MaxDataSize
parameter value (in my case equal to maxmessagesize).

More info:

https://github.com/rsyslog/rsyslog/issues/1782#issuecomment-339124192

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

This message is intended only for the individual or entity to which it is 
addressed.  It may contain privileged, confidential information which is exempt 
from disclosure under applicable laws.  If you are not the intended recipient, 
you are strictly prohibited from disseminating or distributing this information 
(other than to the intended recipient) or copying this information.  If you 
have received this communication in error, please notify us immediately by 
e-mail or by telephone at the above number. Thank you.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] preserving metadata on message split

2017-11-09 Thread Rainer Gerhards 
2017-11-09 14:46 GMT+01:00 Scot Kreienkamp <scot.kreienk...@la-z-boy.com>:
> Hi David,
>
> Any ideas on any way to get around this if there's no way to preserve the 
> metadata on an oversized message?

You need to increase the message size. It's a simple config parmater
[global(maxmessagesize="xx")] I mean.

Rainer
>
>
> Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
> One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
> Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
> -Original Message-
> From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David 
> Lang
> Sent: Monday, November 6, 2017 4:49 PM
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] preserving metadata on message split
>
> No, because when the message hits the max size, the processing of the next
> message (which is a fragment) is completely independent of the first message.
>
> now, if this is a message where the metadata should be available in processing
> the second fragment, I agree that it should be available.
>
> can you give an example of a message where the metadata is lost?
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
>
> This message is intended only for the individual or entity to which it is 
> addressed.  It may contain privileged, confidential information which is 
> exempt from disclosure under applicable laws.  If you are not the intended 
> recipient, you are strictly prohibited from disseminating or distributing 
> this information (other than to the intended recipient) or copying this 
> information.  If you have received this communication in error, please notify 
> us immediately by e-mail or by telephone at the above number. Thank you.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] preserving metadata on message split

2017-11-09 Thread Scot Kreienkamp 
Hi David,

Any ideas on any way to get around this if there's no way to preserve the 
metadata on an oversized message?


Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
-Original Message-
From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Monday, November 6, 2017 4:49 PM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] preserving metadata on message split

No, because when the message hits the max size, the processing of the next
message (which is a fragment) is completely independent of the first message.

now, if this is a message where the metadata should be available in processing
the second fragment, I agree that it should be available.

can you give an example of a message where the metadata is lost?

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

This message is intended only for the individual or entity to which it is 
addressed.  It may contain privileged, confidential information which is exempt 
from disclosure under applicable laws.  If you are not the intended recipient, 
you are strictly prohibited from disseminating or distributing this information 
(other than to the intended recipient) or copying this information.  If you 
have received this communication in error, please notify us immediately by 
e-mail or by telephone at the above number. Thank you.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] preserving metadata on message split

2017-11-06 Thread Scot Kreienkamp 
I have lots of them on my system but I can't give the individual message, just 
the results of the metadata not being present.

Here's my template example again:

template (name="messages" type="string" 
string="/opt/syslog/%HOSTNAME%/messages-%$YEAR%-%$MONTH%-%$DAY%")

Following the template, this is an ls -l in /opt/syslog:

drwxr-xr-x 2 root root   32 Nov  6 10:00 15099804145432
drwxr-xr-x 2 root root   32 Nov  6 11:00 15099840135432
drwxr-xr-x 2 root root   32 Nov  6 12:00 15099876035432
drwxr-xr-x 2 root root   32 Nov  6 12:00 15099876135432
drwxr-xr-x 2 root root   40 Nov  6 12:01 15099876755432
drwxr-xr-x 2 root root   40 Nov  6 13:00 15099912135432
drwxr-xr-x 2 root root   40 Nov  6 14:00 15099948155432
drwxr-xr-x 2 root root   40 Nov  6 15:00 15099984135432
drwxr-xr-x 2 root root   32 Nov  6 16:00 15100020135432
drwxr-xr-x 2 root root   40 Nov  6 17:00 15100056185432
drwxr-xr-x 3 root root 4096 Nov  6 00:55 corpvely.na.lzb.hq
drwxr-xr-x 2 root root 4096 Nov  6 00:55 corpvskreienfs1.na.lzb.hq
drwxr-xr-x 2 root root 4096 Nov  6 00:55 corpvskreienfs2.na.lzb.hq
drwxr-xr-x 3 root root 4096 Nov  6 00:55 corpvskreienl.na.lzb.hq
drwxr-xr-x 2 root root 4096 Nov  6 00:55 dayaglecpd.na.lzb.hq
drwxr-xr-x 2 root root 4096 Nov  6 00:55 dayaglpd01.na.lzb.hq
drwxr-xr-x 2 root root 4096 Nov  6 00:55 dayaglpd02.na.lzb.hq
drwxr-xr-x 2 root root   32 Oct 31 09:01 sax
drwxr-xr-x 2 root root   40 Oct 31 09:02 schema

And the contents of one directory:
[skreien@monvsyslog 15099876135432]$ cat messages-2017-11-06
Nov  6 12:00:16 15099876135432

Anything that ends in .na.lzb.hq is correct, anything that isn't is picking up 
the value from a split message.  I know this because I get messages in my 
system logs about messages being split.  Underneath all the incorrect folder 
names is a messages file following the template correctly, but because the 
HOSTNAME metadata is not available it tries to pick it up from the fragment 
that's left since it's processing it as a separate message, so of course it's 
wrong.



Scot Kreienkamp  | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162 |  Office: 734-384-6403 |  |  
Mobile: 7349151444 | Email: scot.kreienk...@la-z-boy.com
-Original Message-
From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Monday, November 6, 2017 4:49 PM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] preserving metadata on message split

No, because when the message hits the max size, the processing of the next
message (which is a fragment) is completely independent of the first message.

now, if this is a message where the metadata should be available in processing
the second fragment, I agree that it should be available.

can you give an example of a message where the metadata is lost?

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

This message is intended only for the individual or entity to which it is 
addressed.  It may contain privileged, confidential information which is exempt 
from disclosure under applicable laws.  If you are not the intended recipient, 
you are strictly prohibited from disseminating or distributing this information 
(other than to the intended recipient) or copying this information.  If you 
have received this communication in error, please notify us immediately by 
e-mail or by telephone at the above number. Thank you.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] preserving metadata on message split

2017-11-06 Thread David Lang 
No, because when the message hits the max size, the processing of the next 
message (which is a fragment) is completely independent of the first message.


now, if this is a message where the metadata should be available in processing 
the second fragment, I agree that it should be available.


can you give an example of a message where the metadata is lost?

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] preserving metadata on message split

2017-11-06 Thread Scot Kreienkamp 
Hey all,

When messages are split due to size the metadata seems to go with the first 
part of the message.  I'm doing hostname based logging, so when a message is 
split the metadata it needs to log the rest of the message to the correct file 
is gone.  Is there any way to preserve that metadata, so the rest of the 
message is appended to the same logfile as the first part?

Here's my file template:
template (name="DailyPerHostDebugAll" type="string" 
string="/opt/syslog/%HOSTNAME%/debug-%$YEAR%-%$MONTH%-%$DAY%")

Thanks!

Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Corporate
One La-Z-Boy Drive | Monroe, Michigan 48162  | * 734-384-6403 | | * 7349151444 
| *  scot.kreienk...@la-z-boy.com
www.la-z-boy.com | 
facebook.com/lazboy
 | twitter.com/lazboy | 
youtube.com/lazboy

[cid:lzbVertical_hres.jpg]



This message is intended only for the individual or entity to which it is 
addressed.  It may contain privileged, confidential information which is exempt 
from disclosure under applicable laws.  If you are not the intended recipient, 
you are strictly prohibited from disseminating or distributing this information 
(other than to the intended recipient) or copying this information.  If you 
have received this communication in error, please notify us immediately by 
e-mail or by telephone at the above number. Thank you.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.