Re: [rsyslog] another dyn_stats question

[singh.janmejay]

Cool, I'll make dyn-stats bucket resettable flag the only thing that
controls resetting of its accumulators. It will or won't reset regardless
of impstats reset flag value.

This is again changing behavior from previous release but is a fairly minor
thing, so we'll just document it and not worry about backward compatibility
of its behavior.
On Mar 30, 2016 2:55 AM, "David Lang"  wrote:

> On Wed, 30 Mar 2016, singh.janmejay wrote:
>
> Metric owner is the subsystem (action is a perfect example). End-user is
>> author of the config.
>>
>
> Except that dyn_stats doen't exist except in the config, so it's the
> "End-user" who sets everything about them.
>
> In this case ideal situation would be to let action decide which of its
>> metrics are resettable, and let end-user decide which of the resettable
>> metric accumulators(subset of all metrics) do they actually want to reset.
>>
>
> I disagree, the module author should not be deciding if something is
> resettable or not.
>
> The module author is either reporting the current state of something (like
> queue length) where there is no resetting, or they are reporting the state
> of a counter (number of messages processed), and it should be up to the
> config author if they want to reset this every reporting period or not.
>
> In case of dyn-stats, end-user is metric-owner.
>>
>> So we should probably disassociate from impstats reset flag and let
>> bucket-level resettable flag be the only way to control it.
>>
>> That is what you expected the behavior would be, right? So it's more
>> intuitive too.
>>
>> Should we do this?
>>
>
> I think so.
>
> For static counters though, the problem still remains. If you want to reset
>> omkafka resettable counters, but don't want resetting for imudp resettable
>> counters, as of now we have no way to configure it like that.
>>
>
> Well, I see that as a legacy interface, so I don't think we need to worry
> about changing that.
>
> Monitoring systems are almost always able to handle counters and convert
> them to rate-of-change.
>
> resetting a counter is always racy with updates.
>
> If a monitoring system is looking at the stats, you almost always want
> them to be non-reset counters. It's only if a person is looking at them
> that you want to show the delta since the last report in the output.
>
> David Lang
>
> On Mar 30, 2016 2:06 AM, "David Lang"  wrote:
>>
>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>>
>>> The problem is, pstats level control is an all or nothing kind of
>>>
 arrangement.

 Ideal setup would be to let metric-owner decide if metric should be
 resettable or not(which allows choosing if its a guage or not) and
 then let end-user choose if they really want to reset each resettable
 metric(counter) (or do they want it as a perpetual counter).


>>> there is only one entity in control of the rsyslog.conf file, how do you
>>> get to 'metric-owner' vs 'end-user'?
>>>
>>> David Lang
>>>
>>> In our case, the end-user control is very coarse-grained, where one
>>>
 can either get all counters as perpetual or all as resetting.

 On Wed, Mar 30, 2016 at 1:41 AM, David Lang  wrote:

 On Wed, 30 Mar 2016, singh.janmejay wrote:
>
> Consider a queue-size, its not really a counter, its a guage. It will
>
>> provide point-in-time value of size of the queue. Resetting such a
>> value will make no sense. Some static metrics we report are guages (so
>> subsystem supporting the metric must identify it to be resettable or
>> not).
>>
>>
>
> but that won't work well with the current policy.
>
> you can't have anything as a gauge unless you eliminate the possiblity
> for
> any of the pstats data to be gauges.
>
> If pstats doesn't reset values, then dyn_stats can't implement gauges.
> If
> pststs does reset values, you have turned a lot of things that probably
> shouldn't reset into gauges.
>
> it would make far more sense to define this purely on a per-stats
> basis.
>
> pstats has some items that are always gauges (size/maxsize/etc) and
> other
> items that are allowed to be treated as gauges or as counters.
>
> In my case, I don't want to reset those things (submitted/enqueued)
> that
> can
> be counters, but that prevents me from having dyn_stats items that are
> gauges.
>
> David Lang
>
>
> Yes, restart is a problem. But restarts are really in-frequent enough
>
>> to not matter. When they do happen, dx/dt chart shows a big dip, but
>> people still do track perpetual counters.
>>
>> Actually its only dyn_inc (we don't have a dyn_dec). But as we go
>> forward, we may expose capability to support guages and may manipulate
>> it with dyn_inc and dyn_dec.
>>
>> Resettable flag in dyn-stats bucket does the same thing as
>> resettable-flag in 

Re: [rsyslog] another dyn_stats question

[David Lang]

On Wed, 30 Mar 2016, singh.janmejay wrote:


Metric owner is the subsystem (action is a perfect example). End-user is
author of the config.


Except that dyn_stats doen't exist except in the config, so it's the "End-user" 
who sets everything about them.



In this case ideal situation would be to let action decide which of its
metrics are resettable, and let end-user decide which of the resettable
metric accumulators(subset of all metrics) do they actually want to reset.


I disagree, the module author should not be deciding if something is resettable 
or not.


The module author is either reporting the current state of something (like queue 
length) where there is no resetting, or they are reporting the state of a 
counter (number of messages processed), and it should be up to the config author 
if they want to reset this every reporting period or not.



In case of dyn-stats, end-user is metric-owner.

So we should probably disassociate from impstats reset flag and let
bucket-level resettable flag be the only way to control it.

That is what you expected the behavior would be, right? So it's more
intuitive too.

Should we do this?


I think so.


For static counters though, the problem still remains. If you want to reset
omkafka resettable counters, but don't want resetting for imudp resettable
counters, as of now we have no way to configure it like that.


Well, I see that as a legacy interface, so I don't think we need to worry about 
changing that.


Monitoring systems are almost always able to handle counters and convert them to 
rate-of-change.


resetting a counter is always racy with updates.

If a monitoring system is looking at the stats, you almost always want them to 
be non-reset counters. It's only if a person is looking at them that you want to 
show the delta since the last report in the output.


David Lang


On Mar 30, 2016 2:06 AM, "David Lang"  wrote:


On Wed, 30 Mar 2016, singh.janmejay wrote:

The problem is, pstats level control is an all or nothing kind of

arrangement.

Ideal setup would be to let metric-owner decide if metric should be
resettable or not(which allows choosing if its a guage or not) and
then let end-user choose if they really want to reset each resettable
metric(counter) (or do they want it as a perpetual counter).



there is only one entity in control of the rsyslog.conf file, how do you
get to 'metric-owner' vs 'end-user'?

David Lang

In our case, the end-user control is very coarse-grained, where one

can either get all counters as perpetual or all as resetting.

On Wed, Mar 30, 2016 at 1:41 AM, David Lang  wrote:


On Wed, 30 Mar 2016, singh.janmejay wrote:

Consider a queue-size, its not really a counter, its a guage. It will

provide point-in-time value of size of the queue. Resetting such a
value will make no sense. Some static metrics we report are guages (so
subsystem supporting the metric must identify it to be resettable or
not).




but that won't work well with the current policy.

you can't have anything as a gauge unless you eliminate the possiblity
for
any of the pstats data to be gauges.

If pstats doesn't reset values, then dyn_stats can't implement gauges. If
pststs does reset values, you have turned a lot of things that probably
shouldn't reset into gauges.

it would make far more sense to define this purely on a per-stats basis.

pstats has some items that are always gauges (size/maxsize/etc) and other
items that are allowed to be treated as gauges or as counters.

In my case, I don't want to reset those things (submitted/enqueued) that
can
be counters, but that prevents me from having dyn_stats items that are
gauges.

David Lang


Yes, restart is a problem. But restarts are really in-frequent enough

to not matter. When they do happen, dx/dt chart shows a big dip, but
people still do track perpetual counters.

Actually its only dyn_inc (we don't have a dyn_dec). But as we go
forward, we may expose capability to support guages and may manipulate
it with dyn_inc and dyn_dec.

Resettable flag in dyn-stats bucket does the same thing as
resettable-flag in statsobj static-counter.

On Tue, Mar 29, 2016 at 3:13 AM, David Lang  wrote:



On Tue, 29 Mar 2016, singh.janmejay wrote:

Well, from counter pov, some counters by nature need to be resettable,

some are actually guages (so resetting makes little sense) and some
are perpetual (so again resetting is not the right thing to do).





can you explain the different categories better? I don't see how any
stats
can be 'by nature' one form or another, since it's possible to convert
the
data from one form to another.

The idea of perpetual counters doesn't work if the contents of the
counters
are lost at restart.

It could be that I didn't read the documentation well enough and missed
references to ways to manipulate them besides dyn_inc and dyn_dec

David Lang


A global control needs to be at impstats level because some users may

want to access data 

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

Metric owner is the subsystem (action is a perfect example). End-user is
author of the config.

In this case ideal situation would be to let action decide which of its
metrics are resettable, and let end-user decide which of the resettable
metric accumulators(subset of all metrics) do they actually want to reset.

In case of dyn-stats, end-user is metric-owner.

So we should probably disassociate from impstats reset flag and let
bucket-level resettable flag be the only way to control it.

That is what you expected the behavior would be, right? So it's more
intuitive too.

Should we do this?

For static counters though, the problem still remains. If you want to reset
omkafka resettable counters, but don't want resetting for imudp resettable
counters, as of now we have no way to configure it like that.
On Mar 30, 2016 2:06 AM, "David Lang"  wrote:

> On Wed, 30 Mar 2016, singh.janmejay wrote:
>
> The problem is, pstats level control is an all or nothing kind of
>> arrangement.
>>
>> Ideal setup would be to let metric-owner decide if metric should be
>> resettable or not(which allows choosing if its a guage or not) and
>> then let end-user choose if they really want to reset each resettable
>> metric(counter) (or do they want it as a perpetual counter).
>>
>
> there is only one entity in control of the rsyslog.conf file, how do you
> get to 'metric-owner' vs 'end-user'?
>
> David Lang
>
> In our case, the end-user control is very coarse-grained, where one
>> can either get all counters as perpetual or all as resetting.
>>
>> On Wed, Mar 30, 2016 at 1:41 AM, David Lang  wrote:
>>
>>> On Wed, 30 Mar 2016, singh.janmejay wrote:
>>>
>>> Consider a queue-size, its not really a counter, its a guage. It will
 provide point-in-time value of size of the queue. Resetting such a
 value will make no sense. Some static metrics we report are guages (so
 subsystem supporting the metric must identify it to be resettable or
 not).

>>>
>>>
>>> but that won't work well with the current policy.
>>>
>>> you can't have anything as a gauge unless you eliminate the possiblity
>>> for
>>> any of the pstats data to be gauges.
>>>
>>> If pstats doesn't reset values, then dyn_stats can't implement gauges. If
>>> pststs does reset values, you have turned a lot of things that probably
>>> shouldn't reset into gauges.
>>>
>>> it would make far more sense to define this purely on a per-stats basis.
>>>
>>> pstats has some items that are always gauges (size/maxsize/etc) and other
>>> items that are allowed to be treated as gauges or as counters.
>>>
>>> In my case, I don't want to reset those things (submitted/enqueued) that
>>> can
>>> be counters, but that prevents me from having dyn_stats items that are
>>> gauges.
>>>
>>> David Lang
>>>
>>>
>>> Yes, restart is a problem. But restarts are really in-frequent enough
 to not matter. When they do happen, dx/dt chart shows a big dip, but
 people still do track perpetual counters.

 Actually its only dyn_inc (we don't have a dyn_dec). But as we go
 forward, we may expose capability to support guages and may manipulate
 it with dyn_inc and dyn_dec.

 Resettable flag in dyn-stats bucket does the same thing as
 resettable-flag in statsobj static-counter.

 On Tue, Mar 29, 2016 at 3:13 AM, David Lang  wrote:

>
> On Tue, 29 Mar 2016, singh.janmejay wrote:
>
> Well, from counter pov, some counters by nature need to be resettable,
>> some are actually guages (so resetting makes little sense) and some
>> are perpetual (so again resetting is not the right thing to do).
>>
>
>
>
> can you explain the different categories better? I don't see how any
> stats
> can be 'by nature' one form or another, since it's possible to convert
> the
> data from one form to another.
>
> The idea of perpetual counters doesn't work if the contents of the
> counters
> are lost at restart.
>
> It could be that I didn't read the documentation well enough and missed
> references to ways to manipulate them besides dyn_inc and dyn_dec
>
> David Lang
>
>
> A global control needs to be at impstats level because some users may
>> want to access data in rate-of-change (dx/dt) form rather than
>> absolute value form (it can be derived by adding up the numbers and
>> studying the slope, but its prone to error when some messages are
>> dropped etc). So user needs to have a say in how counters should be
>> reported, should they be reset or not.
>>
>> Same goes for dyn-stats. User may want a perpetual counter for one
>> dyn-stats bucket (regardless of other buckets being configured to
>> reset), so its important to allow a particular bucket to be configured
>> as non-resetting.
>>
>> However, if user chooses to view all the counters in cumulative form,
>> the same 

Re: [rsyslog] another dyn_stats question

[David Lang]

On Wed, 30 Mar 2016, singh.janmejay wrote:


The problem is, pstats level control is an all or nothing kind of arrangement.

Ideal setup would be to let metric-owner decide if metric should be
resettable or not(which allows choosing if its a guage or not) and
then let end-user choose if they really want to reset each resettable
metric(counter) (or do they want it as a perpetual counter).


there is only one entity in control of the rsyslog.conf file, how do you get to 
'metric-owner' vs 'end-user'?


David Lang


In our case, the end-user control is very coarse-grained, where one
can either get all counters as perpetual or all as resetting.

On Wed, Mar 30, 2016 at 1:41 AM, David Lang  wrote:

On Wed, 30 Mar 2016, singh.janmejay wrote:


Consider a queue-size, its not really a counter, its a guage. It will
provide point-in-time value of size of the queue. Resetting such a
value will make no sense. Some static metrics we report are guages (so
subsystem supporting the metric must identify it to be resettable or
not).



but that won't work well with the current policy.

you can't have anything as a gauge unless you eliminate the possiblity for
any of the pstats data to be gauges.

If pstats doesn't reset values, then dyn_stats can't implement gauges. If
pststs does reset values, you have turned a lot of things that probably
shouldn't reset into gauges.

it would make far more sense to define this purely on a per-stats basis.

pstats has some items that are always gauges (size/maxsize/etc) and other
items that are allowed to be treated as gauges or as counters.

In my case, I don't want to reset those things (submitted/enqueued) that can
be counters, but that prevents me from having dyn_stats items that are
gauges.

David Lang



Yes, restart is a problem. But restarts are really in-frequent enough
to not matter. When they do happen, dx/dt chart shows a big dip, but
people still do track perpetual counters.

Actually its only dyn_inc (we don't have a dyn_dec). But as we go
forward, we may expose capability to support guages and may manipulate
it with dyn_inc and dyn_dec.

Resettable flag in dyn-stats bucket does the same thing as
resettable-flag in statsobj static-counter.

On Tue, Mar 29, 2016 at 3:13 AM, David Lang  wrote:


On Tue, 29 Mar 2016, singh.janmejay wrote:


Well, from counter pov, some counters by nature need to be resettable,
some are actually guages (so resetting makes little sense) and some
are perpetual (so again resetting is not the right thing to do).




can you explain the different categories better? I don't see how any
stats
can be 'by nature' one form or another, since it's possible to convert
the
data from one form to another.

The idea of perpetual counters doesn't work if the contents of the
counters
are lost at restart.

It could be that I didn't read the documentation well enough and missed
references to ways to manipulate them besides dyn_inc and dyn_dec

David Lang



A global control needs to be at impstats level because some users may
want to access data in rate-of-change (dx/dt) form rather than
absolute value form (it can be derived by adding up the numbers and
studying the slope, but its prone to error when some messages are
dropped etc). So user needs to have a say in how counters should be
reported, should they be reset or not.

Same goes for dyn-stats. User may want a perpetual counter for one
dyn-stats bucket (regardless of other buckets being configured to
reset), so its important to allow a particular bucket to be configured
as non-resetting.

However, if user chooses to view all the counters in cumulative form,
the same config that works for static-counters should work for
dyn-stats counters too (so impstats flag is relevant as well).

And'ing is necessary because it doesn't make sense to reset a
by-nature non-resettable counter.


On Tue, Mar 29, 2016 at 12:25 AM, David Lang  wrote:



What is the thinking in having two switches that get ANDed together for
resetting things? I could see two independent controls, or I could see
the
pstats setting controlling everything, but I don't understand two
getting
ANDed together.

David Lang


On Tue, 29 Mar 2016, singh.janmejay wrote:


Will add this to the documentation when I touch it for
output-restructuring.

On Tue, Mar 29, 2016 at 12:11 AM, David Lang  wrote:




ahh, that's not at all clear from the documentation.

I turned off resetting the counters at the pstats level because I was
getting soem errors/coredumps from them (quite a few months ago, so
it
may
be solved now) I'm currently getting >100 lines of pstats output now,
before
I add dyn_stats to things.

I'll wait for the restructuring of the dyn_stats output and the new
foreach
and then try again.

David Lang

On Mon, 28 Mar 2016, singh.janmejay wrote:


Date: Mon, 28 Mar 2016 23:59:46 +0530

From: singh.janmejay 
Reply-To: rsyslog-users 
To: 

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

The problem is, pstats level control is an all or nothing kind of arrangement.

Ideal setup would be to let metric-owner decide if metric should be
resettable or not(which allows choosing if its a guage or not) and
then let end-user choose if they really want to reset each resettable
metric(counter) (or do they want it as a perpetual counter).

In our case, the end-user control is very coarse-grained, where one
can either get all counters as perpetual or all as resetting.

On Wed, Mar 30, 2016 at 1:41 AM, David Lang  wrote:
> On Wed, 30 Mar 2016, singh.janmejay wrote:
>
>> Consider a queue-size, its not really a counter, its a guage. It will
>> provide point-in-time value of size of the queue. Resetting such a
>> value will make no sense. Some static metrics we report are guages (so
>> subsystem supporting the metric must identify it to be resettable or
>> not).
>
>
> but that won't work well with the current policy.
>
> you can't have anything as a gauge unless you eliminate the possiblity for
> any of the pstats data to be gauges.
>
> If pstats doesn't reset values, then dyn_stats can't implement gauges. If
> pststs does reset values, you have turned a lot of things that probably
> shouldn't reset into gauges.
>
> it would make far more sense to define this purely on a per-stats basis.
>
> pstats has some items that are always gauges (size/maxsize/etc) and other
> items that are allowed to be treated as gauges or as counters.
>
> In my case, I don't want to reset those things (submitted/enqueued) that can
> be counters, but that prevents me from having dyn_stats items that are
> gauges.
>
> David Lang
>
>
>> Yes, restart is a problem. But restarts are really in-frequent enough
>> to not matter. When they do happen, dx/dt chart shows a big dip, but
>> people still do track perpetual counters.
>>
>> Actually its only dyn_inc (we don't have a dyn_dec). But as we go
>> forward, we may expose capability to support guages and may manipulate
>> it with dyn_inc and dyn_dec.
>>
>> Resettable flag in dyn-stats bucket does the same thing as
>> resettable-flag in statsobj static-counter.
>>
>> On Tue, Mar 29, 2016 at 3:13 AM, David Lang  wrote:
>>>
>>> On Tue, 29 Mar 2016, singh.janmejay wrote:
>>>
 Well, from counter pov, some counters by nature need to be resettable,
 some are actually guages (so resetting makes little sense) and some
 are perpetual (so again resetting is not the right thing to do).
>>>
>>>
>>>
>>> can you explain the different categories better? I don't see how any
>>> stats
>>> can be 'by nature' one form or another, since it's possible to convert
>>> the
>>> data from one form to another.
>>>
>>> The idea of perpetual counters doesn't work if the contents of the
>>> counters
>>> are lost at restart.
>>>
>>> It could be that I didn't read the documentation well enough and missed
>>> references to ways to manipulate them besides dyn_inc and dyn_dec
>>>
>>> David Lang
>>>
>>>
 A global control needs to be at impstats level because some users may
 want to access data in rate-of-change (dx/dt) form rather than
 absolute value form (it can be derived by adding up the numbers and
 studying the slope, but its prone to error when some messages are
 dropped etc). So user needs to have a say in how counters should be
 reported, should they be reset or not.

 Same goes for dyn-stats. User may want a perpetual counter for one
 dyn-stats bucket (regardless of other buckets being configured to
 reset), so its important to allow a particular bucket to be configured
 as non-resetting.

 However, if user chooses to view all the counters in cumulative form,
 the same config that works for static-counters should work for
 dyn-stats counters too (so impstats flag is relevant as well).

 And'ing is necessary because it doesn't make sense to reset a
 by-nature non-resettable counter.


 On Tue, Mar 29, 2016 at 12:25 AM, David Lang  wrote:
>
>
> What is the thinking in having two switches that get ANDed together for
> resetting things? I could see two independent controls, or I could see
> the
> pstats setting controlling everything, but I don't understand two
> getting
> ANDed together.
>
> David Lang
>
>
> On Tue, 29 Mar 2016, singh.janmejay wrote:
>
>> Will add this to the documentation when I touch it for
>> output-restructuring.
>>
>> On Tue, Mar 29, 2016 at 12:11 AM, David Lang  wrote:
>>>
>>>
>>>
>>> ahh, that's not at all clear from the documentation.
>>>
>>> I turned off resetting the counters at the pstats level because I was
>>> getting soem errors/coredumps from them (quite a few months ago, so
>>> it
>>> may
>>> be solved now) I'm currently getting >100 lines of pstats output now,
>>> before
>>> I add dyn_stats to things.
>>>

Re: [rsyslog] another dyn_stats question

[David Lang]

On Wed, 30 Mar 2016, singh.janmejay wrote:


Consider a queue-size, its not really a counter, its a guage. It will
provide point-in-time value of size of the queue. Resetting such a
value will make no sense. Some static metrics we report are guages (so
subsystem supporting the metric must identify it to be resettable or
not).


but that won't work well with the current policy.

you can't have anything as a gauge unless you eliminate the possiblity for any 
of the pstats data to be gauges.


If pstats doesn't reset values, then dyn_stats can't implement gauges. If pststs 
does reset values, you have turned a lot of things that probably shouldn't reset 
into gauges.


it would make far more sense to define this purely on a per-stats basis.

pstats has some items that are always gauges (size/maxsize/etc) and other items 
that are allowed to be treated as gauges or as counters.


In my case, I don't want to reset those things (submitted/enqueued) that can be 
counters, but that prevents me from having dyn_stats items that are gauges.


David Lang


Yes, restart is a problem. But restarts are really in-frequent enough
to not matter. When they do happen, dx/dt chart shows a big dip, but
people still do track perpetual counters.

Actually its only dyn_inc (we don't have a dyn_dec). But as we go
forward, we may expose capability to support guages and may manipulate
it with dyn_inc and dyn_dec.

Resettable flag in dyn-stats bucket does the same thing as
resettable-flag in statsobj static-counter.

On Tue, Mar 29, 2016 at 3:13 AM, David Lang  wrote:

On Tue, 29 Mar 2016, singh.janmejay wrote:


Well, from counter pov, some counters by nature need to be resettable,
some are actually guages (so resetting makes little sense) and some
are perpetual (so again resetting is not the right thing to do).



can you explain the different categories better? I don't see how any stats
can be 'by nature' one form or another, since it's possible to convert the
data from one form to another.

The idea of perpetual counters doesn't work if the contents of the counters
are lost at restart.

It could be that I didn't read the documentation well enough and missed
references to ways to manipulate them besides dyn_inc and dyn_dec

David Lang



A global control needs to be at impstats level because some users may
want to access data in rate-of-change (dx/dt) form rather than
absolute value form (it can be derived by adding up the numbers and
studying the slope, but its prone to error when some messages are
dropped etc). So user needs to have a say in how counters should be
reported, should they be reset or not.

Same goes for dyn-stats. User may want a perpetual counter for one
dyn-stats bucket (regardless of other buckets being configured to
reset), so its important to allow a particular bucket to be configured
as non-resetting.

However, if user chooses to view all the counters in cumulative form,
the same config that works for static-counters should work for
dyn-stats counters too (so impstats flag is relevant as well).

And'ing is necessary because it doesn't make sense to reset a
by-nature non-resettable counter.


On Tue, Mar 29, 2016 at 12:25 AM, David Lang  wrote:


What is the thinking in having two switches that get ANDed together for
resetting things? I could see two independent controls, or I could see
the
pstats setting controlling everything, but I don't understand two getting
ANDed together.

David Lang


On Tue, 29 Mar 2016, singh.janmejay wrote:


Will add this to the documentation when I touch it for
output-restructuring.

On Tue, Mar 29, 2016 at 12:11 AM, David Lang  wrote:



ahh, that's not at all clear from the documentation.

I turned off resetting the counters at the pstats level because I was
getting soem errors/coredumps from them (quite a few months ago, so it
may
be solved now) I'm currently getting >100 lines of pstats output now,
before
I add dyn_stats to things.

I'll wait for the restructuring of the dyn_stats output and the new
foreach
and then try again.

David Lang

On Mon, 28 Mar 2016, singh.janmejay wrote:


Date: Mon, 28 Mar 2016 23:59:46 +0530

From: singh.janmejay 
Reply-To: rsyslog-users 
To: rsyslog-users 
Subject: Re: [rsyslog] another dyn_stats question

Sorry, I meant and'ed not or'ed.

On Mon, Mar 28, 2016 at 11:02 PM, singh.janmejay
 wrote:




No, this reset mechanism is no different than the mechanism that
static-counters use. If you turn it off at impstats level, it won't
reset.

This flag basically sets CTR_FLAG_RESETTABLE (statsobj.h).

Its or'ed with stats-reporter's choice of resetting or not in this
way: (bResetCtrs && (pCtr->flags & CTR_FLAG_RESETTABLE)), where
bResetCtrs comes from that boolean param in impstats.

On Mon, Mar 28, 2016 at 9:58 PM, David Lang  wrote:




the first one should not be resetting, but the 

Re: [rsyslog] another dyn_stats question

[singh.janmejay]

Consider a queue-size, its not really a counter, its a guage. It will
provide point-in-time value of size of the queue. Resetting such a
value will make no sense. Some static metrics we report are guages (so
subsystem supporting the metric must identify it to be resettable or
not).

Yes, restart is a problem. But restarts are really in-frequent enough
to not matter. When they do happen, dx/dt chart shows a big dip, but
people still do track perpetual counters.

Actually its only dyn_inc (we don't have a dyn_dec). But as we go
forward, we may expose capability to support guages and may manipulate
it with dyn_inc and dyn_dec.

Resettable flag in dyn-stats bucket does the same thing as
resettable-flag in statsobj static-counter.

On Tue, Mar 29, 2016 at 3:13 AM, David Lang  wrote:
> On Tue, 29 Mar 2016, singh.janmejay wrote:
>
>> Well, from counter pov, some counters by nature need to be resettable,
>> some are actually guages (so resetting makes little sense) and some
>> are perpetual (so again resetting is not the right thing to do).
>
>
> can you explain the different categories better? I don't see how any stats
> can be 'by nature' one form or another, since it's possible to convert the
> data from one form to another.
>
> The idea of perpetual counters doesn't work if the contents of the counters
> are lost at restart.
>
> It could be that I didn't read the documentation well enough and missed
> references to ways to manipulate them besides dyn_inc and dyn_dec
>
> David Lang
>
>
>> A global control needs to be at impstats level because some users may
>> want to access data in rate-of-change (dx/dt) form rather than
>> absolute value form (it can be derived by adding up the numbers and
>> studying the slope, but its prone to error when some messages are
>> dropped etc). So user needs to have a say in how counters should be
>> reported, should they be reset or not.
>>
>> Same goes for dyn-stats. User may want a perpetual counter for one
>> dyn-stats bucket (regardless of other buckets being configured to
>> reset), so its important to allow a particular bucket to be configured
>> as non-resetting.
>>
>> However, if user chooses to view all the counters in cumulative form,
>> the same config that works for static-counters should work for
>> dyn-stats counters too (so impstats flag is relevant as well).
>>
>> And'ing is necessary because it doesn't make sense to reset a
>> by-nature non-resettable counter.
>>
>>
>> On Tue, Mar 29, 2016 at 12:25 AM, David Lang  wrote:
>>>
>>> What is the thinking in having two switches that get ANDed together for
>>> resetting things? I could see two independent controls, or I could see
>>> the
>>> pstats setting controlling everything, but I don't understand two getting
>>> ANDed together.
>>>
>>> David Lang
>>>
>>>
>>> On Tue, 29 Mar 2016, singh.janmejay wrote:
>>>
 Will add this to the documentation when I touch it for
 output-restructuring.

 On Tue, Mar 29, 2016 at 12:11 AM, David Lang  wrote:
>
>
> ahh, that's not at all clear from the documentation.
>
> I turned off resetting the counters at the pstats level because I was
> getting soem errors/coredumps from them (quite a few months ago, so it
> may
> be solved now) I'm currently getting >100 lines of pstats output now,
> before
> I add dyn_stats to things.
>
> I'll wait for the restructuring of the dyn_stats output and the new
> foreach
> and then try again.
>
> David Lang
>
> On Mon, 28 Mar 2016, singh.janmejay wrote:
>
>> Date: Mon, 28 Mar 2016 23:59:46 +0530
>>
>> From: singh.janmejay 
>> Reply-To: rsyslog-users 
>> To: rsyslog-users 
>> Subject: Re: [rsyslog] another dyn_stats question
>>
>> Sorry, I meant and'ed not or'ed.
>>
>> On Mon, Mar 28, 2016 at 11:02 PM, singh.janmejay
>>  wrote:
>>>
>>>
>>>
>>> No, this reset mechanism is no different than the mechanism that
>>> static-counters use. If you turn it off at impstats level, it won't
>>> reset.
>>>
>>> This flag basically sets CTR_FLAG_RESETTABLE (statsobj.h).
>>>
>>> Its or'ed with stats-reporter's choice of resetting or not in this
>>> way: (bResetCtrs && (pCtr->flags & CTR_FLAG_RESETTABLE)), where
>>> bResetCtrs comes from that boolean param in impstats.
>>>
>>> On Mon, Mar 28, 2016 at 9:58 PM, David Lang  wrote:



 the first one should not be resetting, but the rest should be,
 correct?

 David Lang


 dyn_stats(name="message_framing" resettable="off"
 maxCardinality="3000"
 unusedMetricLife="1200")
 dyn_stats(name="msgs_per_host" resettable="on" maxCardinality="3000"
 

Re: [rsyslog] rsyslog and json format

[David Lang]

On Tue, 29 Mar 2016, Giulio Vaccari wrote:


Hi everybody
I have just start using Rsyslog because I need to send the access_log of
apache to a collector.
By the way, I want to send them in Json format so, in this way, the
database can see any field without any sort of post processing (or at
least I hope!).

First, I'm using centos7 with Rsyslog v 7.7.4  and centos6 with Rsyslog
v 5.8.10.
I know that those version are outdated but I can't install or update
nothing on the server, I must use them and that's all! :\

I have alredy convert the standard log file in Json format and they are
correct (tested using a Json validator)
(as example)

{ "host": "192.168.122.1", "user": "-", "timestamp":
"25-03-201609:44:47+0100", "protocol": "HTTP/1.1", "method": "GET",
"alive": "4", "urlpath": "/noindex/css/fonts/Bold/OpenSans-Bold.ttf",
"urlquery": "", "status":" 404", "bytes": "238", "header":
"http://192.168.122.20/noindex/css/open-sans.css;, "useragent":
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101
Firefox/45.0", "duration": "331", "connection": "+", "bReceived": "393",
"bSent": "453", "firstLine": "GET
/noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1", "vhost":
"centos7_test.example.com" }

As a test I have try to copy the custom log and copy in the same machine
where I have installed the collector and I have added them using the
file input option... Simply perfect...

Now i want send them to the remote collector (Logstash) using rsyslog
and the imfile protocol:

input(type="imfile"
 File="/var/log/httpd/logstash_access_log"  <-- (custom log folder)
 Tag="apache"
 StateFile="statefile1")

if $programname == 'apache' then {
   action(
   type="omfwd"
   Target="192.168.122.32"
   Port="5514"
   Protocol="tcp"
   )
   stop
}


This syntax doesn't work with rsyslog v5.x


Logstash can receive the message but it give me a jsonparsefailure (so
for the program that isn't a json)
(as example)

  "message" => "<133>Mar 25 17:10:49 centos7_test apache
192.168.122.1 - - [25/Mar/2016:17:10:48 +0100] \"GET
/noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1\" 404 238
\"http://192.168.122.20/noindex/css/open-sans.css\; \"Mozilla/5.0 (X11;
Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0\"",
 "tags" => [
   [0] "_jsonparsefailure"
   ],
 "@version" => "1",
   "@timestamp" => "2016-03-25T16:10:50.544Z",
 "host" => "192.168.122.20",
 "port" => 52698,
 "type" => "apache"
}

Looks like the message is not formatted in json anymore! Why? Rsyslog
can't import formatted file?


there are a lot of things that current versions of rsyslog can do that ancient 
(5+ year old) versions of rsyslog can't do.


but in this case, you are telling it to write out in a standard format that is 
not json, but telling logstash to read the file as if it were all json, the 
writer and reader need to agree on the format of a file.


But why are you having Apache write the log to a file, then having rsyslog read 
the file and write to a different file for logstash to read and then deliver to 
ElasticSearch? It would seem that you should cut out one of the items in this 
chain. Either have logstash read the file from Apache, or have rsyslog deliver 
the log into ElasticSearch.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] retry if output module returns RS_RET_SUSPENDED

[Kane Kim]

Updated config to:
module(load="omrelp")
local0.* action(type="omrelp"
target="10.0.0.1"
port="2514"
name="debug_relp"
queue.filename="relp_queue"
queue.maxdiskspace="1G"
queue.saveonshutdown="on"
queue.spoolDirectory="/var/spool/rsyslog"
queue.type="Disk"
action.resumeRetryCount="-1"
action.resumeInterval="1"
)

Still message is lost:

3489.658349703:imuxsock.c : Message from UNIX socket: #3
3489.658367582:imuxsock.c : main Q: qqueueAdd: entry added, size now
log 1, phys 1 entries
3489.658372561:imuxsock.c : main Q: EnqueueMsg advised worker start
3489.658374985:imuxsock.c : imuxsock calling select, active
file descriptors (max 3): 3
3489.658384201:main Q:Reg/w0  : wti 0x23c8ba0: worker awoke from idle
processing
3489.658386792:main Q:Reg/w0  : DeleteProcessedBatch: we deleted 0 objects
and enqueued 0 objects
3489.658388548:main Q:Reg/w0  : doDeleteBatch: delete batch from store, new
sizes: log 1, phys 1
3489.658391356:main Q:Reg/w0  : processBATCH: batch of 1 elements must be
processed
3489.658392786:main Q:Reg/w0  : processBATCH: next msg 0: <134>Mar 29
17:44:49 logger: test
3489.658395027:main Q:Reg/w0  : PRIFILT 'local0.*'
3489.658397540:main Q:Reg/w0  : pmask:  X  X  X  X  X  X  X  X  X  X  X
 X  X  X  X  X FF  X  X  X  X  X  X  X  X  X
3489.658413235:main Q:Reg/w0  : PRIFILT condition result is 1
3489.658414686:main Q:Reg/w0  : ACTION 0 [omrelp:action(type="omrelp"
...)]
3489.658417648:main Q:Reg/w0  : executing action 0
3489.658419520:main Q:Reg/w0  : action 'debug_relp': called, logging to
omrelp (susp 0/0, direct q 0)
3489.658433462:main Q:Reg/w0  : strm 0x23b74c0: file -1(relp_queue) flush,
buflen 410
3489.658435496:main Q:Reg/w0  : strmPhysWrite, stream 0x23b74c0, len 410
3489.658484243:main Q:Reg/w0  : file
'/var/spool/rsyslog/relp_queue.0001' opened as #5 with mode 384
3489.658487347:main Q:Reg/w0  : strm 0x23b74c0: opened file
'/var/spool/rsyslog/relp_queue.0001' for WRITE as 5
3489.658500498:main Q:Reg/w0  : strm 0x23b74c0: file 5 write wrote 410 bytes
3489.658502751:main Q:Reg/w0  : debug_relp queue: write wrote 410 octets to
disk, queue disk size now 410 octets, EnqOnly:0
3489.658504568:main Q:Reg/w0  : debug_relp queue: qqueueAdd: entry added,
size now log 1, phys 1 entries
3489.658506705:main Q:Reg/w0  : debug_relp queue:Reg: high activity -
starting 1 additional worker thread(s).
3489.658528951:main Q:Reg/w0  : debug_relp queue:Reg: started with state 0,
num workers now 1
3489.658530689:main Q:Reg/w0  : debug_relp queue: EnqueueMsg advised worker
start
3489.658532183:main Q:Reg/w0  : action 'debug_relp': set suspended state to
0
3489.658533672:main Q:Reg/w0  : PRIFILT
'*.info;mail.none;authpriv.none;cron.none'
3489.658536164:main Q:Reg/w0  : pmask: 7F 7F  X 7F 7F 7F 7F 7F 7F  X  X
7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F 7F
3489.658552240:main Q:Reg/w0  : PRIFILT condition result is 1
3489.658553393:main Q:Reg/w0  : ACTION 1
[builtin:omfile:/var/log/messages]
3489.658556301:main Q:Reg/w0  : executing action 1
3489.658557878:main Q:Reg/w0  : action 'action 1': called, logging to
builtin:omfile (susp 0/0, direct q 1)
3489.658561265:main Q:Reg/w0  : action 'action 1': is transactional -
executing in commit phase
3489.658563696:main Q:Reg/w0  : Action 1 transitioned to state: itx
3489.658564902:main Q:Reg/w0  : action 'action 1': set suspended state to 0
3489.658566053:main Q:Reg/w0  : PRIFILT 'authpriv.*'
3489.658568233:main Q:Reg/w0  : pmask:  X  X  X  X  X  X  X  X  X  X FF
 X  X  X  X  X  X  X  X  X  X  X  X  X  X  X
3489.658582888:main Q:Reg/w0  : PRIFILT condition result is 0
3489.658584058:main Q:Reg/w0  : PRIFILT 'mail.*'
3489.658586243:main Q:Reg/w0  : pmask:  X  X FF  X  X  X  X  X  X  X  X
 X  X  X  X  X  X  X  X  X  X  X  X  X  X  X
3489.658600961:main Q:Reg/w0  : PRIFILT condition result is 0
3489.658602136:main Q:Reg/w0  : PRIFILT 'cron.*'
3489.658604562:main Q:Reg/w0  : pmask:  X  X  X  X  X  X  X  X  X FF  X
 X  X  X  X  X  X  X  X  X  X  X  X  X  X  X
3489.658619159:main Q:Reg/w0  : PRIFILT condition result is 0
3489.658620352:main Q:Reg/w0  : PRIFILT '*.emerg'
3489.658622551:main Q:Reg/w0  : pmask:  1  1  1  1  1  1  1  1  1  1  1
 1  1  1  1  1  1  1  1  1  1  1  1  1  1  1
3489.658638435:main Q:Reg/w0  : PRIFILT condition result is 0
3489.658639529:main Q:Reg/w0  : PRIFILT 'uucp,news.crit'
3489.658641711:main Q:Reg/w0  : pmask:  X  X  X  X  X  X  X  7  7  X  X
 X  X  X  X  X  X  X  X  X  X  X  X  X  X  X
3489.658656380:main Q:Reg/w0  : PRIFILT condition result is 0
3489.658657532:main Q:Reg/w0  : PRIFILT 'local7.*'
3489.658659794:main Q:Reg/w0  : pmask:  X  X  X  X  X  X  X  X  X  X  X
 X  X  X  X  X  X  X  X  X  X  X  X FF  X  X
3489.658674346:main Q:Reg/w0  : PRIFILT condition result is 0
3489.658675686:main Q:Reg/w0  : END batch execution phase, entering to
commit phase [processed 1 of 1 messages]
3489.658677400:main 

Re: [rsyslog] could not load module '/usr/lib/rsyslog/lmnsd_gtls.so

[Thomas D.]

Hi,

Muhammad Asif wrote:
> I need your help regarding TLS. I have install rsyslog 8.17 on Ubuntu
> 14.04. I want to use rsyslog as agent to send logs to remote TCP server on
> TLS. When I add my configurations and restart rsyslog, I face the following
> error.
> 
> Mar 29 16:32:16 ubuntu rsyslogd-2078: unexpected GnuTLS error -64 in
> nsd_gtls.c:611: Error while reading file.

Looks like rsyslog cannot read your certificate. Make sure the path is
valid (don't forget ACLs) and that the file contains a *formal* valid
x509 certificate in PEM format.


-Thomas

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] could not load module '/usr/lib/rsyslog/lmnsd_gtls.so

[Muhammad Asif]

Hi Geeks,

I need your help regarding TLS. I have install rsyslog 8.17 on Ubuntu
14.04. I want to use rsyslog as agent to send logs to remote TCP server on
TLS. When I add my configurations and restart rsyslog, I face the following
error.

Mar 29 16:32:16 ubuntu rsyslogd-2078: unexpected GnuTLS error -64 in
nsd_gtls.c:611: Error while reading file.  [v8.17.0 try
http://www.rsyslog.com/e/2078

 ]
Mar 29 16:32:16 ubuntu rsyslogd-2068: could not load module
'/usr/lib/rsyslog/lmnsd_gtls.so', rsyslog error -2078  [v8.17.0 try
http://www.rsyslog.com/e/2068

 ]

My Configurations are as below.

$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /home/geek/fw/config/certs/server/ca.pem
$DefaultNetstreamDriverCertFile
/home/geek/fw/config/certs/client/client-cert.pem
$DefaultNetstreamDriverKeyFile
/home/geek/fw/config/certs/client/client-key.pem
$ActionSendStreamDriverAuthMode anon
$ActionSendStreamDriverMode 1

Any clue would be appreciated.

Regards
M.Asif
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Problem with RELP TLS

[Rainer Gerhards]

2016-03-29 11:50 GMT+02:00 Elmopi, Stefano :
> Hi,
>
> thanks for the reply. So with CentOS 6.6, without recompiling librelp, can
> I not configure TLS ?? Even if you install an earlier version of rsyslog ??
> I tried to configure the module imtcp with TLS but it still has to be in
> error also with the option anonymous TLS !!!

If you just use

tls="on"

and no other TLS parameters, this works. I have done it 5 minutes ago ;)

> In short, without recompilation, is there no option to configure TLS on
> CentOS 6.6 ??

But... I am working on this case

https://github.com/rsyslog/rsyslog/issues/907

which shows that CentOS 6 requires a patch to librelp because it's
version of GnuTLS needs libgcrypt. The patch is upcoming and will lead
to a new release of librelp within the next days.

Rainer

>
> Best Regard
>
> Message: 29
> Date: Thu, 24 Mar 2016 13:18:15 +0100
> From: "Elmopi, Stefano" 
> To: rsyslog@lists.adiscon.com
> Subject: [rsyslog] Problem with RELP TLS
> Message-ID:
> 
> Content-Type: text/plain; charset=UTF-8
>
> Hi,
> I installed rsyslog on CentOS 6.6, these are the installed packages:
> rsyslog-8.17.0-1.el6.x86_64
> rsyslog-mysql-8.17.0-1.el6.x86_64
> rsyslog-gnutls-8.17.0-1.el6.x86_64
> rsyslog-snmp-8.17.0-1.el6.x86_64
> rsyslog-relp-8.17.0-1.el6.x86_64
> librelp-1.2.9-1.el6.x86_64
> gnutls-2.8.5-19.el6_7.x86_64
>
> and I would like to configure Relp with TLS:
>
> input(type="imrelp" name="TCP_Collector" ruleset="Collector" port="20514"
> tls="on"
> tls.caCert="/etc/pki/rsyslog/CA/CA_Collaudo.crt"
> tls.myCert="/etc/pki/rsyslog/Certs/sys01devpom.sociale.it.crt"
> tls.myPrivKey="/etc/pki/rsyslog/Key/sys01devpom.sociale.it.pem"
> tls.authMode="name"
> tls.permittedpeer=["*.sociale.it"]
> )
>
>
> but when I start rsyslog, read in the log file:
>
> rsyslogd-2291: imrelp: could not activate relp listner, code 10046 [v8.17.0
> try http://www.rsyslog.com/e/2291 ]
>
> only this line and nothing else.
> I made many tests and searched a lot on Google but nothing, I found no
> solution
> Thanks for your help
>
>
> Best Regards
>
>
> Ing. Stefano Elmopi
> Cooperativa Capodarco - Resp. Area ICT Gestione Esercizio
> Via Ostiense 131/L Corpo B, 00154 Roma
>
> cell. 3466147165
> tel.  0657060500
>
> email:stefano.elm...@sociale.it
>
> --
> "Ai sensi e per gli effetti della legge sulla tutela dei dati personali
> (D.lgs 196/2003),
> le informazioni contenute nella presente @mail sono di natura riservata e
> destinate
> ad un uso aziendale-lavorativo con esclusione di utilizzi ad uso personale;
> come tali,
> pertanto, sono riservate esclusivamente ai destinatari sopra indicati. E'
> proibito leggere,
> copiare, usare o diffondere il contenuto della presente @mail senza
> autorizzazione.
> Se avete ricevuto questa @mail per errore, siete pregati di rispedire la
> stessa al mittente.
> Grazie"
>
>
> --
>
> Message: 30
> Date: Thu, 24 Mar 2016 14:52:54 +0100
> From: Rainer Gerhards 
> To: rsyslog-users 
> Subject: Re: [rsyslog] Problem with RELP TLS
> Message-ID:
> 
> Content-Type: text/plain; charset=UTF-8
>
> 2016-03-24 13:18 GMT+01:00 Elmopi, Stefano :
>> Hi,
>> I installed rsyslog on CentOS 6.6, these are the installed packages:
>> rsyslog-8.17.0-1.el6.x86_64
>> rsyslog-mysql-8.17.0-1.el6.x86_64
>> rsyslog-gnutls-8.17.0-1.el6.x86_64
>> rsyslog-snmp-8.17.0-1.el6.x86_64
>> rsyslog-relp-8.17.0-1.el6.x86_64
>> librelp-1.2.9-1.el6.x86_64
>> gnutls-2.8.5-19.el6_7.x86_64
>>
>> and I would like to configure Relp with TLS:
>>
>> input(type="imrelp" name="TCP_Collector" ruleset="Collector" port="20514"
>> tls="on"
>> tls.caCert="/etc/pki/rsyslog/CA/CA_Collaudo.crt"
>> tls.myCert="/etc/pki/rsyslog/Certs/sys01devpom.sociale.it.crt"
>> tls.myPrivKey="/etc/pki/rsyslog/Key/sys01devpom.sociale.it.pem"
>> tls.authMode="name"
>> tls.permittedpeer=["*.sociale.it"]
>> )
>>
>>
>> but when I start rsyslog, read in the log file:
>>
>> rsyslogd-2291: imrelp: could not activate relp listner, code 10046
> [v8.17.0
>> try http://www.rsyslog.com/e/2291 ]
>
> >From librelp:
> #define RELP_RET_ERR_NO_TLS_AUTH RELPERR_BASE + 46/**< platform
> does not provide TLS auth support */
>
> So it looks like CentOS 6.6 does not offer a recent enough version of
> GnuTLS to support TLS authentication. Some of the APIs we need are
> missing. I remember we implemented a work-around to at least support
> anonymous TLS for such platforms.
>
> You can solve this by installing a new GnuTLS version and building
> librelp (and possibly rsyslog) from source.
>
> As a side note, the same information is present in the link rsyslog
> gave in the error message ;)
>
> HTH
> Rainer
>
>
>
> Ing. Stefano Elmopi
> Cooperativa Capodarco - Resp. Area ICT 

[rsyslog] Problem with RELP TLS

[Elmopi, Stefano]

Hi,

thanks for the reply. So with CentOS 6.6, without recompiling librelp, can
I not configure TLS ?? Even if you install an earlier version of rsyslog ??
I tried to configure the module imtcp with TLS but it still has to be in
error also with the option anonymous TLS !!!
In short, without recompilation, is there no option to configure TLS on
CentOS 6.6 ??

Best Regard

Message: 29
Date: Thu, 24 Mar 2016 13:18:15 +0100
From: "Elmopi, Stefano" 
To: rsyslog@lists.adiscon.com
Subject: [rsyslog] Problem with RELP TLS
Message-ID:

Content-Type: text/plain; charset=UTF-8

Hi,
I installed rsyslog on CentOS 6.6, these are the installed packages:
rsyslog-8.17.0-1.el6.x86_64
rsyslog-mysql-8.17.0-1.el6.x86_64
rsyslog-gnutls-8.17.0-1.el6.x86_64
rsyslog-snmp-8.17.0-1.el6.x86_64
rsyslog-relp-8.17.0-1.el6.x86_64
librelp-1.2.9-1.el6.x86_64
gnutls-2.8.5-19.el6_7.x86_64

and I would like to configure Relp with TLS:

input(type="imrelp" name="TCP_Collector" ruleset="Collector" port="20514"
tls="on"
tls.caCert="/etc/pki/rsyslog/CA/CA_Collaudo.crt"
tls.myCert="/etc/pki/rsyslog/Certs/sys01devpom.sociale.it.crt"
tls.myPrivKey="/etc/pki/rsyslog/Key/sys01devpom.sociale.it.pem"
tls.authMode="name"
tls.permittedpeer=["*.sociale.it"]
)


but when I start rsyslog, read in the log file:

rsyslogd-2291: imrelp: could not activate relp listner, code 10046 [v8.17.0
try http://www.rsyslog.com/e/2291 ]

only this line and nothing else.
I made many tests and searched a lot on Google but nothing, I found no
solution
Thanks for your help


Best Regards


Ing. Stefano Elmopi
Cooperativa Capodarco - Resp. Area ICT Gestione Esercizio
Via Ostiense 131/L Corpo B, 00154 Roma

cell. 3466147165
tel.  0657060500

email:stefano.elm...@sociale.it

--
"Ai sensi e per gli effetti della legge sulla tutela dei dati personali
(D.lgs 196/2003),
le informazioni contenute nella presente @mail sono di natura riservata e
destinate
ad un uso aziendale-lavorativo con esclusione di utilizzi ad uso personale;
come tali,
pertanto, sono riservate esclusivamente ai destinatari sopra indicati. E'
proibito leggere,
copiare, usare o diffondere il contenuto della presente @mail senza
autorizzazione.
Se avete ricevuto questa @mail per errore, siete pregati di rispedire la
stessa al mittente.
Grazie"


--

Message: 30
Date: Thu, 24 Mar 2016 14:52:54 +0100
From: Rainer Gerhards 
To: rsyslog-users 
Subject: Re: [rsyslog] Problem with RELP TLS
Message-ID:

Content-Type: text/plain; charset=UTF-8

2016-03-24 13:18 GMT+01:00 Elmopi, Stefano :
> Hi,
> I installed rsyslog on CentOS 6.6, these are the installed packages:
> rsyslog-8.17.0-1.el6.x86_64
> rsyslog-mysql-8.17.0-1.el6.x86_64
> rsyslog-gnutls-8.17.0-1.el6.x86_64
> rsyslog-snmp-8.17.0-1.el6.x86_64
> rsyslog-relp-8.17.0-1.el6.x86_64
> librelp-1.2.9-1.el6.x86_64
> gnutls-2.8.5-19.el6_7.x86_64
>
> and I would like to configure Relp with TLS:
>
> input(type="imrelp" name="TCP_Collector" ruleset="Collector" port="20514"
> tls="on"
> tls.caCert="/etc/pki/rsyslog/CA/CA_Collaudo.crt"
> tls.myCert="/etc/pki/rsyslog/Certs/sys01devpom.sociale.it.crt"
> tls.myPrivKey="/etc/pki/rsyslog/Key/sys01devpom.sociale.it.pem"
> tls.authMode="name"
> tls.permittedpeer=["*.sociale.it"]
> )
>
>
> but when I start rsyslog, read in the log file:
>
> rsyslogd-2291: imrelp: could not activate relp listner, code 10046
[v8.17.0
> try http://www.rsyslog.com/e/2291 ]

>From librelp:
#define RELP_RET_ERR_NO_TLS_AUTH RELPERR_BASE + 46/**< platform
does not provide TLS auth support */

So it looks like CentOS 6.6 does not offer a recent enough version of
GnuTLS to support TLS authentication. Some of the APIs we need are
missing. I remember we implemented a work-around to at least support
anonymous TLS for such platforms.

You can solve this by installing a new GnuTLS version and building
librelp (and possibly rsyslog) from source.

As a side note, the same information is present in the link rsyslog
gave in the error message ;)

HTH
Rainer



Ing. Stefano Elmopi
Cooperativa Capodarco - Resp. Area ICT Gestione Esercizio
Via Ostiense 131/L Corpo B, 00154 Roma

cell. 3466147165
tel.  0657060500

email:stefano.elm...@sociale.it

-- 
"Ai sensi e per gli effetti della legge sulla tutela dei dati personali 
(D.lgs 196/2003),
le informazioni contenute nella presente @mail sono di natura riservata e 
destinate
ad un uso aziendale-lavorativo con esclusione di utilizzi ad uso personale; 
come tali,
pertanto, sono riservate esclusivamente ai destinatari sopra indicati. E' 
proibito leggere,
copiare, usare o diffondere il contenuto della presente @mail senza 
autorizzazione.
Se avete ricevuto questa @mail per errore, siete pregati di rispedire la 
stessa al mittente.
Grazie"

Re: [rsyslog] Parsing severity from log4j files?

[Joan]

Hell David,

Thanks for the answer, more or less I did what you recommend, but without
mmnormalize (basically I couldn't find enough information to make it work)
and a hackier way

I created templates based on RSYSLOG_SyslogProtocol23Format (just changing
the PRI value)

template(name="Protocol23DEBUG"
> type="string"
>
>string= "<159>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME%
> %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA%
> %msg%\n"
>
>
> )
>
>
>
> template(name="Protocol23INFO"
> type="string"
>
>string= "<158>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME%
> %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA%
> %msg%\n"
>
>
> )
>
>
>
>
>
> template(name="Protocol23WARN"
> type="string"
>
>string= "<156>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME%
> %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA%
> %msg%\n"
>
> )
>

And then do the conditional sending with

if $msg contains 'INFO' then {
> local3.* @@192.168.1.10:514;Protocol23INFO
> } else if $msg contains 'DEBUG' then {
> local3.* @@192.168.1.10:514;Protocol23DEBUG
> } else if $msg contains 'WARN' then {
> local3.* @@192.168.1.10:514;Protocol23WARN
> } else {
> local3.* @@192.168.1.10:514;RSYSLOG_SyslogProtocol23Format
> }
>

Any pointers on doing this with a single template?

2016-03-23 0:44 GMT+01:00 David Lang :

> On Fri, 11 Mar 2016, Joan wrote:
>
> I have some files generated by java and written into logfiles by log4j that
>> I want to stream to graylog, I'd rather choose syslog with rfc5424 if
>> possible.
>> The files are like this
>>
>> --
>> 2016-03-10 07:01:09,778 INFO  [taskname] long description 1
>> 2016-03-10 07:01:09,784 DEBUG  [taskname] long description 2
>> --
>>
>>
>> Currently I am using imfile to forward the logs and its working quite
>> nicely
>> --
>> module(load="imfile"
>>
>>
>> mode="inotify"
>>
>> )
>>
>>
>>
>> input(type="imfile"
>>
>>
>> File="/opt/project/log/*.log"
>>
>> Tag="taskproject:"
>> Facility="local3"
>> )
>>
>> #Make data available on server reboot
>> $ActionQueueFileName reenviador
>> $ActionResumeRetryCount -1
>> $ActionQueueSaveOnShutdown on
>>
>> #Proper microsecond format
>> local3.* @@192.168.1.136:514;RSYSLOG_SyslogProtocol23Format
>> --
>>
>> The only issue, is that I'd like to have the severity set into the
>> forwarded logs and stripped from the log being sent, I've had a look into
>> mmnormalize, but I'm not sure it applies to the syslog format (I've seen
>> mostly json related examples)
>>
>
> There's not an easy way to do this.
>
> You could use mmnormalize to parse the fields and then either a series of
> if statements or a table lookup to convert the string to a number, and then
> do calculations to compute the new PRI value and use that value in a custom
> template when you forward the message.
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] rsyslog and json format

[Giulio Vaccari]

Hi everybody
I have just start using Rsyslog because I need to send the access_log of
apache to a collector.
By the way, I want to send them in Json format so, in this way, the
database can see any field without any sort of post processing (or at
least I hope!).

First, I'm using centos7 with Rsyslog v 7.7.4  and centos6 with Rsyslog
v 5.8.10.
I know that those version are outdated but I can't install or update
nothing on the server, I must use them and that's all! :\

I have alredy convert the standard log file in Json format and they are
correct (tested using a Json validator)
(as example)

{ "host": "192.168.122.1", "user": "-", "timestamp":
"25-03-201609:44:47+0100", "protocol": "HTTP/1.1", "method": "GET",
"alive": "4", "urlpath": "/noindex/css/fonts/Bold/OpenSans-Bold.ttf",
"urlquery": "", "status":" 404", "bytes": "238", "header":
"http://192.168.122.20/noindex/css/open-sans.css;, "useragent":
"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101
Firefox/45.0", "duration": "331", "connection": "+", "bReceived": "393",
"bSent": "453", "firstLine": "GET
/noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1", "vhost":
"centos7_test.example.com" }

As a test I have try to copy the custom log and copy in the same machine
where I have installed the collector and I have added them using the
file input option... Simply perfect...

Now i want send them to the remote collector (Logstash) using rsyslog
and the imfile protocol:

input(type="imfile"
  File="/var/log/httpd/logstash_access_log"  <-- (custom log folder)
  Tag="apache"
  StateFile="statefile1")

if $programname == 'apache' then {
action(
type="omfwd"
Target="192.168.122.32"
Port="5514"
Protocol="tcp"
)
stop
}

Logstash can receive the message but it give me a jsonparsefailure (so
for the program that isn't a json)
(as example)

   "message" => "<133>Mar 25 17:10:49 centos7_test apache
192.168.122.1 - - [25/Mar/2016:17:10:48 +0100] \"GET
/noindex/css/fonts/Bold/OpenSans-Bold.ttf HTTP/1.1\" 404 238
\"http://192.168.122.20/noindex/css/open-sans.css\; \"Mozilla/5.0 (X11;
Ubuntu; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0\"",
  "tags" => [
[0] "_jsonparsefailure"
],
  "@version" => "1",
"@timestamp" => "2016-03-25T16:10:50.544Z",
  "host" => "192.168.122.20",
  "port" => 52698,
  "type" => "apache"
}

Looks like the message is not formatted in json anymore! Why? Rsyslog
can't import formatted file?



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.