Re: [rsyslog] Fwd: Re: RES: Client/server architecture with omrelp/imrelp - Lost of messages in case of server side failure

[mostolog--- via rsyslog]

Probably David or Rainer could give further details, but based on our 
experience that didnt work as expected until we set resumeInterval.



On 30/10/17 16:04, Jether B S via rsyslog wrote:

Hi,

Is really there a RetryInterval parameter? Wouldn't it be the
action.resumeInterval? I've not used it because I belive It is using the
default value (60 seconds).

Thank you for your feedback.

Regards,

Jether


2017-10-30 9:46 GMT-02:00 mostolog--- via rsyslog <rsyslog@lists.adiscon.com

:
Hi


You are missing retryInterval, and I would say it's needed for proper
retry after a queue lock.

In our tests, once relp queue was full, it never resumed (although
resumeretrycount=-1 was set) until we added interval setting.


  On 26/10/17 15:52, Jether B. Santos via rsyslog wrote:


Hi,

In order to not loose any message I realized I have to change the
queue.dequeuebatchsize to "1" and action.resumeretrycount to a value
greater than default "0" (I used "-1" for infinite). That configuration has
worked for me:

ruleset(name="remote_logserver"){
action(type="omrelp" Target="10.0.0.43" Port="514"
queue.filename="fila-msg" queue.size="180" queue.saveonshutdown="on"
   queue.type="LinkedList" queue.dequeuebatchsize="1"
queue.maxdiskspace="536870912" action.resumeretrycount="-1"
  )
}

Regards,

Jether

De: Jether B. Santos
Enviada em: segunda-feira, 23 de outubro de 2017 20:40
Para: 'rsyslog@lists.adiscon.com' <rsyslog@lists.adiscon.com>
Assunto: RES: Client/server architecture with omrelp/imrelp - Lost of
messages in case of server side failure

Hello,

I realized the parameter RebindInterval was causing duplicated messages
in both balanced servers. I removed that and apparentely the messages
haven't been lost anymore.

Tomorrow I will stress it more.

Regards,

Jether

De: Jether B. Santos
Enviada em: segunda-feira, 23 de outubro de 2017 11:35
Para: 'rsyslog@lists.adiscon.com' <rsyslog@lists.adiscon.com>
Assunto: Client/server architecture with omrelp/imrelp - Lost of messages
in case of server side failure

Hi everybody,

I'm testing the following rsyslog client/server architecture:

- Client side: a rsyslog server that consumes messages from a local
apache log file through imfile module. The main worker threads save the
messages in a disk assisted queue of an omrelp action. The action threads
dequeue the messages and send them to rsyslog servers behind a haproxy
instance. Rsyslog's client configuration:


---
module(load="imfile" mode="inotify")
input(type="imfile" file="/var/log/httpd/access_log" tag="apache:"
facility="local3" severity="notice" ruleset="remote_logserver")

module(load="omrelp")
ruleset(name="remote_logserver"){
  action(type="omrelp" Target="10.0.0.43" Port="514"
queue.filename="fila-msg" queue.size="180" queue.saveonshutdown="on"
  queue.type="LinkedList" action.resumeretrycount="-1"
action.resumeInterval="1" RebindInterval="72")
}


---

- Server side: two rsyslog servers behind a haproxy load balancer
instance. The servers listen the logs through imrelp module. Rsyslog's
servers configuration:


---
module(load="imrelp")
input(type="imrelp" port="514")
local3.*/var/log/httpd-access.log


---

The problem: Everytime I stop and start the rsyslog daemon in the remote
rsyslog server where the client is conected, despite haproxy switches the
connection to the other rsyslog server, at least a message is lost. I tried
configure retrycount="-1" but without success.

I read about similar problem in the mail list (
http://lists.adiscon.net/pipermail/rsyslog/2016-March/042379.html) but
I've not understood exaclly what's been done to solve that (
https://github.com/rsyslog/rsyslog/issues/974?_pjax=%23js-
repo-pjax-container):

- Is it really mandatory to configure batch of only one message a time
(queue.dequeuebatchsize="1") in order to not lose any message in 

[rsyslog] Fwd: Re: RES: Client/server architecture with omrelp/imrelp - Lost of messages in case of server side failure

[mostolog--- via rsyslog]

Hi


You are missing retryInterval, and I would say it's needed for proper
retry after a queue lock.

In our tests, once relp queue was full, it never resumed (although
resumeretrycount=-1 was set) until we added interval setting.


 On 26/10/17 15:52, Jether B. Santos via rsyslog wrote:

Hi,

In order to not loose any message I realized I have to change the queue.dequeuebatchsize to "1" and 
action.resumeretrycount to a value greater than default "0" (I used "-1" for infinite). 
That configuration has worked for me:

ruleset(name="remote_logserver"){
   action(type="omrelp" Target="10.0.0.43" Port="514" queue.filename="fila-msg" 
queue.size="180" queue.saveonshutdown="on"
  queue.type="LinkedList" queue.dequeuebatchsize="1" 
queue.maxdiskspace="536870912" action.resumeretrycount="-1"
 )
}

Regards,

Jether

De: Jether B. Santos
Enviada em: segunda-feira, 23 de outubro de 2017 20:40
Para: 'rsyslog@lists.adiscon.com' 
Assunto: RES: Client/server architecture with omrelp/imrelp - Lost of messages 
in case of server side failure

Hello,

I realized the parameter RebindInterval was causing duplicated messages in both 
balanced servers. I removed that and apparentely the messages haven't been lost 
anymore.

Tomorrow I will stress it more.

Regards,

Jether

De: Jether B. Santos
Enviada em: segunda-feira, 23 de outubro de 2017 11:35
Para: 'rsyslog@lists.adiscon.com' 
>
Assunto: Client/server architecture with omrelp/imrelp - Lost of messages in 
case of server side failure

Hi everybody,

I'm testing the following rsyslog client/server architecture:

- Client side: a rsyslog server that consumes messages from a local apache log 
file through imfile module. The main worker threads save the messages in a disk 
assisted queue of an omrelp action. The action threads dequeue the messages and 
send them to rsyslog servers behind a haproxy instance. Rsyslog's client 
configuration:
---
module(load="imfile" mode="inotify")
input(type="imfile" file="/var/log/httpd/access_log" tag="apache:" facility="local3" 
severity="notice" ruleset="remote_logserver")

module(load="omrelp")
ruleset(name="remote_logserver"){
 action(type="omrelp" Target="10.0.0.43" Port="514" queue.filename="fila-msg" 
queue.size="180" queue.saveonshutdown="on"
 queue.type="LinkedList" action.resumeretrycount="-1" 
action.resumeInterval="1" RebindInterval="72")
}
---

- Server side: two rsyslog servers behind a haproxy load balancer instance. The 
servers listen the logs through imrelp module. Rsyslog's servers configuration:
---
module(load="imrelp")
input(type="imrelp" port="514")
local3.*/var/log/httpd-access.log
---

The problem: Everytime I stop and start the rsyslog daemon in the remote rsyslog server 
where the client is conected, despite haproxy switches the connection to the other 
rsyslog server, at least a message is lost. I tried configure retrycount="-1" 
but without success.

I read about similar problem in the mail list 
(http://lists.adiscon.net/pipermail/rsyslog/2016-March/042379.html) but I've 
not understood exaclly what's been done to solve that 
(https://github.com/rsyslog/rsyslog/issues/974?_pjax=%23js-repo-pjax-container):

- Is it really mandatory to configure batch of only one message a time 
(queue.dequeuebatchsize="1") in order to not lose any message in case of the 
remote rsyslog server fail?
- Has the "isolation" of a "corrupt message" implemented (Rainer said in the 
mail list mentioned above: I thought that if a batch failed, it pushed all the messages back on the 
queue and retried with a half size batch until it got to the one message that could not be 
processed and only did a fatal fail on that message)?
- What about the action retry feature? When it is enabled and you have a batch 
of many messages and just one fails, what's expected to happen?

My servers environment:
- Centos 7.3
- Rsyslog version: 8.24

Regards,

Jether



"Esta mensagem e reservada e sua divulgacao, distribuicao, reproducao ou qualquer 
forma de uso e proibida e depende de previa autorizacao desta instituicao. O remetente 
utiliza o correio eletronico no exercicio do seu trabalho ou em razao dele, eximindo esta 

Re: [rsyslog] RES: Client/server architecture with omrelp/imrelp - Lost of messages in case of server side failure

[mostolog--- via rsyslog]

Hi

You are missing retryInterval, and I would say it's needed for proper 
retry after a queue lock.


In our tests, once relp queue was full, it never resumed (although 
resumeretrycount=-1 was set) until we added interval setting.




On 26/10/17 15:52, Jether B. Santos via rsyslog wrote:

Hi,

In order to not loose any message I realized I have to change the queue.dequeuebatchsize to "1" and 
action.resumeretrycount to a value greater than default "0" (I used "-1" for infinite). 
That configuration has worked for me:

ruleset(name="remote_logserver"){
   action(type="omrelp" Target="10.0.0.43" Port="514" queue.filename="fila-msg" 
queue.size="180" queue.saveonshutdown="on"
  queue.type="LinkedList" queue.dequeuebatchsize="1" 
queue.maxdiskspace="536870912" action.resumeretrycount="-1"
 )
}

Regards,

Jether

De: Jether B. Santos
Enviada em: segunda-feira, 23 de outubro de 2017 20:40
Para: 'rsyslog@lists.adiscon.com' 
Assunto: RES: Client/server architecture with omrelp/imrelp - Lost of messages 
in case of server side failure

Hello,

I realized the parameter RebindInterval was causing duplicated messages in both 
balanced servers. I removed that and apparentely the messages haven't been lost 
anymore.

Tomorrow I will stress it more.

Regards,

Jether

De: Jether B. Santos
Enviada em: segunda-feira, 23 de outubro de 2017 11:35
Para: 'rsyslog@lists.adiscon.com' 
>
Assunto: Client/server architecture with omrelp/imrelp - Lost of messages in 
case of server side failure

Hi everybody,

I'm testing the following rsyslog client/server architecture:

- Client side: a rsyslog server that consumes messages from a local apache log 
file through imfile module. The main worker threads save the messages in a disk 
assisted queue of an omrelp action. The action threads dequeue the messages and 
send them to rsyslog servers behind a haproxy instance. Rsyslog's client 
configuration:
---
module(load="imfile" mode="inotify")
input(type="imfile" file="/var/log/httpd/access_log" tag="apache:" facility="local3" 
severity="notice" ruleset="remote_logserver")

module(load="omrelp")
ruleset(name="remote_logserver"){
 action(type="omrelp" Target="10.0.0.43" Port="514" queue.filename="fila-msg" 
queue.size="180" queue.saveonshutdown="on"
 queue.type="LinkedList" action.resumeretrycount="-1" 
action.resumeInterval="1" RebindInterval="72")
}
---

- Server side: two rsyslog servers behind a haproxy load balancer instance. The 
servers listen the logs through imrelp module. Rsyslog's servers configuration:
---
module(load="imrelp")
input(type="imrelp" port="514")
local3.*/var/log/httpd-access.log
---

The problem: Everytime I stop and start the rsyslog daemon in the remote rsyslog server 
where the client is conected, despite haproxy switches the connection to the other 
rsyslog server, at least a message is lost. I tried configure retrycount="-1" 
but without success.

I read about similar problem in the mail list 
(http://lists.adiscon.net/pipermail/rsyslog/2016-March/042379.html) but I've 
not understood exaclly what's been done to solve that 
(https://github.com/rsyslog/rsyslog/issues/974?_pjax=%23js-repo-pjax-container):

- Is it really mandatory to configure batch of only one message a time 
(queue.dequeuebatchsize="1") in order to not lose any message in case of the 
remote rsyslog server fail?
- Has the "isolation" of a "corrupt message" implemented (Rainer said in the 
mail list mentioned above: I thought that if a batch failed, it pushed all the messages back on the 
queue and retried with a half size batch until it got to the one message that could not be 
processed and only did a fatal fail on that message)?
- What about the action retry feature? When it is enabled and you have a batch 
of many messages and just one fails, what's expected to happen?

My servers environment:
- Centos 7.3
- Rsyslog version: 8.24

Regards,

Jether



"Esta mensagem e reservada e sua divulgacao, distribuicao, reproducao ou qualquer 
forma de uso e proibida e depende de previa autorizacao desta instituicao. O remetente 
utiliza o correio eletronico no exercicio do seu trabalho ou em razao dele, eximindo 

[rsyslog] Truncated UDP message error is shown on debug?

[mostolog--- via rsyslog]

Hi

As we were not receiving messages from one server while doing some 
tests, we run:

    rsyslogd -dn
but we weren't able to see anything wrong.

However, running:
    rsyslog -n
showed the following message:
    rsyslogd: omfwd/udp: message is 113152 bytes long, but UDP can send 
at most 65507 bytes (by RFC limit) - truncating message [v8.30.0 try 
http://www.rsyslog.com/e/2440 ]


Could anyone verify this message being written on debug also?

Thanks

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Exclude patterns for imfile whith different multiline format

[mostolog--- via rsyslog]

It doesn't seem possible to exclude specific files when using directory 
wilcards too.


eg: /logs/**/^{file_format_X.log}


Perhaps http://www.linuxjournal.com/content/bash-extended-globbing could 
do the trick, but i was wondering if there won't make more sense to use 
REGEX or exclude paratemer (or...)



Regards

On 17/10/17 16:59, Rainer Gerhards wrote:
You cannot exclude, bit you can include those that you want. I guess 
with ugly globbing that should be possible. Best try with ls first.


http://tldp.org/LDP/abs/html/globbingref.html

Sent from phone, thus brief.

Am 17.10.2017 15:23 schrieb "mostolog--- via rsyslog" 
<rsyslog@lists.adiscon.com <mailto:rsyslog@lists.adiscon.com>>:


Hi


One of our relay rsyslog servers has 2 kind of logs: apache and
java. Sadly, *not all java applications share the same log format.*

We have solved this using a bash script to generate our config
which has one input for each file.

To avoid having to reconfigure rsyslog each time a log is created
or an application is added/removed we though about using
wildcards. More or less, this would result in:

   input(
        addMetadata="on"
        file="/logs/apache/*/*"
        persistStateInterval="3"
        ruleset="ruleset"
        tag="apache"
        type="imfile"
   )
   input(
        addMetadata="on"
        file="/logs/java/*/*"
        persistStateInterval="3"
        readTimeout="5"
        ruleset="ruleset"
        startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
        tag="java"
        type="imfile"
   )

But, as stated before, *this would have conflicts with those files
having a different multiline patterns*. For example:

   /logs/java/foo/foo.log
        startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
   /logs/java/bar/bar.log
        startmsg.regex="^[[:alpha:]]{3} [[:digit:]]{2},
[[:digit:]]{4}"
   /logs/java/other/other.log
   startmsg.regex="^[[:digit:]]{2}-[[:digit:]]{2}-[[:digit:]]{4}"
   /logs/java/one/one.log
        startmsg.regex="^[[:alpha:]]{4,5}
   [[:digit:]]{4}/[[:digit:]]{2}/[[:digit:]]{2}"
   /logs/java/two/two.log
   startmsg.regex="^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}"


Is there any option to exclude this files from the wilcard?

Couldn't wilcard accept a regex as pattern? Eg: all files except
those in foo, or bar, or...
    file="/logs/java/((?!(foo|bar|other|one|two)).)*/.*"

Any other solution?

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/professional-services/
<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Exclude patterns for imfile whith different multiline format

[mostolog--- via rsyslog]

good one! ugly globbing to the rescue!

Thanks Rainer


On 17/10/17 16:59, Rainer Gerhards wrote:
You cannot exclude, bit you can include those that you want. I guess 
with ugly globbing that should be possible. Best try with ls first.


http://tldp.org/LDP/abs/html/globbingref.html

Sent from phone, thus brief.

Am 17.10.2017 15:23 schrieb "mostolog--- via rsyslog" 
<rsyslog@lists.adiscon.com <mailto:rsyslog@lists.adiscon.com>>:


Hi


One of our relay rsyslog servers has 2 kind of logs: apache and
java. Sadly, *not all java applications share the same log format.*

We have solved this using a bash script to generate our config
which has one input for each file.

To avoid having to reconfigure rsyslog each time a log is created
or an application is added/removed we though about using
wildcards. More or less, this would result in:

   input(
        addMetadata="on"
        file="/logs/apache/*/*"
        persistStateInterval="3"
        ruleset="ruleset"
        tag="apache"
        type="imfile"
   )
   input(
        addMetadata="on"
        file="/logs/java/*/*"
        persistStateInterval="3"
        readTimeout="5"
        ruleset="ruleset"
        startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
        tag="java"
        type="imfile"
   )

But, as stated before, *this would have conflicts with those files
having a different multiline patterns*. For example:

   /logs/java/foo/foo.log
        startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
   /logs/java/bar/bar.log
        startmsg.regex="^[[:alpha:]]{3} [[:digit:]]{2},
[[:digit:]]{4}"
   /logs/java/other/other.log
   startmsg.regex="^[[:digit:]]{2}-[[:digit:]]{2}-[[:digit:]]{4}"
   /logs/java/one/one.log
        startmsg.regex="^[[:alpha:]]{4,5}
   [[:digit:]]{4}/[[:digit:]]{2}/[[:digit:]]{2}"
   /logs/java/two/two.log
   startmsg.regex="^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}"


Is there any option to exclude this files from the wilcard?

Couldn't wilcard accept a regex as pattern? Eg: all files except
those in foo, or bar, or...
    file="/logs/java/((?!(foo|bar|other|one|two)).)*/.*"

Any other solution?

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/professional-services/
<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Exclude patterns for imfile whith different multiline format

[mostolog--- via rsyslog]

If Mohammed doesn't go to the mountain, the mountain must go to Mohammed.

AKA: That's not possible. It would be easier to make them change the 
format (and, as you know, that's already impossible.)


On 17/10/17 16:43, David Lang wrote:
have the java apps put the different types of logs in different 
directories??


David Lang


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Exclude patterns for imfile whith different multiline format

[mostolog--- via rsyslog]

Hi


One of our relay rsyslog servers has 2 kind of logs: apache and java. 
Sadly, *not all java applications share the same log format.*


We have solved this using a bash script to generate our config which has 
one input for each file.


To avoid having to reconfigure rsyslog each time a log is created or an 
application is added/removed we though about using wildcards. More or 
less, this would result in:


   input(
    addMetadata="on"
    file="/logs/apache/*/*"
    persistStateInterval="3"
    ruleset="ruleset"
    tag="apache"
    type="imfile"
   )
   input(
    addMetadata="on"
    file="/logs/java/*/*"
    persistStateInterval="3"
    readTimeout="5"
    ruleset="ruleset"
    startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
    tag="java"
    type="imfile"
   )

But, as stated before, *this would have conflicts with those files 
having a different multiline patterns*. For example:


   /logs/java/foo/foo.log
    startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
   /logs/java/bar/bar.log
    startmsg.regex="^[[:alpha:]]{3} [[:digit:]]{2}, [[:digit:]]{4}"
   /logs/java/other/other.log
   startmsg.regex="^[[:digit:]]{2}-[[:digit:]]{2}-[[:digit:]]{4}"
   /logs/java/one/one.log
    startmsg.regex="^[[:alpha:]]{4,5}
   [[:digit:]]{4}/[[:digit:]]{2}/[[:digit:]]{2}"
   /logs/java/two/two.log
   startmsg.regex="^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}"


Is there any option to exclude this files from the wilcard?

Couldn't wilcard accept a regex as pattern? Eg: all files except those 
in foo, or bar, or...

    file="/logs/java/((?!(foo|bar|other|one|two)).)*/.*"

Any other solution?

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Does imfile-to-relp need DA queues?

[mostolog--- via rsyslog]

Clear as water.

The idea which was in our minds due to rsyslog issues with RELP (mention 
on another thread) is, by these and Rainer words completely banned since 
now.


Thanks a lot


On 04/10/17 10:23, David Lang wrote:

On Wed, 4 Oct 2017, mostolog--- via rsyslog wrote:

About this: is there any option/planned feature to ensure message has 
been delivered before marking it as processed?


that is what is done, the problem is that it's marked as processed on 
the queue. If the memory is lost (crash, kill -9), the status and even 
existance of messages on the queues are lost.


rsyslog does:

source --> queue --> action (where --> can be a single message or a 
batch of messages)


unless the queue mode is direct, the source and action are 
asynchronous from each other.


If you want each message processed individually, everything slows to a 
crawl.


With a disk based queue (and some very paranoid settings) you can 
force each message read from the source to be committed to disk and 
fsync called (on the file and the directory it's in) before it's 
marked as read on the input. And then the message can be read from the 
disk and processed to the output (such as RELP) before it's marked as 
delivered on disk.


This is safe, but as I said, it resulted in a ~1000x slowdown several 
years ago (and disks are not much faster than that $5000 PCIe SSD I 
was using then), there are just a huge amount of system calls and disk 
I/O to do (and don't use ext3, it resulted in a futher slowdown, it 
doesn't do fsync well)


By default, rsyslog is not that paranoid, keeping logs in RAM and 
flushing them to disk at shutdown is considered pretty severe 
protection of the logs. Normally we don't do fsync calls when writing 
things to disk, we assume that the system isn't going to crash before 
the OS writes things.


Rsyslog isn't intended to handle things like financial transactions 
that must not be lost at any cost. It's designed to handle system and 
application logs in a best effort fashion (that best effort is pretty 
darn good, but we don't insist on perfection)


Does the application that writes the logs that you are intending to 
read do a fsync on the log file and the directory that it's in every 
time it writes a log to the file? if not, you are at risk of loosing 
logs in a system crash just from that. when it creates a new file does 
it check if it needs to sync the parent directory of the directory 
it's in (in case adding the file cause the directory to grow larger)? 
And are you writing this to a RAID array? Does your controller card 
buffer any writes? Does it have a battery backup?


It's very easy to say that the requirement is zero logs lost, but when 
you really dive into the details of what that requires, it gets very 
messy (and expensive)


David Lang


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

[mostolog--- via rsyslog]

No matter if we configure it differently, this parameter is "updated" 
internally.


stats shows that it's stuck at fullDelay and nothing changes from that 
point.


debug doesn't seem to show anything useful.

considering this is our scenario:

    source--->relp--->us--->kafka

which log would you find useful? all? stats? debug?




On 04/10/17 09:38, Rainer Gerhards wrote:

I'll update the doc, but basically this is so that we do not high
highwatermark and leave room for non-delayable messages. You can of
course configure it differently. Nevertheless, the queue should empty,
so this shall not cause anything to stay around...

Rainer

2017-10-03 10:52 GMT+02:00 mosto...@gmail.com <mosto...@gmail.com>:

Full delay mark is set t0 50% of highwatermark, without apparent/documented
reason

https://github.com/rsyslog/rsyslog/blob/a6ae18ee4cb4530e4142a2e98aed404dd3b03346/runtime/queue.c#L2373


On 03/10/17 10:50, Rainer Gerhards wrote:

What exactly do you mean?

Rainer

Sent from phone, thus brief.

Am 03.10.2017 10:27 schrieb "mostolog--- via rsyslog"
<rsyslog@lists.adiscon.com>:

Hmm...wondering if theres is a bug here as:

main_queue(
 queue.filename="main"
 queue.maxdiskspace="4G"
 queue.saveonshutdown="on"
 queue.lowwatermark="3"
 queue.highwatermark="6"
 queue.size="10"
 queue.type="LinkedList"
 queue.fulldelaymark="9"
 queue.lightdelaymark="75000"
 #queue.discardmark="9750"
 #queue.discardseverity="8"
)

results in:

8544.970496200:main thread: main Q: queue.c: params: type 1, enq-only
0, disk assisted 1, spoolDir '/data', maxFileSz 16777216, maxQSize 10,
lqsize 0, pqsize 0, child 0, full delay 3, light delay 75000, deq batch
size 1024, high wtrmrk 6, low wtrmrk 3, discardmrk 98000, max wrkr
1, min msgs f. wrkr 10

8544.970615922:main thread: main Q[DA]: queue.c: params: type 2,
enq-only 0, disk assisted 0, spoolDir '/data', maxFileSz 16777216, maxQSize
0, lqsize 0, pqsize 0, child 1, full delay -1, light delay -1, deq batch
size 8, high wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f.
wrkr 0




On 02/10/17 11:00, mosto...@gmail.com wrote:

Hi

Some time ago we reported issues with RELP /hanging/ and not forwarding
messages until restart.

Last week we were able to replicate and still figuring out *how rsyslog
delay system works*.

According to documentation,

 /queue.fulldelaymark//
 //Number of messages when the queue should block delayable
 messages. Messages are NO LONGER PROCESSED until the queue has
 sufficient space again. If a message is delayable depends on the
 input. For example, messages received via imtcp are delayable
 (because TCP can push back), but those received via imudp are not
 (as UDP does not permit a push back). The intent behind this
 setting is to leave some space in an almost-full queue for
 non-delayable messages, which would be lost if the queue runs out
 of space. Please note that if you use a DA queue, setting the
 fulldelaymark BELOW the highwatermark makes the queue never
 activate disk mode for delayable inputs. So this is probably not
 what you want//.

 queue.lightdelaymark number./

It's our understanding that

   * TCP ack's would be delayed _/a bit/_ if queue.size=lightdelaymark.
   * TCP ack's would be nack's if queue.size=fulldelaymark//.
 //

*Is that correct? Doesit behaves as low/highwatermark instead?* (full
delay when size=fulldelaymark, stops delaying when size is back to
lightdelaymark)


The following configuration:

main_queue(
 queue.filename="main"
 queue.maxdiskspace="1G"
 queue.saveonshutdown="on"
 queue.lowwatermark="15"
 queue.highwatermark="18"
 queue.size="20"
 queue.type="LinkedList"
 queue.fulldelaymark="-1"
 queue.lightdelaymark="-1"
)

Resulted in this debug trace:

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567069705:main thread:
main Q: queue.c: params: type 1, enq-only 0, disk assisted 1, spoolDir
'/data', maxFileSz 16777216, maxQSize 20, lqsize 0, pqsize 0, child 0,
*full delay 9, light delay 14*, deq batch size 1024, high wtrmrk
18, low wtrmrk 15, discardmrk 196000, max wrkr 1, min msgs f. wrkr
20

*Why full is lower than light? *

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567182561:main thread:
main Q[DA]: queue.c: params: type 2, enq-only 0, disk assisted 0, spoolDir
'/data', maxFileSz 16777216, maxQSize 0, lqsize 0, pqsize 0, child 1, full
delay -1, light delay -1, deq batch size 8, high wtrmrk 0, low wtrmrk 1,
discardmrk 0, max wrkr 1, min msgs f. wrkr 0


This other configuration,

 queue.filename="relp"
 queue.maxdisks

Re: [rsyslog] Does imfile-to-relp need DA queues?

[mostolog--- via rsyslog]

As Kafka is a specific platform, which we wouldn't like "clients" to 
use, we were looking for something like this.


1. source reads file from current position
2. source sends to rsyslog *(relp is standarized)*
3. rsyslogs doesn't ack and forwards to kafka
4. kafka acks the message once properly saved
5. rsyslogs ack's to source the message
6. source current position++
7. repeat from #1

Of course, kafka could be replaced with any other reliable target

I don't know if this makes sense for you.


On 04/10/17 09:36, Rainer Gerhards wrote:

2017-10-04 9:34 GMT+02:00 mosto...@gmail.com <mosto...@gmail.com>:

I was trying to make use of kafka redundancy

As Kafka implements a safe mechanisms (copying the message into multiple
nodes), I was wondering if theres a relayed-delivery mode
ie: rsyslog acks the message once kafka has ack'ed the message.

omkafka does this, but to do so it maintains a separate file of
yet-unacked messages.

Rainer




On 04/10/17 09:31, Rainer Gerhards wrote:

2017-10-04 9:29 GMT+02:00 mostolog--- via rsyslog
<rsyslog@lists.adiscon.com>:

Thanks Rainer and David

About this: is there any option/planned feature to ensure message has
been
delivered before marking it as processed?

eg: tell source message has been acknowledged when target has stored the
event
  source--->relp--->rsyslog--->kafka

This would be "as easy as" not moving the reading cursor from files for
imfile status. (devil in details)

as David said, you can achieve this by using the "direct" queue mode.
Of course, performance is bad, but that's always the case if you go to
half-duplex operations.

HTH
Rainer





On 03/10/17 19:38, David Lang wrote:

and if the system crashes, or issues a kill -9 to rsyslog, there will
not
be a chance to save the memory queue and some messages will be lost.

you would have to use a disk queue, or disable queues entirely (direct
mode) to avoid any chance of log loss. This will cripple your
performance
(In a test several years ago, with a high-end PCI SSD, this resulted in
a
1000x slowdown)

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T
LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

[mostolog--- via rsyslog]

Do you want me to open an issue?


On 03/10/17 10:56, mosto...@gmail.com wrote:
Even more...assuming that a full queue will trigger full delays...when 
it would be this delay disabled again?...Cause in our scenario, seems 
the process is stuck forever, until a rsyslog restart.




On 03/10/17 10:52, mosto...@gmail.com wrote:


Full delay mark is set t0 50% of highwatermark, without 
apparent/documented reason


https://github.com/rsyslog/rsyslog/blob/a6ae18ee4cb4530e4142a2e98aed404dd3b03346/runtime/queue.c#L2373


On 03/10/17 10:50, Rainer Gerhards wrote:

What exactly do you mean?

Rainer

Sent from phone, thus brief.

Am 03.10.2017 10:27 schrieb "mostolog--- via rsyslog" 
<rsyslog@lists.adiscon.com <mailto:rsyslog@lists.adiscon.com>>:


Hmm...wondering if theres is a bug here as:

main_queue(
    queue.filename="main"
    queue.maxdiskspace="4G"
    queue.saveonshutdown="on"
    queue.lowwatermark="3"
    queue.highwatermark="6"
    queue.size="10"
    queue.type="LinkedList"
    queue.fulldelaymark="9"
    queue.lightdelaymark="75000"
    #queue.discardmark="9750"
    #queue.discardseverity="8"
)

results in:

8544.970496200:main thread    : main Q: queue.c: params: type 1,
enq-only 0, disk assisted 1, spoolDir '/data', maxFileSz
16777216, maxQSize 10, lqsize 0, pqsize 0, child 0, full
delay 3, light delay 75000, deq batch size 1024, high wtrmrk
6, low wtrmrk 3, discardmrk 98000, max wrkr 1, min msgs
f. wrkr 10

8544.970615922:main thread    : main Q[DA]: queue.c: params:
type 2, enq-only 0, disk assisted 0, spoolDir '/data', maxFileSz
16777216, maxQSize 0, lqsize 0, pqsize 0, child 1, full delay
-1, light delay -1, deq batch size 8, high wtrmrk 0, low wtrmrk
1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0




On 02/10/17 11:00, mosto...@gmail.com
<mailto:mosto...@gmail.com> wrote:

Hi

Some time ago we reported issues with RELP /hanging/ and not
forwarding messages until restart.

Last week we were able to replicate and still figuring out
*how rsyslog delay system works*.

According to documentation,

    /queue.fulldelaymark//
    //Number of messages when the queue should block delayable
    messages. Messages are NO LONGER PROCESSED until the
queue has
    sufficient space again. If a message is delayable
depends on the
    input. For example, messages received via imtcp are
delayable
    (because TCP can push back), but those received via
imudp are not
    (as UDP does not permit a push back). The intent behind this
    setting is to leave some space in an almost-full queue for
    non-delayable messages, which would be lost if the queue
runs out
    of space. Please note that if you use a DA queue,
setting the
    fulldelaymark BELOW the highwatermark makes the queue never
    activate disk mode for delayable inputs. So this is
probably not
    what you want//.

    queue.lightdelaymark number./

It's our understanding that

  * TCP ack's would be delayed _/a bit/_ if
queue.size=lightdelaymark.
  * TCP ack's would be nack's if queue.size=fulldelaymark//.
    //

*Is that correct? Doesit behaves as low/highwatermark
instead?* (full delay when size=fulldelaymark, stops
delaying when size is back to lightdelaymark)


The following configuration:

main_queue(
    queue.filename="main"
    queue.maxdiskspace="1G"
    queue.saveonshutdown="on"
    queue.lowwatermark="15"
    queue.highwatermark="18"
    queue.size="20"
    queue.type="LinkedList"
    queue.fulldelaymark="-1"
    queue.lightdelaymark="-1"
)

Resulted in this debug trace:

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567069705:main
thread    : main Q: queue.c: params: type 1, enq-only 0,
disk assisted 1, spoolDir '/data', maxFileSz 16777216,
maxQSize 20, lqsize 0, pqsize 0, child 0, *full delay
9, light delay 14*, deq batch size 1024, high wtrmrk
18, low wtrmrk 15, discardmrk 196000, max wrkr 1,
min msgs f. wrkr 20

*Why full is lower than light? *

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567182561:main
thread    : main Q[DA]: queue.c: params: type 2, enq-only 0,
disk assisted 0, spoolDir '/data', maxFileSz 16777216,
maxQSize 0, lqsize 0, pqs

Re: [rsyslog] Does imfile-to-relp need DA queues?

[mostolog--- via rsyslog]

I was trying to make use of kafka redundancy

As Kafka implements a safe mechanisms (copying the message into multiple 
nodes), I was wondering if theres a relayed-delivery mode

ie: rsyslog acks the message once kafka has ack'ed the message.



On 04/10/17 09:31, Rainer Gerhards wrote:

2017-10-04 9:29 GMT+02:00 mostolog--- via rsyslog <rsyslog@lists.adiscon.com>:

Thanks Rainer and David

About this: is there any option/planned feature to ensure message has been
delivered before marking it as processed?

eg: tell source message has been acknowledged when target has stored the
event
 source--->relp--->rsyslog--->kafka

This would be "as easy as" not moving the reading cursor from files for
imfile status. (devil in details)

as David said, you can achieve this by using the "direct" queue mode.
Of course, performance is bad, but that's always the case if you go to
half-duplex operations.

HTH
Rainer





On 03/10/17 19:38, David Lang wrote:

and if the system crashes, or issues a kill -9 to rsyslog, there will not
be a chance to save the memory queue and some messages will be lost.

you would have to use a disk queue, or disable queues entirely (direct
mode) to avoid any chance of log loss. This will cripple your performance
(In a test several years ago, with a high-end PCI SSD, this resulted in a
1000x slowdown)

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Does imfile-to-relp need DA queues?

[mostolog--- via rsyslog]

Thanks Rainer and David

About this: is there any option/planned feature to ensure message has 
been delivered before marking it as processed?


eg: tell source message has been acknowledged when target has stored the 
event

    source--->relp--->rsyslog--->kafka

This would be "as easy as" not moving the reading cursor from files for 
imfile status. (devil in details)





On 03/10/17 19:38, David Lang wrote:
and if the system crashes, or issues a kill -9 to rsyslog, there will 
not be a chance to save the memory queue and some messages will be lost.


you would have to use a disk queue, or disable queues entirely (direct 
mode) to avoid any chance of log loss. This will cripple your 
performance (In a test several years ago, with a high-end PCI SSD, 
this resulted in a 1000x slowdown)


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

[mostolog--- via rsyslog]

Even more...assuming that a full queue will trigger full delays...when 
it would be this delay disabled again?...Cause in our scenario, seems 
the process is stuck forever, until a rsyslog restart.




On 03/10/17 10:52, mosto...@gmail.com wrote:


Full delay mark is set t0 50% of highwatermark, without 
apparent/documented reason


https://github.com/rsyslog/rsyslog/blob/a6ae18ee4cb4530e4142a2e98aed404dd3b03346/runtime/queue.c#L2373


On 03/10/17 10:50, Rainer Gerhards wrote:

What exactly do you mean?

Rainer

Sent from phone, thus brief.

Am 03.10.2017 10:27 schrieb "mostolog--- via rsyslog" 
<rsyslog@lists.adiscon.com <mailto:rsyslog@lists.adiscon.com>>:


Hmm...wondering if theres is a bug here as:

main_queue(
    queue.filename="main"
    queue.maxdiskspace="4G"
    queue.saveonshutdown="on"
    queue.lowwatermark="3"
    queue.highwatermark="6"
    queue.size="10"
    queue.type="LinkedList"
    queue.fulldelaymark="9"
    queue.lightdelaymark="75000"
    #queue.discardmark="9750"
    #queue.discardseverity="8"
)

results in:

8544.970496200:main thread    : main Q: queue.c: params: type 1,
enq-only 0, disk assisted 1, spoolDir '/data', maxFileSz
16777216, maxQSize 10, lqsize 0, pqsize 0, child 0, full
delay 3, light delay 75000, deq batch size 1024, high wtrmrk
6, low wtrmrk 3, discardmrk 98000, max wrkr 1, min msgs
f. wrkr 10

8544.970615922:main thread    : main Q[DA]: queue.c: params: type
2, enq-only 0, disk assisted 0, spoolDir '/data', maxFileSz
16777216, maxQSize 0, lqsize 0, pqsize 0, child 1, full delay -1,
light delay -1, deq batch size 8, high wtrmrk 0, low wtrmrk 1,
discardmrk 0, max wrkr 1, min msgs f. wrkr 0




On 02/10/17 11:00, mosto...@gmail.com <mailto:mosto...@gmail.com>
wrote:

Hi

Some time ago we reported issues with RELP /hanging/ and not
forwarding messages until restart.

Last week we were able to replicate and still figuring out
*how rsyslog delay system works*.

According to documentation,

    /queue.fulldelaymark//
    //Number of messages when the queue should block delayable
    messages. Messages are NO LONGER PROCESSED until the
queue has
    sufficient space again. If a message is delayable depends
on the
    input. For example, messages received via imtcp are delayable
    (because TCP can push back), but those received via imudp
are not
    (as UDP does not permit a push back). The intent behind this
    setting is to leave some space in an almost-full queue for
    non-delayable messages, which would be lost if the queue
runs out
    of space. Please note that if you use a DA queue, setting the
    fulldelaymark BELOW the highwatermark makes the queue never
    activate disk mode for delayable inputs. So this is
probably not
    what you want//.

    queue.lightdelaymark number./

It's our understanding that

  * TCP ack's would be delayed _/a bit/_ if
queue.size=lightdelaymark.
  * TCP ack's would be nack's if queue.size=fulldelaymark//.
    //

*Is that correct? Doesit behaves as low/highwatermark
instead?* (full delay when size=fulldelaymark, stops delaying
when size is back to lightdelaymark)


The following configuration:

main_queue(
    queue.filename="main"
    queue.maxdiskspace="1G"
    queue.saveonshutdown="on"
    queue.lowwatermark="15"
    queue.highwatermark="18"
    queue.size="20"
    queue.type="LinkedList"
    queue.fulldelaymark="-1"
    queue.lightdelaymark="-1"
)

Resulted in this debug trace:

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567069705:main
thread    : main Q: queue.c: params: type 1, enq-only 0, disk
assisted 1, spoolDir '/data', maxFileSz 16777216, maxQSize
20, lqsize 0, pqsize 0, child 0, *full delay 9, light
delay 14*, deq batch size 1024, high wtrmrk 18, low
wtrmrk 15, discardmrk 196000, max wrkr 1, min msgs f.
wrkr 20

*Why full is lower than light? *

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567182561:main
thread    : main Q[DA]: queue.c: params: type 2, enq-only 0,
disk assisted 0, spoolDir '/data', maxFileSz 16777216,
maxQSize 0, lqsize 0, pqsize 0, child 1, full delay -1, light
delay -1, deq batch size 8, high 

Re: [rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

[mostolog--- via rsyslog]

Full delay mark is set t0 50% of highwatermark, without 
apparent/documented reason


https://github.com/rsyslog/rsyslog/blob/a6ae18ee4cb4530e4142a2e98aed404dd3b03346/runtime/queue.c#L2373


On 03/10/17 10:50, Rainer Gerhards wrote:

What exactly do you mean?

Rainer

Sent from phone, thus brief.

Am 03.10.2017 10:27 schrieb "mostolog--- via rsyslog" 
<rsyslog@lists.adiscon.com <mailto:rsyslog@lists.adiscon.com>>:


Hmm...wondering if theres is a bug here as:

main_queue(
    queue.filename="main"
    queue.maxdiskspace="4G"
    queue.saveonshutdown="on"
    queue.lowwatermark="3"
    queue.highwatermark="6"
    queue.size="10"
    queue.type="LinkedList"
    queue.fulldelaymark="9"
    queue.lightdelaymark="75000"
    #queue.discardmark="9750"
    #queue.discardseverity="8"
)

results in:

8544.970496200:main thread    : main Q: queue.c: params: type 1,
enq-only 0, disk assisted 1, spoolDir '/data', maxFileSz 16777216,
maxQSize 10, lqsize 0, pqsize 0, child 0, full delay 3,
light delay 75000, deq batch size 1024, high wtrmrk 6, low
wtrmrk 3, discardmrk 98000, max wrkr 1, min msgs f. wrkr 10

8544.970615922:main thread    : main Q[DA]: queue.c: params: type
2, enq-only 0, disk assisted 0, spoolDir '/data', maxFileSz
16777216, maxQSize 0, lqsize 0, pqsize 0, child 1, full delay -1,
light delay -1, deq batch size 8, high wtrmrk 0, low wtrmrk 1,
discardmrk 0, max wrkr 1, min msgs f. wrkr 0




On 02/10/17 11:00, mosto...@gmail.com <mailto:mosto...@gmail.com>
wrote:

Hi

Some time ago we reported issues with RELP /hanging/ and not
forwarding messages until restart.

Last week we were able to replicate and still figuring out
*how rsyslog delay system works*.

According to documentation,

    /queue.fulldelaymark//
    //Number of messages when the queue should block delayable
    messages. Messages are NO LONGER PROCESSED until the queue has
    sufficient space again. If a message is delayable depends
on the
    input. For example, messages received via imtcp are delayable
    (because TCP can push back), but those received via imudp
are not
    (as UDP does not permit a push back). The intent behind this
    setting is to leave some space in an almost-full queue for
    non-delayable messages, which would be lost if the queue
runs out
    of space. Please note that if you use a DA queue, setting the
    fulldelaymark BELOW the highwatermark makes the queue never
    activate disk mode for delayable inputs. So this is
probably not
    what you want//.

    queue.lightdelaymark number./

It's our understanding that

  * TCP ack's would be delayed _/a bit/_ if
queue.size=lightdelaymark.
  * TCP ack's would be nack's if queue.size=fulldelaymark//.
    //

*Is that correct? Doesit behaves as low/highwatermark
instead?* (full delay when size=fulldelaymark, stops delaying
when size is back to lightdelaymark)


The following configuration:

main_queue(
    queue.filename="main"
    queue.maxdiskspace="1G"
    queue.saveonshutdown="on"
    queue.lowwatermark="15"
    queue.highwatermark="18"
    queue.size="20"
    queue.type="LinkedList"
    queue.fulldelaymark="-1"
    queue.lightdelaymark="-1"
)

Resulted in this debug trace:

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567069705:main
thread    : main Q: queue.c: params: type 1, enq-only 0, disk
assisted 1, spoolDir '/data', maxFileSz 16777216, maxQSize
20, lqsize 0, pqsize 0, child 0, *full delay 9, light
delay 14*, deq batch size 1024, high wtrmrk 18, low
wtrmrk 15, discardmrk 196000, max wrkr 1, min msgs f. wrkr
20

*Why full is lower than light? *

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567182561:main
thread    : main Q[DA]: queue.c: params: type 2, enq-only 0,
disk assisted 0, spoolDir '/data', maxFileSz 16777216,
maxQSize 0, lqsize 0, pqsize 0, child 1, full delay -1, light
delay -1, deq batch size 8, high wtrmrk 0, low wtrmrk 1,
discardmrk 0, max wrkr 1, min msgs f. wrkr 0


This other configuration,

        queue.filename="relp"
        queue.maxdiskspace="1G"
        queue.saveonshutdown="on"
        qu

[rsyslog] Does imfile-to-relp need DA queues?

[mostolog--- via rsyslog]

Hi

We're trying to confirm if *imfile-to-relp* is already 100% reliable and 
doesn't need any queue/DA, or if -*in the event of a failure*- messages 
being processed (in memory) will be lost/skipped.


Would a configuration like this (without DA queue in ruleset) ensure all 
file contents are sent to relp server? (In case of error, it will 
continue reading from the last truly-processed line)


   ruleset(
    name="relp"
    ) {
    action(
        name="omrelp"
        action.reportSuspension="on"
        action.reportSuspensionContinuation="on"
        action.resumeInterval="60"
        action.resumeRetryCount="-1"
        port="20514"
        target="server"
        type="omrelp"
    )
   }
   input(
    type="imfile"
    file="file1"
    tag="file1"
    ruleset="relp"
   )

Thanks

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

[mostolog--- via rsyslog]

Hmm...wondering if theres is a bug here as:

main_queue(
    queue.filename="main"
    queue.maxdiskspace="4G"
    queue.saveonshutdown="on"
    queue.lowwatermark="3"
    queue.highwatermark="6"
    queue.size="10"
    queue.type="LinkedList"
    queue.fulldelaymark="9"
    queue.lightdelaymark="75000"
    #queue.discardmark="9750"
    #queue.discardseverity="8"
)

results in:

8544.970496200:main thread    : main Q: queue.c: params: type 1, 
enq-only 0, disk assisted 1, spoolDir '/data', maxFileSz 16777216, 
maxQSize 10, lqsize 0, pqsize 0, child 0, full delay 3, light 
delay 75000, deq batch size 1024, high wtrmrk 6, low wtrmrk 3, 
discardmrk 98000, max wrkr 1, min msgs f. wrkr 10


8544.970615922:main thread    : main Q[DA]: queue.c: params: type 2, 
enq-only 0, disk assisted 0, spoolDir '/data', maxFileSz 16777216, 
maxQSize 0, lqsize 0, pqsize 0, child 1, full delay -1, light delay -1, 
deq batch size 8, high wtrmrk 0, low wtrmrk 1, discardmrk 0, max wrkr 1, 
min msgs f. wrkr 0





On 02/10/17 11:00, mosto...@gmail.com wrote:

Hi

Some time ago we reported issues with RELP /hanging/ and not 
forwarding messages until restart.


Last week we were able to replicate and still figuring out *how 
rsyslog delay system works*.


According to documentation,

/queue.fulldelaymark//
//Number of messages when the queue should block delayable
messages. Messages are NO LONGER PROCESSED until the queue has
sufficient space again. If a message is delayable depends on the
input. For example, messages received via imtcp are delayable
(because TCP can push back), but those received via imudp are not
(as UDP does not permit a push back). The intent behind this
setting is to leave some space in an almost-full queue for
non-delayable messages, which would be lost if the queue runs out
of space. Please note that if you use a DA queue, setting the
fulldelaymark BELOW the highwatermark makes the queue never
activate disk mode for delayable inputs. So this is probably not
what you want//.

queue.lightdelaymark number./

It's our understanding that

  * TCP ack's would be delayed _/a bit/_ if queue.size=lightdelaymark.
  * TCP ack's would be nack's if queue.size=fulldelaymark//.
//

*Is that correct? Doesit behaves as low/highwatermark instead?* 
(full delay when size=fulldelaymark, stops delaying when size is back 
to lightdelaymark)



The following configuration:

main_queue(
    queue.filename="main"
    queue.maxdiskspace="1G"
    queue.saveonshutdown="on"
    queue.lowwatermark="15"
    queue.highwatermark="18"
    queue.size="20"
    queue.type="LinkedList"
    queue.fulldelaymark="-1"
    queue.lightdelaymark="-1"
)

Resulted in this debug trace:

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567069705:main thread    : 
main Q: queue.c: params: type 1, enq-only 0, disk assisted 1, spoolDir 
'/data', maxFileSz 16777216, maxQSize 20, lqsize 0, pqsize 0, 
child 0, *full delay 9, light delay 14*, deq batch size 1024, 
high wtrmrk 18, low wtrmrk 15, discardmrk 196000, max wrkr 1, 
min msgs f. wrkr 20


*Why full is lower than light? *

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567182561:main thread    : 
main Q[DA]: queue.c: params: type 2, enq-only 0, disk assisted 0, 
spoolDir '/data', maxFileSz 16777216, maxQSize 0, lqsize 0, pqsize 0, 
child 1, full delay -1, light delay -1, deq batch size 8, high wtrmrk 
0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0



This other configuration,

        queue.filename="relp"
        queue.maxdiskspace="1G"
        queue.saveonshutdown="on"
        queue.lowwatermark="10"
        queue.highwatermark="60"
        queue.size="100"
        queue.type="LinkedList"
        queue.fulldelaymark="500"
        queue.lightdelaymark="500"

resulted in:

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.566625167:main thread    : 
relp: queue.c: params: type 1, enq-only 0, disk assisted 1, spoolDir 
'/data', maxFileSz 16777216, maxQSize 100, lqsize 0, pqsize 0, 
child 0, *full delay 30, light delay 70*, deq batch size 1024, 
high wtrmrk 60, low wtrmrk 10, discardmrk 98, max wrkr 1, 
min msgs f. wrkr 100


*What are the valid values for delay marks?*
*If a full delay behave as closing the socket...when it would be set 
to normal again?*


Sep 29 13:07:22 host11 rsyslog[57150]: 3242.566787556:main thread    : 
relp[DA]: queue.c: params: type 2, enq-only 0, disk assisted 0, 
spoolDir '/data', maxFileSz 16777216, maxQSize 0, lqsize 0, pqsize 0, 
child 1, full delay -1, light delay -1, deq batch size 8, high wtrmrk 
0, low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0




I think this doubts would be quite useful for other people having this 
kind of issues.


Regards




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog

[rsyslog] How rsyslog delay system works and occasinal (reproducible) RELP hangs

[mostolog--- via rsyslog]

Hi

Some time ago we reported issues with RELP /hanging/ and not forwarding 
messages until restart.


Last week we were able to replicate and still figuring out *how rsyslog 
delay system works*.


According to documentation,

   /queue.fulldelaymark//
   //Number of messages when the queue should block delayable messages.
   Messages are NO LONGER PROCESSED until the queue has sufficient
   space again. If a message is delayable depends on the input. For
   example, messages received via imtcp are delayable (because TCP can
   push back), but those received via imudp are not (as UDP does not
   permit a push back). The intent behind this setting is to leave some
   space in an almost-full queue for non-delayable messages, which
   would be lost if the queue runs out of space. Please note that if
   you use a DA queue, setting the fulldelaymark BELOW the
   highwatermark makes the queue never activate disk mode for delayable
   inputs. So this is probably not what you want//.

   queue.lightdelaymark number./

It's our understanding that

 * TCP ack's would be delayed _/a bit/_ if queue.size=lightdelaymark.
 * TCP ack's would be nack's if queue.size=fulldelaymark//.
   //

*Is that correct? Doesit behaves as low/highwatermark instead?* 
(full delay when size=fulldelaymark, stops delaying when size is back to 
lightdelaymark)



The following configuration:

main_queue(
    queue.filename="main"
    queue.maxdiskspace="1G"
    queue.saveonshutdown="on"
    queue.lowwatermark="15"
    queue.highwatermark="18"
    queue.size="20"
    queue.type="LinkedList"
    queue.fulldelaymark="-1"
    queue.lightdelaymark="-1"
)

Resulted in this debug trace:

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567069705:main thread    : 
main Q: queue.c: params: type 1, enq-only 0, disk assisted 1, spoolDir 
'/data', maxFileSz 16777216, maxQSize 20, lqsize 0, pqsize 0, child 
0, *full delay 9, light delay 14*, deq batch size 1024, high 
wtrmrk 18, low wtrmrk 15, discardmrk 196000, max wrkr 1, min 
msgs f. wrkr 20


*Why full is lower than light? *

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.567182561:main thread    : 
main Q[DA]: queue.c: params: type 2, enq-only 0, disk assisted 0, 
spoolDir '/data', maxFileSz 16777216, maxQSize 0, lqsize 0, pqsize 0, 
child 1, full delay -1, light delay -1, deq batch size 8, high wtrmrk 0, 
low wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0



This other configuration,

        queue.filename="relp"
        queue.maxdiskspace="1G"
        queue.saveonshutdown="on"
        queue.lowwatermark="10"
        queue.highwatermark="60"
        queue.size="100"
        queue.type="LinkedList"
        queue.fulldelaymark="500"
        queue.lightdelaymark="500"

resulted in:

Sep 29 13:07:22 host11 rsyslog[57150]: 3242.566625167:main thread    : 
relp: queue.c: params: type 1, enq-only 0, disk assisted 1, spoolDir 
'/data', maxFileSz 16777216, maxQSize 100, lqsize 0, pqsize 0, child 
0, *full delay 30, light delay 70*, deq batch size 1024, high 
wtrmrk 60, low wtrmrk 10, discardmrk 98, max wrkr 1, min 
msgs f. wrkr 100


*What are the valid values for delay marks?*
*If a full delay behave as closing the socket...when it would be set to 
normal again?*


Sep 29 13:07:22 host11 rsyslog[57150]: 3242.566787556:main thread    : 
relp[DA]: queue.c: params: type 2, enq-only 0, disk assisted 0, spoolDir 
'/data', maxFileSz 16777216, maxQSize 0, lqsize 0, pqsize 0, child 1, 
full delay -1, light delay -1, deq batch size 8, high wtrmrk 0, low 
wtrmrk 1, discardmrk 0, max wrkr 1, min msgs f. wrkr 0




I think this doubts would be quite useful for other people having this 
kind of issues.


Regards


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] feedback request: debug log improvements

[mostolog--- via rsyslog]

My 2 cents:

As Rsyslog is used to handle several log formats, and encourage to use 
them, it would be great it also use a well-known format.
Adding timestamps, caller (which component produces the log), thread-id 
or an identifier to "trace" an event going from input to output (print 
message id) would be great.


Perhaps I would test a depth value instead of a loglevel. Enabling 
selective debug (specific modules only) would also be great.


Regards.


On 14/08/17 10:59, Jan Gerhards wrote:

2017-08-12 23:39 GMT+02:00 Joe Blow via rsyslog :

Is the output already in JSON? If not it'd be great to have something that
you could index and query directly.

Right now the output isn't in JSON. It is only accessible in the
debug-log and only if debugging is enabled. However, I'm unsure about
the benefits of having it in JSON format, since the messages
themselves have no structure. The only index-able parts would be the
(relative) timestamp, the thread and the file in which the output is
generated.

Because of this, I don't see many uses for debug-output in JSON,
especially taking into consideration that the most important parts are
the message itself. However, I would also be interested in hearing
your opinion about the possible cases where JSON could be used.

Jan


Cheers,

JB

On Fri, Aug 11, 2017 at 9:13 AM, Jan Gerhards  wrote:


Hi all,

I am working on a refactoring of the debug output generated by rsyslog
to increase usability not only for developers, but also for end users.

For more information see
 https://www.linkedin.com/pulse/improving-rsyslog-debug-
output-jan-gerhards

As of rsyslog version 8.29 I have started by implementing first
configuration options. As mentioned in the article, I would also be
glad to hear from you about your ideas.

Jan

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Imfile module parameters not known

[mostolog--- via rsyslog]

you have two ")" at the end.. also 2 StateFile...review your config


On 21/07/17 15:06, Luv via rsyslog wrote:

This is the way I am reading logs from file,

input(type="imfile" File="/var/log/nginx/infotrack_access.log.5.gz"
StateFile="/var/spool/rsyslog/statefile1"
ruleset="nginxoldruleset"
Tag="nginx"
StateFile="/var/spool/rsyslog/statefile1"
Severity="info"
Facility="local7")
)



But I am getting these errors,

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'ruleset' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'Facility' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'Severity' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'StateFile' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'Tag' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]

Jul 21 18:30:58 AlphaServer rsyslogd: error during parsing file
/etc/rsyslog.d/11-alpha-elastic-old-logs.conf, on or before line 12:
parameter 'File' not known -- typo in config file? [v8.27.0 try
http://www.rsyslog.com/e/2207 ]



It is saying that none of these parameters are known. Why so ?

Here are the 2 links I read,

http://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html

http://www.rsyslog.com/using-the-text-file-input-module/



--
View this message in context: 
http://rsyslog-users.1305293.n2.nabble.com/Imfile-module-parameters-not-known-tp7592623.html
Sent from the rsyslog-users mailing list archive at Nabble.com.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] If a ruleset is bound to a specific input, will rsyslog check a message arriving on that input against a different ruleset?

[mostolog--- via rsyslog]

It's implicit, AFAIK


On 11/07/17 07:48, deoren wrote:

Here is some pseducode based off of another recent thread:

ruleset(name="remote-rules"){
action(
...
)
action(
...
)
stop
}

input(type="imudp" port="1514" address="127.0.0.1" 
ruleset="remote-rules")
input(type="imptcp" port="1514" address="127.0.0.1" 
ruleset="remote-rules")


I see here that the stop directive is used as the last item within 
that ruleset. Is that necessary? I had the idea (evidently mistaken) 
that when you assign a ruleset to an input that only that ruleset 
would be applied to messages arriving on that input.


Is the stop directive necessary here?

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] How to downgrade rsyslog from 2.7 to 2.4

[mostolog--- via rsyslog]

This Dockerfile may help you building whatever rsyslog you need.

FROM ubuntu:16.04
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y curl dnsutils vim && apt-get 
clean && rm -rf /var/lib/apt/lists/*
RUN apt-get update && apt-get install -y software-properties-common && 
add-apt-repository -y ppa:adiscon/v8-stable && apt-get update && apt-get 
install -y build-essential pkg-config libestr-dev libfastjson-dev 
zlib1g-dev liblogging-stdlog-dev librelp-dev liblognorm-dev 
libcurl4-gnutls-dev librdkafka-dev curl && apt-get clean && rm -rf 
/var/lib/apt/lists/*

ARG version=8.27.0
RUN curl -sO 
http://www.rsyslog.com/files/download/rsyslog/rsyslog-${version}.tar.gz 
&& tar -xzf rsys*.tar.gz && rm rsys*.tar.gz && mv rsyslog*/ /rsyslog

WORKDIR /rsyslog
RUN ./configure --prefix=/usr --enable-debug --enable-impstats 
--enable-omkafka --enable-relp --enable-imfile --enable-mmjsonparse 
--enable-mmnormalize --enable-elasticsearch --disable-klog 
--disable-uuid --disable-libgcrypt && make && make install && rm -fr 
/rsyslog

CMD ["/rsyslog.sh"]



On 06/07/17 13:53, Muhammad Asif via rsyslog wrote:

Hi Guys,

Mistakenly i have upgraded my rsyslog from 2.4 to 2.7. Now problem is that
my rsyslog is not sending logs to remote server. There is no error. How can
i downgrade rsyslog from 2.7 to 2.4. Is there any link of deb packages of
rsyslog builds with previous versions. Compiling from source is a bit
difficult.

Regards
M. Asif
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog going down frequently

[mostolog--- via rsyslog]

impstats will show yo how many queues you have, and you can figure it 
out how much memory they are taking.


Each queue you use, increases the memory footprint.


On 29/06/17 11:12, Luv via rsyslog wrote:

I am sending data to elasticsearch via rsyslog.

I have been successful in that. But from yesterday, rsyslog is going down.
There was no problem in elasticsearch server, but rsyslog went down a couple
of times,

alphauser@AlphaServer:~$ dmesg | grep -A1 -B1 "Killed process" | grep
"rsyslog"
[3829358.159352] Out of memory: Kill process 8340 (rsyslogd) score 621 or
sacrifice child
[3829358.183579] Killed process 8340 (rsyslogd) total-vm:1577504kB,
anon-rss:1064096kB, file-rss:0kB
[3829358.495953] init: rsyslog main process (8340) killed by KILL signal
[3829822.288212] Out of memory: Kill process 10452 (rsyslogd) score 436 or
sacrifice child
[3829822.304238] Killed process 10452 (rsyslogd) total-vm:1315360kB,
anon-rss:747232kB, file-rss:0kB
[3829822.518360] init: rsyslog main process (10452) killed by KILL signal
[3830138.346746] Out of memory: Kill process 11990 (rsyslogd) score 377 or
sacrifice child
[3830138.369427] Killed process 11990 (rsyslogd) total-vm:1184288kB,
anon-rss:646832kB, file-rss:0kB
[3830138.748289] init: rsyslog main process (11990) killed by KILL signal
[3830608.796438] Out of memory: Kill process 13230 (rsyslogd) score 620 or
sacrifice child
[3830608.810256] Killed process 13230 (rsyslogd) total-vm:1643040kB,
anon-rss:1062824kB, file-rss:0kB
[3830609.161384] init: rsyslog main process (13230) killed by KILL signal
[3830848.524858] Out of memory: Kill process 15113 (rsyslogd) score 290 or
sacrifice child
[3830848.538968] Killed process 15113 (rsyslogd) total-vm:1053216kB,
anon-rss:497632kB, file-rss:0kB
[3830848.700492] init: rsyslog main process (15113) killed by KILL signal
[3831401.272656] Out of memory: Kill process 16116 (rsyslogd) score 621 or
sacrifice child
[3831401.287363] Killed process 16116 (rsyslogd) total-vm:1577504kB,
anon-rss:1064088kB, file-rss:0kB
[3831401.784038] init: rsyslog main process (16116) killed by KILL signal


It is telling me that it is going out of memory. I used rsyslog because it
is light and fast, but even this is getting a large OOM score which I did
not expect.
What is the possible reason ? How can I avoid it ?



--
View this message in context: 
http://rsyslog-users.1305293.n2.nabble.com/rsyslog-going-down-frequently-tp7592527.html
Sent from the rsyslog-users mailing list archive at Nabble.com.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] RELP does not resume when target goes back after a few hours down

[mostolog--- via rsyslog]

According to http://www.rsyslog.com/doc/v8-stable/configuration/actions.html

/If multiple retires fail, the interval is automatically extended. This 
is to prevent excessive ressource use for retires. After each 10 
retries, the interval is extended by itself. To be precise, the actual 
interval is (numRetries / 10 + 1) * $ActionResumeInterval. so after the 
10th try, it by default is 60 and after the 100th try it is 330./


Does this mean this interval could grow until overflow? Is this value a 
"per-message" setting, or it is shared for the qhole queue? How this 
will behave on a high-throughput queue?


Going to enable debug, trying to see something.

Any other feedback would be highly appreciated.



On 12/06/17 10:13, Peter Viskup wrote:

Check the rsyslog error messages on "action 'NAME' suspended, next
retry is" the next message should be "action 'NAME' resumed".
The options $ActionResumeInterval and $ActionResumeRetryCount needs to
be configured according your expectations.

More information in Documentation:
http://www.rsyslog.com/doc/rsconf1_actionresumeinterval.html
http://www.rsyslog.com/doc/v8-stable/tutorials/reliable_forwarding.html



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] RELP does not resume when target goes back after a few hours down

[mostolog--- via rsyslog]

Hi

We have been running a RELP->KAFKA infrastructure for a few weeks now, 
and this tests allowed us to detect issues and problems on our 
processing pipeline.


Before planning imkafka tests and deployment, we have been making some 
fault-tolerant tests and we have observed the same undesired behavior 
every time kafka goes down for a few hours on a high-traffic/throughput 
queue: rsyslog /forwarder/ is not resuming sending at all (even hour 
later) when kafka goes back online, until a rsyslog restart.


If I understood correctly, relp will retry forever if /resumeretrycount/ 
is set to -1, but increasing wait periods between calls. Is that right?
What's the maximum retry-timeout value? Is that value printed on 
impstats? Can that value be set in config? Do you think this could be 
unrelated to our issue?


Will enabling log (on demand debug) would show what's going on here?

We are using 8.25 on forwarder and 8.27 on receiver. Disk assisted 
queues on both, with a discard limit on origin. Here are the configs:


Forwarder:

   set $!appname = "group/unknown" ;
   template(name="ajsontemplate" type="string"
   string="<%pri%>%timegenerated:::date-rfc3339% %$myhostname%
   %$!appname%: %$!data%"
   )
   ruleset(name="fwdto"
   queue.spoolDirectory="/var/spool/rsyslog"
   queue.type="LinkedList"
   queue.filename="ruleset_forward"
   queue.maxdiskspace="768M"
   queue.saveonshutdown="on"
   queue.size="100"
   queue.discardmark="90"
   queue.highwatermark="60"
   queue.lowwatermark="10"
   queue.discardseverity="5"
   ) {
  set $!data!msg=$rawmsg;
  action(
   name="fwdto_action"
   type="omrelp"
   action.resumeRetryCount="-1"
   target="receiver"
   port="20514"
   template="ajsontemplate"
  )
   }

Receiver:

   input(
   port="20514"
   type="imrelp"
   name="imrelp"
   ruleset="relp"
   )

   ruleset(
   name="relp"
   #queue.filename="relp" (no borrar esta línea, se usa en rsyslog.sh
   para usar una cola diferente en cada nodo)
   queue.maxdiskspace="1G"
   queue.saveonshutdown="on"
   queue.lowwatermark="10"
   queue.highwatermark="60"
   queue.size="100"
   queue.type="LinkedList"
) {
set $!host_received=$$myhostname;
set $!time_received=exec_template("time_received");
set $!host_forwarded=$hostname; #$fromhost
set $!time_forwarded=exec_template("time_reported");
# TODO permitSlashInProgramname (v8.25)
set $!group=field($syslogtag,"/",1);
set $!app=field($syslogtag,"/",2);
set $!app=replace($!app,":","");
set $!period=$$now;
set $.type="logs";
set $!time_generated=exec_template("time_reported");
action(
   name="json"
   cookie=""
   type="mmjsonparse"
)
if $parsesuccess != "OK" then {
   action(
   name="error"
   type="omfile"
   file="/logs/rsyslog-errors.log"
)
   stop
}
action(
   name="kafka"
   action.resumeRetryCount="-1"
   action.reportsuspension="on"
   #action.reportSuspensionContinuation="on"
   type="omkafka"
   broker=["kafka:9092"]
   dynatopic="on"
   topic="topic"
   partitions.auto="on"
   dynatopic.cachesize="300"
   template="json"
   errorFile="/logs/rsyslog-kafka.json"
)
   }


Any idea? Shall I file a new issue?
Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Configuration for rsyslogd to send logs to remote server when it will have internet and after system reboot

[mostolog--- via rsyslog]

Hi Ruben, this is more or less what I would do:

 * Each device logs into a file using omfile (this could be avoided if
   using /var/log/messages)
 * Theres a imfile which always retry and uses ruleset to forward
   (using omrelp)
 * My omrelp would just read and retry without a queue (or queue
   size=1?), cause it would be always "readable from file"

David, that should work, isnt it?


On 08/06/17 01:56, David Lang wrote:

you never gave it a filename to use for the disk queue by the way.

On Wed, 7 Jun 2017, Ruben Papovyan wrote:


Date: Wed, 7 Jun 2017 16:53:33 -0700
From: Ruben Papovyan 
To: David Lang 
Cc: Ruben Papovyan via rsyslog 
Subject: Re: [rsyslog] Configuration for rsyslogd to send logs to remote
server when it will have internet and after system reboot

I read it,
I changed my config to this one
But it not working the way I want. (It not sending logs after reboot and
after disconnect and connect to network )

$template ProxiesTemplate,"%msg%\n"
$MainMsgQueueSaveOnShutdown on
$MainMsgQueueType Disk
$MainMsgQueueSyncQueueFiles on
local4.* @@52.44.103.241:3345;ProxiesTemplate


On 7 June 2017 at 16:14, David Lang  wrote:

start by reading 
http://www.rsyslog.com/doc/v8-stable/concepts/queues.html

and http://www.rsyslog.com/doc/v8-stable/rainerscript/queue_para
meters.html







___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] The long way to OC questions

[mostolog--- via rsyslog]

Sorry, wrong list :S


On 06/06/17 14:07, mosto...@gmail.com wrote:


Hi.

Straight to the topic. Thanks in advance.

  * Does Opencast ingest (or Galicaster) splits a video into several
segments, in order to process each part by different workers,
hence increasing process speed? Otherwise, would it make sense?
how could this be done?

  * As we are interested in faster processing, we thought to ingest,
encode and do "as most as possible" with locally attached disk,
and store/publish final videos on NFS. If I understood properly,
this configuration could work. Am I right?

org.opencastproject.storage.dir=/data/nfs-path
org.opencastproject.episode.rootdir=/data/nfs-path
org.opencastproject.workspace.rootdir=/data/local-path

org.opencastproject.staticfiles.rootdir=${org.opencastproject.storage.dir}/staticfiles
org.opencastproject.file.repo.path= ???

  * According to list comments, multiple engage nodes requiere a
separate solr node. As stated in
https://docs.opencast.org/latest/admin/modules/searchindex/:

/It is important to understand that a solr server provides exactly
one schema, and one schema only. If you want to cluster both the
workflow service and the search service, you will need two
separate solr servers. These solr servers can run on the same
machine, but each will needs its own servlet container and port./
//

  * Does that mean I should have a separate solr server for each of
the following?

#org.opencastproject.solr.dir=${karaf.data}/solr-indexes
#org.opencastproject.search.solr.url=http://solr:8983/solr/

#org.opencastproject.archive.solr.dir=${org.opencastproject.solr.dir}/archive

#org.opencastproject.scheduler.solr.dir=${org.opencastproject.solr.dir}/scheduler
#org.opencastproject.search.solr.dir=${org.opencastproject.solr.dir}/search
#org.opencastproject.series.solr.dir=${org.opencastproject.solr.dir}/series

#org.opencastproject.workflow.solr.dir=${org.opencastproject.solr.dir}/workflow

  * Do I have to copy

matterhorn/modules/matterhorn-search-service-impl/src/main/resources/solr/conf,
and

matterhorn/modules/matterhorn-workflow-service-impl/src/main/resources/solr/conf
files to solr servers?
Please, observe documentation is explaining how to deploy a solr,
instead of how opencast should be linked to a solr server (which,
IMHO, might do)

  * Why custom_properties contains a
/org.opencastproject.elasticsearch.config.dir/ property? wasn't
opencast using solr already?

  * Sorry to bother, but I still haven't figured out the proper values
for a multi-service-node environment. Please, confirm they are ok:

org.opencastproject.organization-mh_default_org.cfg
server=domain.com
prop.org.opencastproject.admin.ui.url=http://admin:8080
prop.org.opencastproject.engage.ui.url=http://engage:8080
custom.properties
org.opencastproject.server.url=https://domain.com #please, note
the httpS
org.opencastproject.jobs.url=engage???

Actually, these aren't working for LTI, but that's another issue.


For sure, more to come...
Regards



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Installing 8.27 from repo in debian stretch

[mostolog--- via rsyslog]

Following http://www.rsyslog.com/debian-repository/ results in:

...
Executing: /tmp/apt-key-gpghome.zwvA0EHp4N/gpg.1.sh --recv-keys 
--keyserver keys.gnupg.net AEF0CF8E
gpg: key 894ECF17AEF0CF8E: public key "Andre Lorbach (Key for signing 
deb packages) " imported

gpg: Total number processed: 1
gpg:   imported: 1
...
W: GPG error: http://debian.adiscon.com/v8-stable wheezy/ Release: The 
following signatures were invalid: 1362E120FE08D280780169DC894ECF17AEF0CF8E
W: The repository 'http://debian.adiscon.com/v8-stable wheezy/ Release' 
is not signed.


Anyone could help?

Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] drop messages without timestamp

[mostolog--- via rsyslog]

Perhaps I didn't understood properly and what you really need is 
startmsg.regex="WHATEVER" ?


rsyslog will consider the message is part of the previous if it doesn't 
start with WHATEVER.



El 24/04/17 a las 18:46, Tim Mori via rsyslog escribió:

Not sure where I would set this. The logs are broken on the ESX hosts and 
getting log forwarding on ESX configured is not very straightforward, but I can 
check on it. At first glance, I'm not seeing this being documented anywhere, 
but that isn't surprising.

Thanks,

Tim



-Original Message-
From: rsyslog [mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of 
mostolog--- via rsyslog
Sent: Monday, April 24, 2017 10:05 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: mosto...@gmail.com
Subject: Re: [rsyslog] drop messages without timestamp

FYI: Java stack traces tend to be quite long, and a few weeks ago we had to 
increase maxmessagesize to 64KB. Would that be enough for your needs?


El 21/04/17 a las 09:18, David Lang escribió:

unless it's a massive log message, the best thing to do is probably
increase maxmessagesize on the receiving systems so that they can
handle the full log message.

David Lang

On Tue, 11 Apr 2017, Tim Mori via rsyslog wrote:


Date: Tue, 11 Apr 2017 21:24:11 +
From: Tim Mori via rsyslog <rsyslog@lists.adiscon.com>
To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
Cc: Tim Mori <tim.m...@sas.com>
Subject: [rsyslog] drop messages without timestamp

I’m trying to work out a problem with the logs from our ESX servers.
It seems for one of the logs, the message can overrun some maximum
and it dumps the remaining part of the message on to the next line.
This is a problem because these message fragments break the
organization set via dynafile and of course they don’t parse.

The only thing I can really match on is that the fragment doesn’t
lead with a timestamp.

I’m wondering if there’s any easy way to drop this message. In
writing this out, I’m not sure basing it on timestamp would be the
best idea as some messages may not immediately start with one. I’m
not sure I can use a variable like timestamp as a condition though.

Tim Mori
SAS Solutions OnDemand
Systems Engineer ▪ Tel: + 1 919 531 1774 ▪
tim.m...@sas.com<mailto:tim.m...@sas.com>
100 SAS Campus Drive ▪ Cary ▪ NC ▪ 27513-2414
www.sas.com<http://www.sas.com>


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] drop messages without timestamp

[mostolog--- via rsyslog]

FYI: Java stack traces tend to be quite long, and a few weeks ago we had 
to increase maxmessagesize to 64KB. Would that be enough for your needs?



El 21/04/17 a las 09:18, David Lang escribió:
unless it's a massive log message, the best thing to do is probably 
increase maxmessagesize on the receiving systems so that they can 
handle the full log message.


David Lang

On Tue, 11 Apr 2017, Tim Mori via rsyslog wrote:


Date: Tue, 11 Apr 2017 21:24:11 +
From: Tim Mori via rsyslog 
To: "rsyslog@lists.adiscon.com" 
Cc: Tim Mori 
Subject: [rsyslog] drop messages without timestamp

I’m trying to work out a problem with the logs from our ESX servers. 
It seems for one of the logs, the message can overrun some maximum 
and it dumps the remaining part of the message on to the next line. 
This is a problem because these message fragments break the 
organization set via dynafile and of course they don’t parse.


The only thing I can really match on is that the fragment doesn’t 
lead with a timestamp.


I’m wondering if there’s any easy way to drop this message. In 
writing this out, I’m not sure basing it on timestamp would be the 
best idea as some messages may not immediately start with one. I’m 
not sure I can use a variable like timestamp as a condition though.


Tim Mori
SAS Solutions OnDemand
Systems Engineer ▪ Tel: + 1 919 531 1774 ▪ 
tim.m...@sas.com

100 SAS Campus Drive ▪ Cary ▪ NC ▪ 27513-2414
www.sas.com


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT 
POST if you DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Send logs in JSON format to Elasticsearch

[mostolog--- via rsyslog]

I dont know if this is what you are talking

http://www.rsyslog.com/doc/master/configuration/modules/mmjsonparse.html

BTW: whats your rsyslog omelasticsearch index rate?
Last test I made (with a basic configuration) we got ~10k/min only with 
latest ES version (perhaps its not fully compatible).


El 23/02/17 a las 00:42, Alec Swan via rsyslog escribió:

Hi there,

We've been using librlognorm templates to parse log files and send their
content to Elasticsearch as JSON. At this point we need to parse logs that
are already in JSON format. So, I am wondering if there is a way to write a
liblognorm template that would create a JSON object from each log line? Or
is there a better way to populate $!all-json variable?

Thanks,

Alec
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] RELP suspendDuration went far too large?

[mostolog--- via rsyslog]

El 14/02/17 a las 16:00, Rainer Gerhards escribió:

Search on "debug on demand" should get you useful results.

Great! Doing it as we speak!


I'd also suggest to review the action () parameters.

Anything in particular?


there is a setting to have it log suspend messages, that gives you the 
duration. It's something that goes in the main() section. I'd have to
dig to find it though.

something like logsuspend=yes
DebugOnDemand seems to be perfect, anyway I wasn't able to found what 
you said.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Am I missing something?

[mostolog--- via rsyslog]

Isn't that configuration option used to allow "slash in 
syslogtag/appname/programname field"? why is then called ~hostname?



El 14/02/17 a las 17:33, Rainer Gerhards escribió:

I don't understand the question.

RG


2017-02-14 16:46 GMT+01:00 mostolog--- via rsyslog 
<rsyslog@lists.adiscon.com <mailto:rsyslog@lists.adiscon.com>>:


Why permitSlashInHostname is not permitSlashInTag instead?

From:
http://www.rsyslog.com/doc/master/configuration/properties.html
<http://www.rsyslog.com/doc/master/configuration/properties.html>

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/professional-services/
<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Am I missing something?

[mostolog--- via rsyslog]

Why permitSlashInHostname is not permitSlashInTag instead?

From: http://www.rsyslog.com/doc/master/configuration/properties.html

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] RELP suspendDuration went far too large?

[mostolog--- via rsyslog]

This *could* also be related to https://github.com/rsyslog/librelp/pull/54
which I am about to merge soon and which will be released alongside rsyslog
8.25.

Will test 8.25 and let you know if it's solved.



there is a setting to have it log suspend messages, that gives you the
duration. It's something that goes in the main() section. I'd have to dig
to find it though.
If you could give me a hint, I could try to guess what you refer by my 
own...





Shall suspendDuration have a maximum value?

I'll open an issue.


just a note, you can define the debug file in the config and then turn
debug on and off while running, no overhead when it's off.

Do you mean:
export RSYSLOG_DEBUGLOG="/var/spool/rsyslog/debug.log"
export RSYSLOG_DEBUG=""

Thanks
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] RELP suspendDuration went far too large?

[mostolog--- via rsyslog]

Hi


While using RELP to forward messages to a central rsyslog, we had a full 
disk issue on target.


After cleaning the disk and recovering space, relay rsyslog seems is not 
forwarding traffic as it should (full speed!).


Messages with priority 12 still arrive, and impstats shows:

Feb 14 11:22:21 relay rsyslogd-pstats: fwdtocentral: origin=core.queue 
size=90 enqueued=107845543 full=0 discarded.full=0 
discarded.nf=41435274 maxqsize=90138



Could it be suspendDuration went too high after 2 days with high 
traffic(60k/min)?


Is there any way to query suspendDuration value while running? (without 
DEBUG!)


Does it suspendDuration gets decreased when connection is recovered? (I 
won't say that after reading action.c)


Shall suspendDuration have a maximum value?

Will HUP signal reset suspendDuration and start forwarding normally?

Is there anything I could test to help fixing this (if it's a bug)?

Regards


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Using liblognorm rule tags

[mostolog--- via rsyslog]

Seem that works. Thanks

El 09/02/17 a las 15:32, David Lang escribió:

On Thu, 9 Feb 2017, mostolog--- via rsyslog wrote:


Hi

While working with liblognorm we have found the /need/ of *using rule 
tags*.


However setting */rule=aa:%.:@syslog% Foo%message:rest%/* doesn't 
seem to add a /tags/ field to the message:


  <12>2017-02-09T13:32:34.884+01:00 computer tag:
  FooWhateverFollowsREDACTED

although it seems to parse it properly:

  { "message": "WhateverFollowsREDACTED", "syslogtag": "tag",
  "hostname": "computer", "date": "2017-02-09T13:32:34.884+01:00",
  "priority": "12"

Where is the metadata/tags field which is supposed to contain rule 
tags? $.tags? $.metadata? Is that documented somewhere?


I believe that it creates $!event.tags

if you are running lognormalize, you need to add a flag (-T IIRC) to 
get the tags shown in the output. I don't remember off the top of my 
head if this needs to be enabled via a config option for mmnormalize.


David Lang


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Using liblognorm rule tags

[mostolog--- via rsyslog]

Hi

While working with liblognorm we have found the /need/ of *using rule tags*.

However setting */rule=aa:%.:@syslog% Foo%message:rest%/* doesn't seem 
to add a /tags/ field to the message:


   <12>2017-02-09T13:32:34.884+01:00 computer tag:
   FooWhateverFollowsREDACTED

although it seems to parse it properly:

   { "message": "WhateverFollowsREDACTED", "syslogtag": "tag",
   "hostname": "computer", "date": "2017-02-09T13:32:34.884+01:00",
   "priority": "12"

Where is the metadata/tags field which is supposed to contain rule tags? 
$.tags? $.metadata? Is that documented somewhere?



Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Rsyslog imkafka

[mostolog--- via rsyslog]

Ping!


El 20/01/17 a las 15:19, mosto...@gmail.com escribió:


Hi.

Just started deploying kafka for testing with omkafka. Have anyone 
considered doing an *imkafka* module?


So far, we have:

  * RELP->rsyslog->omelasticsearch->elasticsearch
  * RELP->rsyslog->omkafka

Otherwise, I'll go for logstash.

Regards



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] working example of omkafka?

[mostolog--- via rsyslog]

Finally managed to get it working, although not fully working :(

First: rsyslog wasn't adding topics to kafka, cause I was using "@" 
within topic names, and that's an unsupported character, and maybe 
because I omitted :9092


Second: logstash is still not able to connect with kafka, cause:

11:41:02.175 [[main] kafka -> splunk stack working pretty well, I 
could probably answer a few of your questions -


You can list topics (and a lot of other stuff) on the kafka brokers 
themselves using the kafka-topics.sh script included with kafka.  e.g.:


bin/kafka-topics.sh —zookeeper=localhost:2181 —list
only __consumer_offsets is shown, so probably I'm not adding topics 
correctly



bin/kafka-topics.sh —zookeeper=locahost:2181 —topic “topic” —describe
Topic: __consumer_offsetsPartition: 0Leader: 9 Replicas: 
9,10,15Isr: 10,9,15
Topic: __consumer_offsetsPartition: 1Leader: 10 Replicas: 
10,15,9Isr: 10,9,15
Topic: __consumer_offsetsPartition: 2Leader: 15 Replicas: 
15,9,10Isr: 10,9,15

...
Topic: __consumer_offsetsPartition: 47Leader: 15 Replicas: 
15,10,9Isr: 10,9,15
Topic: __consumer_offsetsPartition: 48Leader: 9 Replicas: 
9,10,15Isr: 10,9,15
Topic: __consumer_offsetsPartition: 49Leader: 10 Replicas: 
10,15,9Isr: 10,9,15


does this mean the cluster is properly formed?

I’d recommend using kafka-manager to manage your cluster.  It’ll give 
you a much quicker look in to your topics, your brokers, consumers, 
and throughput.  It also makes creating and deleting topics easy.

It isn't able to show cluster list, so perhaps problems connecting to zk?

If you’re not seeing your topics get created the first place I’d look 
is in the kafka broker logs themselves - server.log and 
kafkaServer.out - then work your way back from there.  As you’ve 
found, omkafka isn’t terribly verbose when it comes to error reporting.

plenty of:
[2017-01-30 16:35:05,177] INFO [Group Metadata Manager on Broker 15]: 
Removed 0 expired offsets in 0 milliseconds. 
(kafka.coordinator.GroupMetadataManager)
[2017-01-30 16:45:05,177] INFO [Group Metadata Manager on Broker 15]: 
Removed 0 expired offsets in 0 milliseconds. 
(kafka.coordinator.GroupMetadataManager)
[2017-01-30 16:55:05,177] INFO [Group Metadata Manager on Broker 15]: 
Removed 0 expired offsets in 0 milliseconds. 
(kafka.coordinator.GroupMetadataManager)




For your timeout issues, the first place I’d look to is the local 
firewall configuration.  Also in your "broker=["cluster_kafka”]” 
portion are you including the port number for the broker (I’m 
assuming 9092)?

I thought port was added by default. I'll try again.

Thanks!


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] working example of omkafka?

[mostolog--- via rsyslog]

El 30/01/17 a las 19:25, Andrew Griffin escribió:
I have a rsyslog -> kafka -> splunk stack working pretty well, I could 
probably answer a few of your questions -


You can list topics (and a lot of other stuff) on the kafka brokers 
themselves using the kafka-topics.sh script included with kafka.  e.g.:


bin/kafka-topics.sh —zookeeper=localhost:2181 —list
only __consumer_offsets is shown, so probably I'm not adding topics 
correctly



bin/kafka-topics.sh —zookeeper=locahost:2181 —topic “topic” —describe
Topic: __consumer_offsetsPartition: 0Leader: 9Replicas: 
9,10,15Isr: 10,9,15
Topic: __consumer_offsetsPartition: 1Leader: 10Replicas: 
10,15,9Isr: 10,9,15
Topic: __consumer_offsetsPartition: 2Leader: 15Replicas: 
15,9,10Isr: 10,9,15

...
Topic: __consumer_offsetsPartition: 47Leader: 15 Replicas: 
15,10,9Isr: 10,9,15
Topic: __consumer_offsetsPartition: 48Leader: 9Replicas: 
9,10,15Isr: 10,9,15
Topic: __consumer_offsetsPartition: 49Leader: 10 Replicas: 
10,15,9Isr: 10,9,15


does this mean the cluster is properly formed?

I’d recommend using kafka-manager to manage your cluster.  It’ll give 
you a much quicker look in to your topics, your brokers, consumers, 
and throughput.  It also makes creating and deleting topics easy.

It isn't able to show cluster list, so perhaps problems connecting to zk?

If you’re not seeing your topics get created the first place I’d look 
is in the kafka broker logs themselves - server.log and 
kafkaServer.out - then work your way back from there.  As you’ve 
found, omkafka isn’t terribly verbose when it comes to error reporting.

plenty of:
[2017-01-30 16:35:05,177] INFO [Group Metadata Manager on Broker 15]: 
Removed 0 expired offsets in 0 milliseconds. 
(kafka.coordinator.GroupMetadataManager)
[2017-01-30 16:45:05,177] INFO [Group Metadata Manager on Broker 15]: 
Removed 0 expired offsets in 0 milliseconds. 
(kafka.coordinator.GroupMetadataManager)
[2017-01-30 16:55:05,177] INFO [Group Metadata Manager on Broker 15]: 
Removed 0 expired offsets in 0 milliseconds. 
(kafka.coordinator.GroupMetadataManager)




For your timeout issues, the first place I’d look to is the local 
firewall configuration.  Also in your "broker=["cluster_kafka”]” 
portion are you including the port number for the broker (I’m assuming 
9092)?

I thought port was added by default. I'll try again.

Thanks!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] working example of omkafka?

[mostolog--- via rsyslog]

Hi

*Do any of you have a working example of rsyslog (omkafka) -> kafka <- 
losgstash ?*


So far I have been able to deploy a zookeper cluster using docker, 
deploy a kafka cluster (to be honest, I don't know how to test if it's 
working as a cluster or if I failed miserably), rsyslog as producer and 
logstash as consumer.


But rsyslog doesn't show anything about kafka in logs, I'm not able to 
list topics within kafka (although they are supposed to be automatically 
created) and logstash, on the other side, doesn't show any events on debug.


Stats dump:

   Mon Jan 30 15:04:49 2017: global: origin=dynstats
   Mon Jan 30 15:04:49 2017: omkafka: submitted=7016 maxoutqsize=3010
   failures=0 topicdynacache.skipped=7014 topicdynacache.miss=2
   topicdynacache.evicted=0
   Mon Jan 30 15:04:49 2017: kafka: origin=core.action processed=7016
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Mon Jan 30 15:04:49 2017: error: origin=core.action processed=0
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Mon Jan 30 15:04:49 2017: unk: origin=core.action processed=0
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Mon Jan 30 15:04:49 2017: imrelp[20514]: origin=imrelp submitted=7016
   Mon Jan 30 15:04:49 2017: resource-usage: origin=impstats
   utime=2116000 stime=158 maxrss=12596 minflt=2582 majflt=0
   inblock=10 oublock=5201 nvcsw=38958 nivcsw=1102
   Mon Jan 30 15:04:49 2017: relp[DA]: origin=core.queue size=0
   enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
   Mon Jan 30 15:04:49 2017: relp: origin=core.queue size=0
   enqueued=7016 full=0 discarded.full=0 discarded.nf=0 maxqsize=504
   Mon Jan 30 15:04:49 2017: main Q: origin=core.queue size=13
   enqueued=111 full=0 discarded.full=0 discarded.nf=0 maxqsize=14
   Mon Jan 30 15:06:30 2017: global: origin=dynstats
   Mon Jan 30 15:06:30 2017: omkafka: submitted=7884 maxoutqsize=3010
   failures=0 topicdynacache.skipped=7882 topicdynacache.miss=2
   topicdynacache.evicted=0
   Mon Jan 30 15:06:30 2017: kafka: origin=core.action processed=7884
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Mon Jan 30 15:06:30 2017: error: origin=core.action processed=0
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Mon Jan 30 15:06:30 2017: unk: origin=core.action processed=0
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Mon Jan 30 15:06:30 2017: imrelp[20514]: origin=imrelp submitted=7884
   Mon Jan 30 15:06:30 2017: resource-usage: origin=impstats
   utime=2356000 stime=1824000 maxrss=12860 minflt=2649 majflt=0
   inblock=10 oublock=6208 nvcsw=43871 nivcsw=1231
   Mon Jan 30 15:06:30 2017: relp[DA]: origin=core.queue size=0
   enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
   Mon Jan 30 15:06:30 2017: relp: origin=core.queue size=0
   enqueued=7884 full=0 discarded.full=0 discarded.nf=0 maxqsize=504
   Mon Jan 30 15:06:30 2017: main Q: origin=core.queue size=13
   enqueued=125 full=0 discarded.full=0 discarded.nf=0 maxqsize=14

rsyslog.conf
template(name="topic" type="string" string="foo")
...
action(
name="kafka"
type="omkafka"
broker=["cluster_kafka"]
dynatopic="on"
topic="topic"
dynatopic.cachesize="300"
template="json"
errorFile="/data/kafka-error.json"
partitions.auto="on"
)

kafka-error.json

   { "errcode": -192, "errmsg": "Local: Message timed out", "data": "{
   \"app\": \"myapp\",\"originalmsg\":
   \"<167>2017-01-30T15:31:22.927536+01:00 host app[pid]: conn=4311996
   fd=20 closed\" }\n" }

kafka.properties

   broker.id=9
   zookeeper.connect=my_zookeeper_cluster:2181
   listeners=PLAINTEXT://10.0.0.9:9092

Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Out of memory

[mostolog--- via rsyslog]

davidelang: did you find something interesting in the 50K@100m/s vs. 
50K@2000m/s ?


Perhaps we should try after leak being fixed?


El 19/01/17 a las 13:55, mosto...@gmail.com escribió:
Attached 2 files. 1 for 100m/s and the other for 2000m/s. Let me know 
if you need longer traces.


I'll try valgrind memcheck later.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Out of memory

[mostolog--- via rsyslog]

Considering the below explanation, an idle (not going to receive
more messages the next 7 days) imrelp rsyslog...shouldn't memory
be freed from current 512MB usage in the near future?


For the scenario I have described: no, as long as the high address 
memory block is allocated, nothing is freed. Not even after 70 days of 
idleness ;-)



You mentioned forcing each 100.000 messages, but that's never
gonna happen if we are still at 20k, and no more messages coming.


right

Isn't that an undesired behavior?
A process should release unused/unneeded memory, shouldn't it? (Maybe 
not "immediately", but sooner than "ever"!)


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Rsyslog imkafka

[mostolog--- via rsyslog]

Hi.

Just started deploying kafka for testing with omkafka. Have anyone 
considered doing an *imkafka* module?


So far, we have:

 * RELP->rsyslog->omelasticsearch->elasticsearch
 * RELP->rsyslog->omkafka

Otherwise, I'll go for logstash.

Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Out of memory

[mostolog--- via rsyslog]

Attached stats and valgrind logs. Let me know if I can do anything else.

rsyslog was built with a few modules, debug and rtinst.

Run:

   valgrind --tool=memcheck --log-file=/data/valgrind.log
   --show-possibly-lost=yes --track-origins=yes --trace-children=yes
   --leak-check=full --show-leak-kinds=all rsyslogd -n

Receiving 100m/s for a few minutes

With the following config:

   module(load="builtin:omfile")
   module(load="imrelp")
   module(load="impstats" log.file="/data/stats.log" interval="65")
   global(
   MaxMessageSize="32k"
   workDirectory="/data"
   parser.escapeControlCharactersOnReceive="off"
   )
   input(
port="20514"
type="imrelp"
name="imrelp"
ruleset="relp"
   )
   template(name="json" type="string" string="%$!%\n")
   template(name="index" type="string" string="%$.index%@%$.interval%")
   template(name="type" type="string" string="%$.type%")
   template(name="ts" type="string" string="%timestamp:::date-rfc3339%")
   ruleset(
name="relp"
   queue.filename="relp"
   queue.maxdiskspace="1G"
   queue.SaveOnShutdown="on"
   #queue.lowwatermark
   #queue.highwatermark
   #queue.size
   queue.type="LinkedList"
) {
set $.index="unknown";
set $.type="unknown";
   set $.interval=$$now & ":" & $$hour;
set $!host_forwarded=$hostname;
set $!host_received=$$myhostname;
set $!time_received=$timegenerated;
set $@timestamp=exec_template("ts");
#action(
# name="elastic"
# action.resumeRetryCount="-1"
# action.reportsuspension="on"
# type="omelasticsearch"
# server="server"
# serverport="9200"
# searchIndex="index"
# dynSearchIndex="on"
# searchType="type"
# dynSearchType="on"
# template="json"
#)
action(
   name="tofile"
   type="omfile"
   file="/data/out.log"
   template="json"
)
   }


Regards
==104== Memcheck, a memory error detector
==104== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==104== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==104== Command: rsyslogd -n
==104== Parent PID: 1
==104== 
==104== 
==104== HEAP SUMMARY:
==104== in use at exit: 53,282,914 bytes in 196,139 blocks
==104==   total heap usage: 10,618,745 allocs, 10,422,606 frees, 1,004,774,708 bytes allocated
==104== 
==104== 4 bytes in 1 blocks are still reachable in loss record 1 of 64
==104==at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==104==by 0x5D1A8D9: strdup (strdup.c:42)
==104==by 0x42BD32: InfoConstruct (obj.c:151)
==104==by 0x43917B: varClassInit (var.c:122)
==104==by 0x42D8A6: objClassInit (obj.c:1409)
==104==by 0x40E5AC: rsrtInit (rsyslog.c:159)
==104==by 0x45AA95: rsyslogd_InitGlobalClasses (rsyslogd.c:473)
==104==by 0x45AA95: initAll (rsyslogd.c:1122)
==104==by 0x40DF87: main (rsyslogd.c:1674)
==104== 
==104== 4 bytes in 1 blocks are still reachable in loss record 2 of 64
==104==at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==104==by 0x5D1A8D9: strdup (strdup.c:42)
==104==by 0x42BD32: InfoConstruct (obj.c:151)
==104==by 0x4223DB: msgClassInit (msg.c:5005)
==104==by 0x40E637: rsrtInit (rsyslog.c:177)
==104==by 0x45AA95: rsyslogd_InitGlobalClasses (rsyslogd.c:473)
==104==by 0x45AA95: initAll (rsyslogd.c:1122)
==104==by 0x40DF87: main (rsyslogd.c:1674)
==104== 
==104== 5 bytes in 1 blocks are still reachable in loss record 3 of 64
==104==at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==104==by 0x5D1A8D9: strdup (strdup.c:42)
==104==by 0x42BD32: InfoConstruct (obj.c:151)
==104==by 0x438ADB: strmClassInit (stream.c:2291)
==104==by 0x42D8BE: objClassInit (obj.c:1411)
==104==by 0x40E5AC: rsrtInit (rsyslog.c:159)
==104==by 0x45AA95: rsyslogd_InitGlobalClasses (rsyslogd.c:473)
==104==by 0x45AA95: initAll (rsyslogd.c:1122)
==104==by 0x40DF87: main (rsyslogd.c:1674)
==104== 
==104== 6 bytes in 1 blocks are still reachable in loss record 4 of 64
==104==at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==104==by 0x5D1A8D9: strdup (strdup.c:42)
==104==by 0x430217: addManagedCounter (statsobj.c:215)
==104==by 0x43037D: addCounter (statsobj.c:257)
==104==by 0x7F56705: ???
==104==by 0x414170: tellModulesActivateConfig (rsconf.c:720)
==104==by 0x414170: activate (rsconf.c:865)
==104==by 0x45B542: initAll (rsyslogd.c:1334)
==104==by 0x40DF87: main (rsyslogd.c:1674)
==104== 
==104== 6 bytes in 1 blocks are still reachable in loss record 5 of 64
==104==at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==104==by 0x5D1A8D9: strdup (strdup.c:42)
==104==by 0x430217: addManagedCounter (statsobj.c:215)
==104==by 0x43037D: addCounter (statsobj.c:257)
==104==by 0x7F5672E: ???

Re: [rsyslog] Out of memory

[mostolog--- via rsyslog]

Really interesting and instructive. Let me see if I understood properly:

Considering the below explanation, an idle (not going to receive more 
messages the next 7 days) imrelp rsyslog...shouldn't memory be freed 
from current 512MB usage in the near future?


You mentioned forcing each 100.000 messages, but that's never gonna 
happen if we are still at 20k, and no more messages coming.


Regards


El 19/01/17 a las 11:16, Rainer Gerhards escribió:
2017-01-19 10:19 GMT+01:00 mostolog--- via rsyslog 
<rsyslog@lists.adiscon.com <mailto:rsyslog@lists.adiscon.com>>:


with very few exceptions, rsyslog releases the memory as it
goes, there should not be any significant amount of memory
freed by rsyslog after it's been idle for a while.

But being idle for 15m should release memory from 512MB to a few
KB if they aren't used, isnt it?


Memory alloc is not as simple as it seems ;-)

First, clib does not always do a proper cleanup. I guess it seems not 
to consolidate free space in all cases. There is a clib call to force 
this, and rsyslog does it from time to time (I think every 100,000 
messages). We do not do it very frequently, because it is an expensive 
operation. Also note that memory is reusable internally, so even 
though it is not returned to the OS, further alloc requests inside 
rsyslog can use this memory and do so. Returning memory to the OS and 
re-claiming it is expensive. Thus you do want to keep some memory 
allocated but internally unused to avoid doing this operation too 
frequently.


Secondly, memory alloc from the OS is done by sbrk[1] IIRC. The 
important point is that we need to alloc and free memory in sequence. 
This means if you alloc 100MB, than alloc 1MB, you have the following 
memory layout


BASE
100MB
1MB

with the break at BASE+101MB. If you now free the 100MB chunk, you 
have this layout:


BASE
100MB free
1MB

if you want to return to the OS, you'd need to copy down the 1MB to 
immediatly after BASE, because otherwise you cannot reset the break to 
BASE+1MB. The allocator does not do this, it would totally wrek 
performance (note: that is not rsyslog specific, that is how the *C 
runtime* works). More importantly, it would mean all pointers to it 
would need to be updated. And the runtime does not know where these 
pointers are located. So it would not only costly, compaction is 
simply impossible. Let's assume we now alloc another 10 MB. Then we have


BASE
10MB
90MB free
1MB

because the allocator uses the free, but still allocated mem. Now, 
let's assume we free the 1MB chunk, we get:


BASE
10MB
91MB free

Now the free space is at the end of the data segment. So the alloc 
subsystem has the choice to reduce memory alloc from the OS. It may or 
may not dealloc. I don't know the exact rules, but the important thing 
is that the alloc system uses some heuristic (plus the call I 
mentioned) to decide if to dealloc. Let's assume it does. Then it 
reduceds the data segement size and we get to


BASE
10MB

effectively reducing rss by 91 MB.

IMHO the alloc system strongly works on the assumption that memory 
allocated (from the OS) but free internally does not really hurt, as 
it is just virtual address space, which, if actually unused, is paged 
out to disk once and then doesn't matter at all until it get's reused 
again.


OF COURSE if we have constantly growing memory, the app seems not to 
free some chunks, so the alloc system doesn't know they are free. This 
has nothing to do with what I explained. What I explained just means 
that an app may do proper free()'s, but the rss size doesn't reflect 
this. The typical (and often visible) effect of that is that the app 
grows to a certain size and then remains at it (no growth, no 
decrease). This is where, via the clib call, we force free.


There is one important point, though: if we have a memory block at the 
far end of the data segment, we cannot return mem to the OS until this 
block has been freed. In rsyslog, you typical see this if


a) old-style "last message repeated n times" is active
b) an infrequently written-to output stores a message object when 
queues are very full

c) no different messages are written to that output for a long time

What here is very probably that in b) a high memory address is used 
for that memory block. As due to c) we do not have any traffic, that 
high address is kept in use for hours, maybe days. Once a different 
message arrives at the output, the old object is processed and freed. 
This can result in a very sharp decrease of rss size, especially if 
the system has low load during such times. Let's assume this layout:


BASE
10MB used
1.5GB free, but allocated from OS
5k msg object

If we now free those 5k, most probably the alloc system heuristic will 
immediately return 1.5GB+5K to the OS, reducing the rss size accordingly.


Bottom line: not everything that looks like a memory leak actually is one.

HTH
[1] https://li

Re: [rsyslog] How to know which modules rsyslog was compiled with

[mostolog--- via rsyslog]

good point ;)


El 19/01/17 a las 16:16, Rainer Gerhards escribió:

2017-01-19 16:12 GMT+01:00 mostolog--- via rsyslog <rsyslog@lists.adiscon.com>:

Is there a simple way to know which modules rsyslog was compiled with?

this is a serious answer: check, which module binaries were created.
If you just have the rsyslog binary, there is no way to see which
other modules were compiled in the same build. That doesn't really
matter either, because you obviously do not have them, so why record
it? The modules themself log upon startup to the debug log which
version they are from.

Rainer


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] How to know which modules rsyslog was compiled with

[mostolog--- via rsyslog]

Is there a simple way to know which modules rsyslog was compiled with?

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Out of memory

[mostolog--- via rsyslog]

Attached 2 files. 1 for 100m/s and the other for 2000m/s. Let me know if 
you need longer traces.


I'll try valgrind memcheck later.
Thu Jan 19 13:46:38 2017: global: origin=dynstats 
Thu Jan 19 13:46:38 2017: tofile: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 
Thu Jan 19 13:46:38 2017: imrelp[20514]: origin=imrelp submitted=0 
Thu Jan 19 13:46:38 2017: resource-usage: origin=impstats utime=4 stime=12000 maxrss=7184 minflt=1024 majflt=0 inblock=7 oublock=16 nvcsw=395 nivcsw=29 
Thu Jan 19 13:46:38 2017: relp[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 
Thu Jan 19 13:46:38 2017: relp: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 
Thu Jan 19 13:46:38 2017: main Q: origin=core.queue size=0 enqueued=6 full=0 discarded.full=0 discarded.nf=0 maxqsize=1 
Thu Jan 19 13:47:43 2017: global: origin=dynstats 
Thu Jan 19 13:47:43 2017: tofile: origin=core.action processed=44374 failed=0 suspended=0 suspended.duration=0 resumed=0 
Thu Jan 19 13:47:43 2017: imrelp[20514]: origin=imrelp submitted=44374 
Thu Jan 19 13:47:43 2017: resource-usage: origin=impstats utime=2348000 stime=246 maxrss=16852 minflt=4194 majflt=0 inblock=7 oublock=8857 nvcsw=59572 nivcsw=297 
Thu Jan 19 13:47:43 2017: relp[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 
Thu Jan 19 13:47:43 2017: relp: origin=core.queue size=0 enqueued=44375 full=0 discarded.full=0 discarded.nf=0 maxqsize=42 
Thu Jan 19 13:47:43 2017: main Q: origin=core.queue size=0 enqueued=13 full=0 discarded.full=0 discarded.nf=0 maxqsize=1 
Thu Jan 19 13:48:48 2017: global: origin=dynstats 
Thu Jan 19 13:48:48 2017: tofile: origin=core.action processed=125020 failed=0 suspended=0 suspended.duration=0 resumed=0 
Thu Jan 19 13:48:48 2017: imrelp[20514]: origin=imrelp submitted=125020 
Thu Jan 19 13:48:48 2017: resource-usage: origin=impstats utime=6816000 stime=7272000 maxrss=39456 minflt=9867 majflt=0 inblock=7 oublock=24925 nvcsw=174306 nivcsw=4849 
Thu Jan 19 13:48:48 2017: relp[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 
Thu Jan 19 13:48:48 2017: relp: origin=core.queue size=0 enqueued=125020 full=0 discarded.full=0 discarded.nf=0 maxqsize=42 
Thu Jan 19 13:48:48 2017: main Q: origin=core.queue size=0 enqueued=20 full=0 discarded.full=0 discarded.nf=0 maxqsize=1 
Thu Jan 19 13:49:53 2017: global: origin=dynstats 
Thu Jan 19 13:49:53 2017: tofile: origin=core.action processed=134375 failed=0 suspended=0 suspended.duration=0 resumed=0 
Thu Jan 19 13:49:53 2017: imrelp[20514]: origin=imrelp submitted=134375 
Thu Jan 19 13:49:53 2017: resource-usage: origin=impstats utime=7264000 stime=776 maxrss=42096 minflt=10525 majflt=0 inblock=7 oublock=26790 nvcsw=187065 nivcsw=4899 
Thu Jan 19 13:49:53 2017: relp[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 
Thu Jan 19 13:49:53 2017: relp: origin=core.queue size=0 enqueued=134375 full=0 discarded.full=0 discarded.nf=0 maxqsize=42 
Thu Jan 19 13:49:53 2017: main Q: origin=core.queue size=0 enqueued=27 full=0 discarded.full=0 discarded.nf=0 maxqsize=2Thu Jan 19 13:37:39 2017: global: origin=dynstats 
Thu Jan 19 13:37:39 2017: tofile: origin=core.action processed=3414 failed=0 suspended=0 suspended.duration=0 resumed=0 
Thu Jan 19 13:37:39 2017: imrelp[20514]: origin=imrelp submitted=3414 
Thu Jan 19 13:37:39 2017: resource-usage: origin=impstats utime=124000 stime=52000 maxrss=7324 minflt=1306 majflt=0 inblock=7 oublock=696 nvcsw=1167 nivcsw=20 
Thu Jan 19 13:37:39 2017: relp[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 
Thu Jan 19 13:37:39 2017: relp: origin=core.queue size=0 enqueued=3414 full=0 discarded.full=0 discarded.nf=0 maxqsize=59 
Thu Jan 19 13:37:39 2017: main Q: origin=core.queue size=1 enqueued=6 full=0 discarded.full=0 discarded.nf=0 maxqsize=2 
Thu Jan 19 13:38:44 2017: global: origin=dynstats 
Thu Jan 19 13:38:44 2017: tofile: origin=core.action processed=23043 failed=0 suspended=0 suspended.duration=0 resumed=0 
Thu Jan 19 13:38:44 2017: imrelp[20514]: origin=imrelp submitted=23043 
Thu Jan 19 13:38:44 2017: resource-usage: origin=impstats utime=1344000 stime=1536000 maxrss=10724 minflt=2690 majflt=0 inblock=7 oublock=4607 nvcsw=36891 nivcsw=4915 
Thu Jan 19 13:38:44 2017: relp[DA]: origin=core.queue size=0 enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0 
Thu Jan 19 13:38:44 2017: relp: origin=core.queue size=1 enqueued=23044 full=0 discarded.full=0 discarded.nf=0 maxqsize=59 
Thu Jan 19 13:38:44 2017: main Q: origin=core.queue size=0 enqueued=13 full=0 discarded.full=0 discarded.nf=0 maxqsize=2 
Thu Jan 19 13:39:49 2017: global: origin=dynstats 
Thu Jan 19 13:39:49 2017: tofile: origin=core.action processed=46672 failed=0 suspended=0 suspended.duration=0 resumed=0 
Thu Jan 19 13:39:49 2017: 

Re: [rsyslog] Out of memory

[mostolog--- via rsyslog]

El 18/01/17 a las 17:05, Rainer Gerhards escribió:
If this is a test environment, I would strongly recommend to run it in 
a terminal session under valgrind control. If it is an actual leak, 
valgrind can provide very good diagnostic information.


That would be very helpful and clarify what is going on.

Rainer

On my way!

Any valgrind special setting?

Will testing 1 message each 5 seconds be enough?

What impstats interval should I use?

Does 1056K says anything to you?


El 18/01/17 a las 17:06, David Lang escribió:

grep -e imrelp -e resource-usage |cut -f 4,6- -d " "

Will do.

I'm less interested in the with and without ES than I am in 100m/sec 
vs 2000m/sec for the same total number of messages. 1m messages/test 
would be great, but that takes a long time, let's start with something 
small like 50K messages per test.


it doesn't need to be exact, but I want to see if ~50K messages @ 
100m/sec and ~50K messages @ 2000m/sec result in similar maxrss sizes 
in the pstats output.

Ok.

with very few exceptions, rsyslog releases the memory as it goes, 
there should not be any significant amount of memory freed by rsyslog 
after it's been idle for a while.
But being idle for 15m should release memory from 512MB to a few KB if 
they aren't used, isnt it?


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Out of memory

[mostolog--- via rsyslog]

Below an impstats trace showing a few tests:

 * receiving some idle messages
 * 1 message per second
 * 10m/sec
 * 100m/s
 * 1000m/s
 * 2000m/s

None of the above created any queue files when using omfile as output.

*Still, memory was slowly increasing (RGROW=1056K with 100m/s input) 
according to atop. Shall we have a look on this?

*

*omfile/without omelasticsearch*

   Wed Jan 18 13:28:46 2017: global: origin=dynstats
   Wed Jan 18 13:28:46 2017: action 0: origin=core.action processed=0
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:28:46 2017: omelasticsearch: origin=omelasticsearch
   submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
   failed.es=0
   Wed Jan 18 13:28:46 2017: json: origin=core.action processed=83
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:28:46 2017: norm: origin=core.action processed=83
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:28:46 2017: tofile: origin=core.action processed=83
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:28:46 2017: error: origin=core.action processed=0
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:28:46 2017: unk: origin=core.action processed=0
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:28:46 2017: imrelp[20514]: origin=imrelp submitted=83
   Wed Jan 18 13:28:46 2017: resource-usage: origin=impstats
   utime=4 stime=28000 maxrss=7176 minflt=1278 majflt=0 inblock=10
   oublock=87 nvcsw=395 nivcsw=48
   Wed Jan 18 13:28:46 2017: relp[DA]: origin=core.queue size=0
   enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
   Wed Jan 18 13:28:46 2017: relp: origin=core.queue size=0 enqueued=83
   full=0 discarded.full=0 discarded.nf=0 maxqsize=53
   Wed Jan 18 13:28:46 2017: main Q: origin=core.queue size=0
   enqueued=12 full=0 discarded.full=0 discarded.nf=0 maxqsize=1
   Wed Jan 18 13:30:51 2017: global: origin=dynstats
   Wed Jan 18 13:30:51 2017: action 0: origin=core.action processed=0
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:30:51 2017: omelasticsearch: origin=omelasticsearch
   submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
   failed.es=0
   Wed Jan 18 13:30:51 2017: json: origin=core.action processed=89
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:30:51 2017: norm: origin=core.action processed=89
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:30:51 2017: tofile: origin=core.action processed=89
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:30:51 2017: error: origin=core.action processed=0
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:30:51 2017: unk: origin=core.action processed=0
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:30:51 2017: imrelp[20514]: origin=imrelp submitted=89
   Wed Jan 18 13:30:51 2017: resource-usage: origin=impstats
   utime=48000 stime=36000 maxrss=7212 minflt=1318 majflt=0 inblock=10
   oublock=92 nvcsw=439 nivcsw=55
   Wed Jan 18 13:30:51 2017: relp[DA]: origin=core.queue size=0
   enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
   Wed Jan 18 13:30:51 2017: relp: origin=core.queue size=0 enqueued=89
   full=0 discarded.full=0 discarded.nf=0 maxqsize=72
   Wed Jan 18 13:30:51 2017: main Q: origin=core.queue size=0
   enqueued=12 full=0 discarded.full=0 discarded.nf=0 maxqsize=4
   Wed Jan 18 13:32:31 2017: global: origin=dynstats
   Wed Jan 18 13:32:31 2017: action 0: origin=core.action processed=0
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:32:31 2017: omelasticsearch: origin=omelasticsearch
   submitted=0 failed.http=0 failed.httprequests=0 failed.checkConn=0
   failed.es=0
   Wed Jan 18 13:32:31 2017: json: origin=core.action processed=486
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:32:31 2017: norm: origin=core.action processed=486
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:32:31 2017: tofile: origin=core.action processed=486
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:32:31 2017: error: origin=core.action processed=0
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:32:31 2017: unk: origin=core.action processed=0
   failed=0 suspended=0 suspended.duration=0 resumed=0
   Wed Jan 18 13:32:31 2017: imrelp[20514]: origin=imrelp submitted=486
   Wed Jan 18 13:32:31 2017: resource-usage: origin=impstats
   utime=8 stime=88000 maxrss=7212 minflt=1323 majflt=0 inblock=10
   oublock=468 nvcsw=1019 nivcsw=63
   Wed Jan 18 13:32:31 2017: relp[DA]: origin=core.queue size=0
   enqueued=0 full=0 discarded.full=0 discarded.nf=0 maxqsize=0
   Wed Jan 18 13:32:31 2017: relp: origin=core.queue size=0
   enqueued=486 full=0 discarded.full=0 discarded.nf=0 maxqsize=72
   Wed Jan 18 13:32:31 2017: main Q: origin=core.queue size=1

Re: [rsyslog] Out of memory

[mostolog--- via rsyslog]

when you give us your config, please include the name= portion so we 
can see what actions match up with what in impstats

Pasted at the end, but they are named according to what they do.


at 10:37, the queue is small and maxrss is still low (8092)

at 10:42, you got a huge burst of new traffic (104878-1889=102989 
messages in this 5 min period), but the number processed is low. see 
the submitted counts that only processed 70614-1889=68725 messages, so 
35001 in the queue with meory usage climging to 75992

100k/5min is our expected load. bursts will be much higher.



at 10:47 you had received an additional 196015-104878=91137 messages, 
but processed 162117-70614=91503 messages, gaining slightly with the 
queue dropping to 33978 (with memory usage climbing to 104684)


the fact that memory use climbed so much with the queue size dropping 
is odd, but that could have been a burst from something processing 
messages


at 10:52 you had received an additonal 299439-196015=103424 
messages.little ground with the queue size at 35002 (with memory usage climbing 
to 135732)


at 10:57 the pattern continues, with lots of messages being received 
and processed (no significant change to the queue size), but memory 
use continuing to climb.

and that's what I don't get.

at 11:32 the number of new messages drops significantly and that lets 
rsyslog deliver everything in the queue to catch up.


what happens after that point?


atop VSIZE=842 / RSIZE=512 remain stuck for rsyslogd, no matter if 
there's not traffic anymore.


it's worth noting that the queue sizes are instananious sizes, not 
average sizes, so if you have a lot of messages that are delivered 
just before the stats run happens, it will show a lot of messages in 
the queue. your stats times only vary by a few seconds over the time 
of the run, so it's impossible to say if the ~35K queue is due to a 
burst of what. You could try changing the stats time to something much 
smaller to see what happens, or change it to something that's not an 
even number of minutes so that if something is generating logs every X 
minutes, you won't stay in sync with them (try something like 100 
seconds for example)

Setting impstats interval to 100 for the next round ;)

But overall, it does look like there is a memory leak somewhere, can 
you run a copy of rsyslog somewhere that will allow you to tinker with 
the config significantly? change the output to go to a file instead of 
ES (using the same template that you are using in ES would be good), 
and then see what happens. If the memory leak stops, it's an 
omelasticsearch issue, if not, we can try tinkering with the other 
actions and see what makes a difference.

Going to do that, as this is actually the testing env.

Thanks as usual.


module(load="impstats" log.file="/data/stats.log")
syslog.=debug /data/rsyslog-stats
global(
MaxMessageSize="32k"
workDirectory="/data"
parser.escapeControlCharactersOnReceive="off"
)
module(load="imrelp")
input(
port="20514"
type="imrelp"
name="imrelp"
ruleset="relp"
)

template(name="json" type="string" string="%$!%\n")
template(name="index" type="string" string="%$.index%@%$.interval%")
template(name="type" type="string" string="%$.type%")
template(name="ts" type="string" string="%timestamp:::date-rfc3339%")

module(load="mmjsonparse")
module(load="mmnormalize")
module(load="omelasticsearch")
ruleset(
name="relp"
queue.filename="relp"
queue.maxdiskspace="1G"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
) {
action(
name="json"
cookie=""
type="mmjsonparse"
)
if $parsesuccess == "FAIL" then {
call error
stop
}
action(
name="norm"
type="mmnormalize"
variable="$!msg"
rulebase="/etc/rsyslog.d/rsyslog.rb"
)
$IncludeConfig /etc/rsyslog.d/apps/conf/1*.conf
$IncludeConfig /etc/rsyslog.d/apps/conf/2*.conf
#there are no 1* neither 2* files

# Set default index and type
set $.index="unknown";
set $.type="unknown";
#defaults
set $.interval=$$now & ":" & $$hour;
if $!app != $!app then {
call unknown
stop
}
$IncludeConfig /etc/rsyslog.d/apps/conf/3*.conf
#a few files like
#else if $!app == "myapp" then {
#set $.index="account-app@" & $$now;
#set $.type="logs";
#call geoip
#}
call clean

set $!host_forwarded=$hostname;
set $!host_received=$$myhostname;
set $!time_received=$timegenerated;
set $@timestamp=exec_template("ts");
action(
name="elastic"
action.resumeRetryCount="-1"
action.reportsuspension="on"
type="omelasticsearch"
server="server"
serverport="9200"
searchIndex="index"
dynSearchIndex="on"
searchType="type"
dynSearchType="on"
template="json"
)
}
$IncludeConfig /etc/rsyslog.d/apps/conf/4*.conf
#a few files like:
#ruleset(name="geoip"){

Re: [rsyslog] Out of memory

[mostolog--- via rsyslog]

   MaxMessageSize="32k"

just a note that that's a rather large message size.
Most messages are ~1KB, but some Java stack traces take up to 31KB. 
Setting this for 1% messages have any impact on the other 99%?


you are obviously not showing us the full config, because nothing 
loads the relp module.

just omitted module load




   action(
   type="mmnormalize"
   #actually it's doing nothing, cause liblognorm issues.
   )

what issues?

https://github.com/rsyslog/liblognorm/issues/235
https://github.com/rsyslog/liblognorm/issues/236




   $IncludeConfig /etc/rsyslog.d/apps/conf/3*.conf

we would need to see what these have in them.

ATM, just have 3 of them, with:

   else if $!app == "app1" then {
#set $.index="group$!app@" & $$now;
set $.type="logs";
call geoip
   }


  $IncludeConfig /etc/rsyslog.d/apps/conf/4*.conf

we would need to see what these have in them

ATM, just a few like:

   ruleset(name="geoip"){
if $!ip != "" then {
set $!geo="true";
unset $!ip;
}
   }

The pstats provided seem to be from the beginning of the run, we need 
to see them for around the time it runs out of memory.


In the timeframe you are providing, maxrss remains constant, so memory 
use is not increasing.

An updated version of stats.
Please, notice that although we are not forwarding traffic at this 
moment, hence memory footprint should be low, atop still show a high usage.

Is rsyslog not freeing unused memory?

PID   MINFLT  MAJFLT  VSTEXT  VSIZERSIZEVGROW  RGROW RUID  
EUID  MEM  CMD
94755  0   0   0K  842.3M  512.7M 0K   
0K  root   root   2% rsyslogd


As you can see below, we started forwarding at 10:40 and sottped around 
12:06



Wed Jan 18 09:47:57 2017: global: origin=dynstats
...
Wed Jan 18 10:32:57 2017: global: origin=dynstats
Wed Jan 18 10:32:57 2017: action 0: origin=core.action processed=0 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:32:57 2017: omelasticsearch: origin=omelasticsearch 
submitted=1729 failed.http=0 failed.httprequests=0 failed.checkConn=0 
failed.es=0
Wed Jan 18 10:32:57 2017: json: origin=core.action processed=1729 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:32:57 2017: norm: origin=core.action processed=1729 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:32:57 2017: elastic: origin=core.action processed=1729 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:32:57 2017: error: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:32:57 2017: unk: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0

Wed Jan 18 10:32:57 2017: imrelp[20514]: origin=imrelp submitted=1729
Wed Jan 18 10:32:57 2017: resource-usage: origin=impstats utime=944000 
stime=812000 maxrss=8092 minflt=1500 majflt=0 inblock=10 oublock=42 
nvcsw=10437 nivcsw=122
Wed Jan 18 10:32:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0 
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:32:57 2017: relp: origin=core.queue size=0 enqueued=1729 
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 10:32:57 2017: main Q: origin=core.queue size=12 enqueued=129 
full=0 discarded.full=0 discarded.nf=0 maxqsize=13

Wed Jan 18 10:37:57 2017: global: origin=dynstats
Wed Jan 18 10:37:57 2017: action 0: origin=core.action processed=0 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: omelasticsearch: origin=omelasticsearch 
submitted=1889 failed.http=0 failed.httprequests=0 failed.checkConn=0 
failed.es=0
Wed Jan 18 10:37:57 2017: json: origin=core.action processed=1889 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: norm: origin=core.action processed=1889 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: elastic: origin=core.action processed=1889 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: error: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: unk: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0

Wed Jan 18 10:37:57 2017: imrelp[20514]: origin=imrelp submitted=1889
Wed Jan 18 10:37:57 2017: resource-usage: origin=impstats utime=1036000 
stime=876000 maxrss=8092 minflt=1500 majflt=0 inblock=10 oublock=45 
nvcsw=11377 nivcsw=129
Wed Jan 18 10:37:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0 
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:37:57 2017: relp: origin=core.queue size=0 enqueued=1889 
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 10:37:57 2017: main Q: origin=core.queue size=12 enqueued=142 
full=0 discarded.full=0 discarded.nf=0 maxqsize=13

Wed Jan 18 10:42:58 2017: global: origin=dynstats
Wed Jan 18 10:42:58 2017: action 0: 

Re: [rsyslog] Out of memory

[mostolog--- via rsyslog]

*Update*

In the log below you may notice suddenly relp queue grow to 35k events 
(that's when the forwarding relay was restarted). And -at the end- it 
drops to 0, as the relay stop forwarding.


Why rsyslog has't been able to reduce queue size from ~35k for quite a 
long time? Is 10k the aprox. maximum ingest with our current config?



Wed Jan 18 09:47:57 2017: global: origin=dynstats
...
Wed Jan 18 10:32:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0 
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:32:57 2017: relp: origin=core.queue size=0 enqueued=1729 
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 10:32:57 2017: main Q: origin=core.queue size=12 enqueued=129 
full=0 discarded.full=0 discarded.nf=0 maxqsize=13

Wed Jan 18 10:37:57 2017: global: origin=dynstats
Wed Jan 18 10:37:57 2017: action 0: origin=core.action processed=0 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: omelasticsearch: origin=omelasticsearch 
submitted=1889 failed.http=0 failed.httprequests=0 failed.checkConn=0 
failed.es=0
Wed Jan 18 10:37:57 2017: json: origin=core.action processed=1889 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: norm: origin=core.action processed=1889 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: elastic: origin=core.action processed=1889 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: error: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:37:57 2017: unk: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0

Wed Jan 18 10:37:57 2017: imrelp[20514]: origin=imrelp submitted=1889
Wed Jan 18 10:37:57 2017: resource-usage: origin=impstats utime=1036000 
stime=876000 maxrss=8092 minflt=1500 majflt=0 inblock=10 oublock=45 
nvcsw=11377 nivcsw=129
Wed Jan 18 10:37:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0 
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:37:57 2017: relp: origin=core.queue size=0 enqueued=1889 
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 10:37:57 2017: main Q: origin=core.queue size=12 enqueued=142 
full=0 discarded.full=0 discarded.nf=0 maxqsize=13

Wed Jan 18 10:42:58 2017: global: origin=dynstats
Wed Jan 18 10:42:58 2017: action 0: origin=core.action processed=0 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:42:58 2017: omelasticsearch: origin=omelasticsearch 
submitted=70614 failed.http=0 failed.httprequests=0 failed.checkConn=0 
failed.es=0
Wed Jan 18 10:42:58 2017: json: origin=core.action processed=70901 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:42:58 2017: norm: origin=core.action processed=70901 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:42:58 2017: elastic: origin=core.action processed=70901 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:42:58 2017: error: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:42:58 2017: unk: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0

Wed Jan 18 10:42:58 2017: imrelp[20514]: origin=imrelp submitted=104878
Wed Jan 18 10:42:58 2017: resource-usage: origin=impstats utime=15748000 
stime=11172000 maxrss=75992 minflt=18537 majflt=0 inblock=10 oublock=48 
nvcsw=219548 nivcsw=7017
Wed Jan 18 10:42:58 2017: relp[DA]: origin=core.queue size=0 enqueued=0 
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 10:42:58 2017: relp: origin=core.queue size=35001 
enqueued=104879 full=0 discarded.full=0 discarded.nf=0 maxqsize=35003
Wed Jan 18 10:42:58 2017: main Q: origin=core.queue size=12 enqueued=155 
full=0 discarded.full=0 discarded.nf=0 maxqsize=13

Wed Jan 18 10:47:58 2017: global: origin=dynstats
Wed Jan 18 10:47:58 2017: action 0: origin=core.action processed=0 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:47:58 2017: omelasticsearch: origin=omelasticsearch 
submitted=162118 failed.http=0 failed.httprequests=0 failed.checkConn=0 
failed.es=0
Wed Jan 18 10:47:58 2017: json: origin=core.action processed=163061 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:47:58 2017: norm: origin=core.action processed=163061 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:47:58 2017: elastic: origin=core.action processed=163061 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:47:58 2017: error: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 10:47:58 2017: unk: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0

Wed Jan 18 10:47:58 2017: imrelp[20514]: origin=imrelp submitted=196015
Wed Jan 18 10:47:58 2017: resource-usage: origin=impstats utime=37292000 
stime=24932000 maxrss=104684 minflt=25667 majflt=0 inblock=10 oublock=51 
nvcsw=496418 nivcsw=14202

Re: [rsyslog] Out of memory

[mostolog--- via rsyslog]

Hi

This might be not enough yet to diagnose, but here's an spoiler. We have 
managed to review configuration and improve memory footprint, but still 
growing over time and ends being killed by oom:


   module(load="impstats" log.file="/data/stats.log")
   syslog.=debug /data/rsyslog-stats

   global(
MaxMessageSize="32k"
workDirectory="/data"
   parser.escapeControlCharactersOnReceive="off"
   )
   input(
type="imrelp"
ruleset="relp"
   )
   ruleset(
name="relp"
queue.filename="relp"
queue.maxdiskspace="1G"
queue.SaveOnShutdown="on"
queue.type="LinkedList"
) {
action(
type="mmjsonparse"
)
if $parsesuccess == "FAIL" then {
call error #to write error.log
stop
}
action(
type="mmnormalize"
#actually it's doing nothing, cause liblognorm issues.
)
set some variables;
if $!app != $!app then {
call unknown #to write unk.log
stop
}
#this line include files with if's to re-set variables
$IncludeConfig /etc/rsyslog.d/apps/conf/3*.conf
call clean
set a few variables;
action(
type="omelasticsearch"
)
   }
   #include other rulesets to handle other things, like error or unknown
   $IncludeConfig /etc/rsyslog.d/apps/conf/4*.conf


Wed Jan 18 09:47:57 2017: global: origin=dynstats
Wed Jan 18 09:47:57 2017: action 0: origin=core.action processed=0 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: omelasticsearch: origin=omelasticsearch 
submitted=235 failed.http=0 failed.httprequests=0 failed.checkConn=0 
failed.es=0
Wed Jan 18 09:47:57 2017: json: origin=core.action processed=235 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: norm: origin=core.action processed=235 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: elastic: origin=core.action processed=235 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: error: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:47:57 2017: unk: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0

Wed Jan 18 09:47:57 2017: imrelp[20514]: origin=imrelp submitted=235
Wed Jan 18 09:47:57 2017: resource-usage: origin=impstats utime=168000 
stime=124000 maxrss=7828 minflt=1383 majflt=0 inblock=10 oublock=16 
nvcsw=1628 nivcsw=44
Wed Jan 18 09:47:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0 
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 09:47:57 2017: relp: origin=core.queue size=0 enqueued=235 
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 09:47:57 2017: main Q: origin=core.queue size=9 enqueued=12 
full=0 discarded.full=0 discarded.nf=0 maxqsize=9

Wed Jan 18 09:52:57 2017: global: origin=dynstats
Wed Jan 18 09:52:57 2017: action 0: origin=core.action processed=0 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: omelasticsearch: origin=omelasticsearch 
submitted=418 failed.http=0 failed.httprequests=0 failed.checkConn=0 
failed.es=0
Wed Jan 18 09:52:57 2017: json: origin=core.action processed=418 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: norm: origin=core.action processed=418 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: elastic: origin=core.action processed=418 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: error: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:52:57 2017: unk: origin=core.action processed=0 failed=0 
suspended=0 suspended.duration=0 resumed=0

Wed Jan 18 09:52:57 2017: imrelp[20514]: origin=imrelp submitted=418
Wed Jan 18 09:52:57 2017: resource-usage: origin=impstats utime=236000 
stime=232000 maxrss=7828 minflt=1385 majflt=0 inblock=10 oublock=18 
nvcsw=2710 nivcsw=59
Wed Jan 18 09:52:57 2017: relp[DA]: origin=core.queue size=0 enqueued=0 
full=0 discarded.full=0 discarded.nf=0 maxqsize=0
Wed Jan 18 09:52:57 2017: relp: origin=core.queue size=0 enqueued=418 
full=0 discarded.full=0 discarded.nf=0 maxqsize=47
Wed Jan 18 09:52:57 2017: main Q: origin=core.queue size=0 enqueued=25 
full=0 discarded.full=0 discarded.nf=0 maxqsize=9

Wed Jan 18 09:57:57 2017: global: origin=dynstats
Wed Jan 18 09:57:57 2017: action 0: origin=core.action processed=0 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:57:57 2017: omelasticsearch: origin=omelasticsearch 
submitted=574 failed.http=0 failed.httprequests=0 failed.checkConn=0 
failed.es=0
Wed Jan 18 09:57:57 2017: json: origin=core.action processed=574 
failed=0 suspended=0 suspended.duration=0 resumed=0
Wed Jan 18 09:57:57 2017: norm: origin=core.action processed=574 
failed=0 suspended=0 suspended.duration=0 

Re: [rsyslog] Parsing very non standard input

[mostolog--- via rsyslog]

have you tried mmnormalize?


El 18/01/17 a las 09:58, Benoit DOLEZ escribió:

Hi,

I don't find how to properly parse a log from tcp/udp input that do 
not respect standard protocol.


The line received has the format :
  -MM-DD HH:MM:SS HOSTNAME SEVERITY ID MESSAGE

sample:
2016-11-12 10:54:24 TEST.company.corp INFO 2346 This is the message

I want this log format be processed (by my big conf) like others 
BSD/IETF logs :

  - timereported : 2016-11-12 10:54:24
  - hostname (and others) : TEST.company.corp
  - severity : info
  - programname (and others) : ID2346

I tried mmexternal (that give the best result), mmnormalize, ... but 
it seem that I need to write a specific parser module.


Do you know a simplest solution ?

Regards

Benoit



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Out of memory

[mostolog--- via rsyslog]

Hi


At this moment, our testing rsyslog is receiving messages from relay 
through RELP in RFC3164 format with message in JSON like:


  DATE RELAYHOST TAG: 
{"field":"foo",..."data":"original_forwarded_message"}


Once received, mmjsonparse is executed over MSG to check if incoming 
messages "comply", before mmnormalize on data (actually doing nothing 
due to liblognorm bugs already reported) and setting some other fields.


After that, omelasticsearch action is executed and messages are indexed.

We are receiving ~1 million events/hour and rsyslog has 2GB of memory 
and process is killed cause out of memory (every other hour).


What steps could we take to diagnose if that's correct, how to avoid 
it...? Any up to date literature?



Thanks. Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Omkafka error log json addition

[mostolog--- via rsyslog]

Hello all,

I am working on a patch that will add intended topic name to the errorFile 
produced by the omkafka module in the writeDataError function called from the 
deliveryCallback.
Thanks for the contribution. We are actually moving to kafka, so will 
benefit for that :)



The question I have is how to get the dynamic topic name inside either the 
writeDataError or deliveryCallback functions.

The non-dynamic topic name is in the instance data so I can do:
jval = json_object_new_string(rd_kafka_topic_name(pData->pTopic));
json_object_object_add(json, "topic", jval);

Is there an easy way to access the post-template / intended topic name so I can 
just put this in an if/else for the output JSON?

Thank you for any guidance,
Doesn't deliveryCallback get /rd_kafka_t__attribute__//((unused)) 
*rk/ which may contain topic name?


https://github.com/edenhill/librdkafka/blob/b7f57f9863367feff64b743b651f4f51f0934581/src/rdkafka_int.h#L132


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault in libfastjson.so.4.0.0 (with workaround)

[mostolog--- via rsyslog]

Could you create an issue for this on github?

could you also paste some related lines from /opt/secrets/log/8.8.8.8.log ?


El 03/01/17 a las 13:54, Lennard Klein escribió:

Hi list,

I've ran into a segfault in libfastjson.

First some details:
RHEL7 using adiscon repositories:
libfastjson4.x86_64 0.99.4-1.el7
rsyslog.x86_64 8.23.0-1.el7

The following bit of config segfaults rsyslog, as soon as it starts processing 
the first message (its own startup message):
if( $!rfc5424-sd!origin!ip == '8.8.8.8' ) then {
action(type="omfile" file="/opt/secrets/log/8.8.8.8.log" 
template="RSYSLOG_DebugFormat")
& stop
}

Running rsyslog -dn, this is the snippet just before the segfault:
4184.634573199:main Q:Reg/w0  : IF
4184.634578377:main Q:Reg/w0  : var '!rfc5424-sd!origin!ip'
4184.634585979:main Q:Reg/w0  :   ==
4184.634591759:main Q:Reg/w0  : string '172.22.222.17'
4184.634601617:main Q:Reg/w0  : eval expr 0x7f615a4b2020, type 'CMP_EQ'
4184.634604257:main Q:Reg/w0  : eval expr 0x7f615a49ec90, type 'V'

Comparing this to some other debugging output I'm guessing the next step in 
processing is resolving the variable.
As I had a theory the segfault might be due to nothing being in the JSON tree 
yet, I Added the following line just before the if-statement:
set $!foo = "bar";

Which resulted in the message being processed as expected.

As I'm not at all familiar with the rsyslog code base, this is as far as I can 
get in my analysis at the moment. Does anyone have suggestions on how to 
proceed?

Regards,
Lennard Klein
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Handling "log explosions"

[mostolog--- via rsyslog]

If you know that a log message is uninteresting, then you want to 
throw it away, but count how many times it happened because the number 
of times that an uninteresting log happens can be interesting.

That's exactly what I'm looking for




1.rsyslog gets message
2.if it's a notifiable error
 if it's already "notified", increase notify counter
what do you do with the notify counter? do you want it spit out along 
with other stats (in which case dyn_stats is the right answer) or do 
you want to do something else?

I just though to use it as drop criteria. I was thinking something like:

#specific error msg received
increase counter
If counter>=5
  drop message (it will be reported later)
else
  forward

#specific ok message received
counter=0
forward

#scheduled task (each X minutes)
report current_counter
counter=0




   drop message (don't log it)
 else #not "notified"
   notify error
   log message
is there anything that resets "not notified"? or do you only want one 
notification per syslog startup.

See above




3.if it's a recovery
   log recovery
   notify recovered
4.otherwise, log normally

In the above, notification could be a snmptrap to our monitoring 
system, and "if notified" could be a "global" errorCount variable or 
something similar...
keep in mind there are the global variables $\ that you can use for 
this sort of flag, but checking them is relatively expensive, so you 
should think about what you are really wanting here.


It may be good enough to not do any tracking of 'already notified' and 
instead just do


if  then increase counter

and then spit the counters out to your monitoring system. If there 
were no messages of that type, you have no message to your monitoring 
system. If there were messages of that type, you have a notification 
of how many times it happened that monitoring period. If the 
monitoring period is relatively short (say 1-5 min), this may be 
sufficient for your system.
Consider that I'm also trying to save "disk space" ie: not storing huge 
amount of "droppeable" events


Seems dyn_stats is a good alternative, but I haven't time yet to play 
with it.

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Wildcard Support to log all the file logs of a directory

[mostolog--- via rsyslog]

imfile module use state files to save each file "reading position", 
among other things.


maybe you should delete them?

Have a look at:

https://raw.githubusercontent.com/mostolog/rsyslog-doc/patch-1/source/overview.rst

https://raw.githubusercontent.com/mostolog/rsyslog-doc/imfile/source/configuration/modules/imfile.rst



El 30/12/16 a las 11:39, Shweta escribió:

Thank you for your response.

I also tried file monitoring using the Wildcard on Rsyslog version 8.23.
The directory had 4 files (file1.txt, file2.txt, file3.txt, file4.log).

The file path I passed was /directory/*.txt

But only file1.txt logs reaches to loggly. What can I do to make wildcard
working.

Additionally, I have verified that the kernel of my machine is inotify
supportable.



On Fri, Dec 30, 2016 at 3:36 PM, rsyslog-users mailing list [via
rsyslog-users]  wrote:


Latest rsyslog version is 8.23
http://www.rsyslog.com/downloads/download-v8-stable/

AFAIK, in order to use imfile, you should run ./configure
--enable-imfile before building source

IIRC, inotify is set by default on recent versions


El 30/12/16 a las 11:03, Shweta escribió:


Yes, I am using latest version of rsyslog i.e., 8.5 which support

wildcards

as per the slide below:

http://www.slideshare.net/rainergerhards1/using-

wildcards-with-rsyslogs-file-monitor-imfile

I am not getting what to build by --enable-imfile?

But you can see my configuration in my previous responses. Can you also
suggest me how to use kernel inotify mode instead of polling mode?

Thanks,
Shweta


On Fri, Dec 30, 2016 at 3:23 PM, rsyslog-users mailing list [via
rsyslog-users] <[hidden email]

> wrote:

Hi Shweta

Have you built with --enable-imfile ?

I'm not an expert on rsyslog, but I would suggest you a few things:

- use new syntax
- use latest rsyslog version

Then, perhaps I could help you...


El 30/12/16 a las 09:36, Shweta Jain escribió:


Any update?

On Thu, Dec 29, 2016 at 6:37 PM, Shweta Jain <[hidden email]

> wrote:

Update:

I tried as per link here: http://www.slideshare.net/

rainergerhards1/using-

wildcards-with-rsyslogs-file-monitor-imfile

I installed Rsyslog 8.5 using tarball. But it seems that file logs

are

not

getting logged to Loggly. My system logs are successfully reaching to
Loggly using Rsyslog version 8.5.

I am getting the below in my logs:

invalid or yet-unknown config file command 'InputRunFileMonitor' -

have

you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ]


My configuration is:

   $ModLoad imfile
   $InputFilePollInterval 10
   $WorkDirectory /var/spool/rsyslog
   $PrivDropToGroup adm

   # File access file:
   $InputFileName /shweta/file1.txt
   $InputFileTag yoo
   $InputFileStateFile stat-yoo
   $InputFileSeverity info
   $InputFilePersistStateInterval 2
   $InputRunFileMonitor
   #Add a tag for file events
   template (name="LogglyFormatFileyoo" type="string"
string="<%pri%>%protocol-version% %timestamp:::date-rfc3339%

%HOSTNAME%

%app-name% %procid% %msgid% [d57950de-c677-4af7-aeee-

b9647ea54b1c@41058

tag=\"file\" ] %msg%\n")
   if $programname == 'yoo' then action(type="omfwd"
protocol="tcp" target="logs-01.loggly.com" port="514"
template="LogglyFormatFileyoo")
   if $programname == 'yoo' then ~

Is it using Polling mode or inotify? Where to set inotify mode?

Quick response will be appreciated.

Thanks,
Shweta


On Thu, Dec 29, 2016 at 4:21 PM, Shweta Jain <[hidden email]

> wrote:

Sorry I missed the link in my previous response:

http://www.slideshare.net/rainergerhards1/using-wildcards-
with-rsyslogs-file-monitor-imfile

On Thu, Dec 29, 2016 at 4:06 PM, Shweta <[hidden email]

> wrote:

I did setup on ubuntu 12 with rsyslog version 8.23 but nothing get

logged

to my local as well as loggly.

I also tried on ubuntu-16 with rsyslog version 8.5, but nothing is
getting
logged.

Moreover , I went thorough the slide at the link below, where it is
mentioned to use inotify mode. Where to set inotify mode?

My second issue is I am not able to log anything at

/var/log/syslog.

Even i
am not seeing syslog file in /var/log. I have checked

/etc/rsyslog.conf

file but the line in that file is uncommented.

Thanks for you quick response.
Shweta

On Thu, Dec 29, 2016 at 2:09 PM, David Lang [via rsyslog-users] <
[hidden email]

> wrote:

you don't say what version you are running, the most current docs

are

at:

http://www.rsyslog.com/doc/v8-stable/configuration/modules/

imfile.html

I'm not sure exactly when wildcards became 

Re: [rsyslog] Wildcard Support to log all the file logs of a directory

[mostolog--- via rsyslog]

Latest rsyslog version is 8.23 
http://www.rsyslog.com/downloads/download-v8-stable/


AFAIK, in order to use imfile, you should run ./configure 
--enable-imfile before building source


IIRC, inotify is set by default on recent versions


El 30/12/16 a las 11:03, Shweta escribió:

Yes, I am using latest version of rsyslog i.e., 8.5 which support wildcards
as per the slide below:

http://www.slideshare.net/rainergerhards1/using-wildcards-with-rsyslogs-file-monitor-imfile

I am not getting what to build by --enable-imfile?

But you can see my configuration in my previous responses. Can you also
suggest me how to use kernel inotify mode instead of polling mode?

Thanks,
Shweta


On Fri, Dec 30, 2016 at 3:23 PM, rsyslog-users mailing list [via
rsyslog-users]  wrote:


Hi Shweta

Have you built with --enable-imfile ?

I'm not an expert on rsyslog, but I would suggest you a few things:

   - use new syntax
   - use latest rsyslog version

Then, perhaps I could help you...


El 30/12/16 a las 09:36, Shweta Jain escribió:


Any update?

On Thu, Dec 29, 2016 at 6:37 PM, Shweta Jain <[hidden email]

> wrote:

Update:

I tried as per link here: http://www.slideshare.net/

rainergerhards1/using-

wildcards-with-rsyslogs-file-monitor-imfile

I installed Rsyslog 8.5 using tarball. But it seems that file logs are

not

getting logged to Loggly. My system logs are successfully reaching to
Loggly using Rsyslog version 8.5.

I am getting the below in my logs:

invalid or yet-unknown config file command 'InputRunFileMonitor' - have
you forgotten to load a module? [try http://www.rsyslog.com/e/3003 ]


My configuration is:

  $ModLoad imfile
  $InputFilePollInterval 10
  $WorkDirectory /var/spool/rsyslog
  $PrivDropToGroup adm

  # File access file:
  $InputFileName /shweta/file1.txt
  $InputFileTag yoo
  $InputFileStateFile stat-yoo
  $InputFileSeverity info
  $InputFilePersistStateInterval 2
  $InputRunFileMonitor
  #Add a tag for file events
  template (name="LogglyFormatFileyoo" type="string"
string="<%pri%>%protocol-version% %timestamp:::date-rfc3339%

%HOSTNAME%

%app-name% %procid% %msgid% [d57950de-c677-4af7-aeee-b9647ea54b1c@41058
   tag=\"file\" ] %msg%\n")
  if $programname == 'yoo' then action(type="omfwd"
protocol="tcp" target="logs-01.loggly.com" port="514"
template="LogglyFormatFileyoo")
  if $programname == 'yoo' then ~

Is it using Polling mode or inotify? Where to set inotify mode?

Quick response will be appreciated.

Thanks,
Shweta


On Thu, Dec 29, 2016 at 4:21 PM, Shweta Jain <[hidden email]

> wrote:

Sorry I missed the link in my previous response:

http://www.slideshare.net/rainergerhards1/using-wildcards-
with-rsyslogs-file-monitor-imfile

On Thu, Dec 29, 2016 at 4:06 PM, Shweta <[hidden email]

> wrote:

I did setup on ubuntu 12 with rsyslog version 8.23 but nothing get

logged

to my local as well as loggly.

I also tried on ubuntu-16 with rsyslog version 8.5, but nothing is
getting
logged.

Moreover , I went thorough the slide at the link below, where it is
mentioned to use inotify mode. Where to set inotify mode?

My second issue is I am not able to log anything at /var/log/syslog.
Even i
am not seeing syslog file in /var/log. I have checked

/etc/rsyslog.conf

file but the line in that file is uncommented.

Thanks for you quick response.
Shweta

On Thu, Dec 29, 2016 at 2:09 PM, David Lang [via rsyslog-users] <
[hidden email]

> wrote:

you don't say what version you are running, the most current docs

are

at:

http://www.rsyslog.com/doc/v8-stable/configuration/modules/

imfile.html

I'm not sure exactly when wildcards became supported, but I believe

it

was

well
into the 8.x series.

David Lang

On Wed, 28 Dec 2016, Shweta wrote:


Date: Wed, 28 Dec 2016 23:21:14 -0700 (MST)
From: Shweta <[hidden email]

>

Reply-To: rsyslog-users <[hidden email]

>

To: [hidden email]



Subject: [rsyslog] Wildcard Support to log all the file logs of a

directory

Hi Support,

I want to monitor file logs at Loggly. I have many files in a

directory

and

want to monitor them all with a wildcard. I have a configuration

like

below:

$ModLoad imfile
$InputFilePollInterval 10
$PrivDropToGroup adm
$WorkDirectory /var/spool/rsyslog

# Input for FILE1
$InputFileName /FILE1
$InputFileTag APPNAME1
$InputFileStateFile stat-APPNAME1 #this must be unique for each

file

being

polled
$InputFileSeverity info

Re: [rsyslog] Handling "log explosions"

[mostolog--- via rsyslog]

Maybe that could work. Thanks


El 29/12/16 a las 13:02, Benoit DOLEZ escribió:

Hi,

Does this documentation answer your needs ?

http://www.rsyslog.com/doc/v8-stable/configuration/dyn_stats.html

Regards

Benoit

Le 29/12/2016 à 12:27, mostolog--- via rsyslog a écrit :



impstats

Ok. I'll have a look


define 'too fast'. And how do you tell the difference between your
logging system having a problem and generating so many messages and
the systems you are collecting logs from generating the messages?

If you setup thresholds and start throwing away messages that arrive
faster than that, I will guarantee that one day you will find that you
are throwing away legitimate logs that you care about as a result.

It's also rather tricky to define such threshold criteria, and
expensive to track.

much better to just report the stats to your existing monitoring
system and use it to decide that something is wrong.


During tests, as omelasticsearch is not v5 fully-compatible, it started
to write warnings for each indexing operation. As we were making
high-traffic tests, our /var/log/messages filled with "deprecated:
blablabla" and we're now thinking how to handle those situations.

As we "can't" change how applications log (hence, elastic will continue
to write those warnings), and considering we would like to get notified
about errors on certain procedures (eg: error when creating indexes) I
was wondering if theres something like flags/counters:

 1.rsyslog gets message
 2.if it's a notifiable error
  if it's already "notified", increase notify counter
drop message (don't log it)
  else #not "notified"
notify error
log message
 3.if it's a recovery
log recovery
notify recovered
 4.otherwise, log normally


In the above, notification could be a snmptrap to our monitoring system,
and "if notified" could be a "global" errorCount variable or something
similar...

Hope explained properly...

Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.





___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Handling "log explosions"

[mostolog--- via rsyslog]

impstats

Ok. I'll have a look

define 'too fast'. And how do you tell the difference between your 
logging system having a problem and generating so many messages and 
the systems you are collecting logs from generating the messages?


If you setup thresholds and start throwing away messages that arrive 
faster than that, I will guarantee that one day you will find that you 
are throwing away legitimate logs that you care about as a result.


It's also rather tricky to define such threshold criteria, and 
expensive to track.


much better to just report the stats to your existing monitoring 
system and use it to decide that something is wrong.


During tests, as omelasticsearch is not v5 fully-compatible, it started 
to write warnings for each indexing operation. As we were making 
high-traffic tests, our /var/log/messages filled with "deprecated: 
blablabla" and we're now thinking how to handle those situations.


As we "can't" change how applications log (hence, elastic will continue 
to write those warnings), and considering we would like to get notified 
about errors on certain procedures (eg: error when creating indexes) I 
was wondering if theres something like flags/counters:


 1.rsyslog gets message
 2.if it's a notifiable error
  if it's already "notified", increase notify counter
drop message (don't log it)
  else #not "notified"
notify error
log message
 3.if it's a recovery
log recovery
notify recovered
 4.otherwise, log normally


In the above, notification could be a snmptrap to our monitoring system, 
and "if notified" could be a "global" errorCount variable or something 
similar...


Hope explained properly...

Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Handling "log explosions"

[mostolog--- via rsyslog]

El 28/12/16 a las 22:41, David Lang escribió:

On Wed, 28 Dec 2016, mostolog--- via rsyslog wrote:

While testing our current infrastructure we have suffered a /log 
explosion/, ie: errors when processing logs caused error logs on the 
machine that also caused errors when processed...and finally, disk 
became full and everything died.


I'm wondering if worrying about this is useful, or how could it be 
managed/prevented (as automatically as possible).

...


monitor disk space and alert if it starts filling up.
Traditionally this has been done using cron, but we just found 
inotify/fanotify :P



monitor logs/sec and alert if they jump much higher than normal

How could I do this using rsyslog? Is there a "counter" module?

overall, this isn't likely to happen once you get the system setup and 
running, so many places don't do anything special for this at all.
Our main concern is that, in the event of failure, as we are processing 
a lot of events it may fill disk with error messages tooo fast


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Does it make sense to have a queue with imfile+omrelp?

[mostolog--- via rsyslog]

Thanks for your clarifying answers.


El 28/12/16 a las 22:38, David Lang escribió:

On Wed, 28 Dec 2016, mostolog--- via rsyslog wrote:


Even more: does it make sense to have queues when using omfile?


usually not, it's usually less effort to write the data to the file 
than to move it to a new queue.


You want a queue on an output (or on a ruleset with multiple outputs) 
if you think there can be problems with an output (either that it 
can't keep up with peak load, or that it may fail entirely for some 
time[1]) and you want to me sure that you can keep processing logs to 
other outputs


imfile keeps track of where it is, so if it can't process messages, 
none are lost[2], so you may not need a queue if your only source is 
imfile.


The other reason for having a queue involved is to allow batching of 
messages. There are a lot of cases where it is far more efficient to 
process multiple messages at a time instead of one at a time.


For example, RELP can send multiple messages while waiting for acks 
for the first message sent and so will operate much faster if it has a 
queue of multiple messages available to send then if it must wait for 
the ack for each message before starting to send the next message.


Another example, I've measured databases where inserting 1000 messages 
as a batch took the same time as inserting 2 messages independently.



As a general rule, you will want to have a queue on anything that 
sends to a network/remote system, and may want to have a queue on 
sending to complex servers on local systems (i.e. databases) so that 
you can keep accepting new messages when the destinations are slow or 
unavailable.


But to have a queue on omfile is seldom appropriate because if you do 
have a problem that prevents the log from being written, you are 
unlikely to be able to do anything else anyway.


David Lang


[1] network outages, remote systems being rebooted, etc

[2] except in the case where you have file rotations going on and 
multiple rotations happen while you are stopped


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Handling "log explosions"

[mostolog--- via rsyslog]

Hello

While testing our current infrastructure we have suffered a /log 
explosion/, ie: errors when processing logs caused error logs on the 
machine that also caused errors when processed...and finally, disk 
became full and everything died.


I'm wondering if worrying about this is useful, or how could it be 
managed/prevented (as automatically as possible).


For example:

 * Rate-limiting for specific log events (eg: rate limiting events with
   syslogtag="foo" or matching a filter)
 * Having counters and ignore events of /type/ if more than N /
   last_X_minutes
 * Being able to reduce rsyslog verbosity, logging "fail and recover"
   messages, instead of logging an error on each failure.

How do you handle those situations? Should we stop worrying about things 
that haven't happened and probably won't ever happen?


Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Does it make sense to have a queue with imfile+omrelp?

[mostolog--- via rsyslog]

Even more: does it make sense to have queues when using omfile?

El 28/12/16 a las 15:52, mosto...@gmail.com escribió:


Hi

Does it make any sense to use queues when reading a file (imfile) and 
forwarding to a central location using RELP (omrelp) ?


We would like to read & send events in a reliable way (reading 
offset/position is only updated if successfully received on server). 
but I don't understand why queues would make sense in such scenario.


Would the following configuration be enough then?

input(
file="*.log"
ruleset="files"
)
ruleset(name="files"
) {
action(
name="relp"
type="omrelp"
server="relay-cluster"
)
}

Regards (and happy holidays)



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Does it make sense to have a queue with imfile+omrelp?

[mostolog--- via rsyslog]

Hi

Does it make any sense to use queues when reading a file (imfile) and 
forwarding to a central location using RELP (omrelp) ?


We would like to read & send events in a reliable way (reading 
offset/position is only updated if successfully received on server). but 
I don't understand why queues would make sense in such scenario.


Would the following configuration be enough then?

input(
file="*.log"
ruleset="files"
)
ruleset(name="files"
) {
action(
name="relp"
type="omrelp"
server="relay-cluster"
)
}

Regards (and happy holidays)

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Deleting already "processed" files

[mostolog--- via rsyslog]

well, if logs are redirected to the host syslog, you don't have the 
logs in files for imfile to have to read, you have them in rsyslog (or 
the systemd journal, but that's your problem if you do :-)


if you have logs that are written to disk, then you have two choices

1. have something tail the logs as they are written, processing the 
logs in near-real-time


2. wait for a log rotation point and at that time, take the rotated 
log (after some delay to give the app writing to it a chance to 
actually close it), and process them.


processing the completed log files can be done in a couple of 
different ways, but fundamentally you are running a script to do the 
processing, so it can do anything


2a. move/copy the files someplace for processing (which could even be 
a different machine)


2b. run a process to read the file and do something with it.

2b can be to run logger against the file and then rm the file after 
logger completes.


That sounds overcomplicated to me, maybe I'm missing something.
Considering each application logs to /appName--MM-DD-HH.log, /we 
could run a cron job (eg:hourly) which compares state file "reading 
position" with file size for older logs (eg: 2 hour ago), deleting if 
they are equal.


Shouldn't that work if we're sending with RELP?


By spec, the write to /dev/log will not complete until rsyslog has the
message in it's queue. So your script can feed the log to rsyslog and
remove it immediatly after it's completed.

I'm not sure if I understood this. Could you elaborate?
The reason that sending machines can stall and not even let you login 
when sending to a remote machine stops is because the syslog spec says 
that a write to /dev/log is not supposed to return until the log 
message is safe on disk (including fsync) and would survive a reboot.

Didn't know that. Do you have some literature about this?

AFAIK "my logs" are written to /var/log/messages, not using /dev/log. I 
guess these doesn't behave that way (not waiting for fsync) and that's 
much faster, isnt it? Otherwise, why aren't logs written on /dev/log?


Rsyslog relaxes this so that it lets the write return as soon as the 
log message is in the rsyslog queue. But if there isn't room in the 
queue (because output has stopped), the write to /dev/log is unable to 
complete.
And, in the event of an error, queue messages that weren't written on 
/dev/log are lost, right?


Thanks. Did I said I'm learning a lot here?
:P
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Deleting already "processed" files

[mostolog--- via rsyslog]

I'll create an issue for that.

As a workaround until this feature is implemented, do you think it will 
be possible to parse state file to get "reading position" on a bash script?

Is there any documentation on state file internal structure?


El 22/12/16 a las 09:48, Rainer Gerhards escribió:
The discussion has moved further, but I'd like to say that the ability 
to delete or move a file after certain period of inactivity sounds 
like a good-to-have feature to me. It may even not be much work (maybe 
something Pascal could work on in q1/q2 of 2016). So possibly 
something worth an enhancement PR. But it won't be prioprity from my 
side.


Rainer

2016-12-21 19:34 GMT+01:00 mostolog--- via rsyslog 
<rsyslog@lists.adiscon.com <mailto:rsyslog@lists.adiscon.com>>:


Hi


To contextualize:

A server is generating multiple log files and sending them using
RELP to a Rsyslog relay. I would love to automatically delete
older logs files if they are already sent/acknowledged.


Is there a way to make Rsyslog delete a file after successfully
sending it via RELP? (and not modified in the last...day?)

Is there an easy way to know, outside from Rsyslog, if a file has
already been processed/sent, in order to delete the local file?

Maybe an script which gets "reading offset" from state file and
compares with file size...


Regards




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
<http://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/professional-services/
<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Liblognorm - Usage of field type iptables

[mostolog via rsyslog]

Sorry. Just awake and didnt look for it before asking...

On Thursday, December 22, 2016, mostolog  wrote:

> Rainer: is liblognorm doc also at github using rst format?
>
> On Wednesday, December 21, 2016, matthew.gaetano  > wrote:
>
>> Alright! Almost out the door for the holiday's and swore i would come
>> back to
>> this before I left in the event anyone stumbled across it. Note that this
>> is
>> intended to primary provide information on using the current iptables data
>> type in version 2 of liblognorm.
>>
>> For reference tests were conducted on CentOS Linux Release 7.2.1511 with
>> liblognorm5-2.0.2-1 and liblognorm5-utils-2.0.2-1 installed from the
>> Addison
>> repository.
>>
>>
>>
>> The documentation for liblognorm is significantly lacking for its current
>> version and the iptables section demonstrates that severely. In version
>> 2.x
>> the data type "iptables" no longer exists, instead it has been replaced by
>> its predecessor "v2-iptables". As per the documentation "v2-iptables"
>> works
>> exactly as stated:
>>
>> "Name=value pairs, separated by spaces, as in Netfilter log messages. Name
>> of the selector is not used; names from the line are used instead. This
>> selector always matches everything till end of the line. Cannot match zero
>> characters."
>>
>> However there are some additional restrictions that are not mentioned.
>> These
>> can be seen when checking out the test files under
>> "https://github.com/rsyslog/liblognorm/blob/master/tests; and they are as
>> follows:
>>
>> 1. Key names can only be uppercase
>> 2. Key name can only contain alphabetic characters (no special characters
>> or
>> numbers)
>> 3. All Key-value pairs must be separated by a single space
>> 4. A minimum of two pairs are required; a single key-value pair will not
>> work
>> 5. value will contain all characters until the next space, this includes
>> any
>> notation like brackets that are not necessarily part of said value.
>>
>>
>> A quick working test:
>>test.rb:
>>version=2
>>rule=:%field:v2-iptables%
>>
>>lognormalizer:
>>echo 'KA=test1 KB="test2"' | lognormalizer -e json -r test.rb
>>
>>
>>
>>
>> It appears that iptables was built for a specific use case, aka Netfilter,
>> and then never adapted to be more applicable to key-value's. Other data
>> types exist like "cee" and "cef" who more accurately act as expected,
>> though
>> when i started i simple assumed that they in themselves made use of the
>> iptables data type.
>>
>> The problem now is that other formats that utilize key-value pairs cant be
>> parsed, especially as most of the time they do not come in any particular
>> order. One could possible use the "alternative" and "repeat" types to
>> solve
>> the problem but this A) seems incredibly inefficient and B) is limited to
>> know key names. The more arbitrary the key-value output the harder it
>> becomes. CEF is very good example of this, though, lucky we have a parser
>> for it. The problem is when we start to look similar formats like MEF,
>> LEEF,
>> or Fortinet logs.
>>
>> CEF and CEE data types are locked down to a very specfic header that can
>> not
>> be altered. In terms of CEF this also problematic because the RFC3164
>> parser
>> will often extract the "CEF:" part of the message as the syslog tag thus
>> making it difficult (not impossible) to use the "CEF" data type.
>>
>> *Note: a new CEF version is expected sometime next year
>>
>>
>> IMO it seems that the CEE and CEF specific data types shouldn't even
>> exist,
>> or if they do, then only as an abstraction; something like:
>>
>> type=@CEF:CEF:0|%vendor:chart-to:|%|%product:chart-to:|%|%ve
>> rsion:chart-to:|%|%signature:chart-to:|%|%name:chart-to:|%|
>> %severity:chart-to:|%|%extensions:iptables%
>>
>> It would also be advantages to assume that the key-value pair might not
>> always be separated by a single space, or have values encased in standard
>> brackets (ie. ", '). Having advanced options to alter those would increase
>> its flexibility.
>>
>> I also do not necessarily agree with the the whole "until end of line"
>> portion. I see that as something that should only ever apply when using
>> the
>> data type "rest" as it is feasible that a portion contain key-values (or
>> other type) but does is proceed by something else.
>>
>> At present I do not have any examples but could probably dig up some
>> vendor
>> sources/docs that might. I have seen in the past custom client
>> applications
>> do awkward things as well as seeing key-value pair submitted encased in
>> other formats; tho i cant really supply those as examples :(
>>
>>
>> Thanks
>>
>> ~Regards
>>
>>
>>
>>
>>
>> -
>> ~Regards
>>
>> Matthew Gaetano
>> --
>> View this message in context: http://rsyslog-users.1305293.n
>> 2.nabble.com/Liblognorm-Usage-of-field-type-iptables-tp75914
>> 41p7592021.html
>> Sent from 

Re: [rsyslog] Liblognorm - Usage of field type iptables

[mostolog via rsyslog]

Rainer: is liblognorm doc also at github using rst format?

On Wednesday, December 21, 2016, matthew.gaetano 
wrote:

> Alright! Almost out the door for the holiday's and swore i would come back
> to
> this before I left in the event anyone stumbled across it. Note that this
> is
> intended to primary provide information on using the current iptables data
> type in version 2 of liblognorm.
>
> For reference tests were conducted on CentOS Linux Release 7.2.1511 with
> liblognorm5-2.0.2-1 and liblognorm5-utils-2.0.2-1 installed from the
> Addison
> repository.
>
>
>
> The documentation for liblognorm is significantly lacking for its current
> version and the iptables section demonstrates that severely. In version 2.x
> the data type "iptables" no longer exists, instead it has been replaced by
> its predecessor "v2-iptables". As per the documentation "v2-iptables" works
> exactly as stated:
>
> "Name=value pairs, separated by spaces, as in Netfilter log messages. Name
> of the selector is not used; names from the line are used instead. This
> selector always matches everything till end of the line. Cannot match zero
> characters."
>
> However there are some additional restrictions that are not mentioned.
> These
> can be seen when checking out the test files under
> "https://github.com/rsyslog/liblognorm/blob/master/tests; and they are as
> follows:
>
> 1. Key names can only be uppercase
> 2. Key name can only contain alphabetic characters (no special characters
> or
> numbers)
> 3. All Key-value pairs must be separated by a single space
> 4. A minimum of two pairs are required; a single key-value pair will not
> work
> 5. value will contain all characters until the next space, this includes
> any
> notation like brackets that are not necessarily part of said value.
>
>
> A quick working test:
>test.rb:
>version=2
>rule=:%field:v2-iptables%
>
>lognormalizer:
>echo 'KA=test1 KB="test2"' | lognormalizer -e json -r test.rb
>
>
>
>
> It appears that iptables was built for a specific use case, aka Netfilter,
> and then never adapted to be more applicable to key-value's. Other data
> types exist like "cee" and "cef" who more accurately act as expected,
> though
> when i started i simple assumed that they in themselves made use of the
> iptables data type.
>
> The problem now is that other formats that utilize key-value pairs cant be
> parsed, especially as most of the time they do not come in any particular
> order. One could possible use the "alternative" and "repeat" types to solve
> the problem but this A) seems incredibly inefficient and B) is limited to
> know key names. The more arbitrary the key-value output the harder it
> becomes. CEF is very good example of this, though, lucky we have a parser
> for it. The problem is when we start to look similar formats like MEF,
> LEEF,
> or Fortinet logs.
>
> CEF and CEE data types are locked down to a very specfic header that can
> not
> be altered. In terms of CEF this also problematic because the RFC3164
> parser
> will often extract the "CEF:" part of the message as the syslog tag thus
> making it difficult (not impossible) to use the "CEF" data type.
>
> *Note: a new CEF version is expected sometime next year
>
>
> IMO it seems that the CEE and CEF specific data types shouldn't even exist,
> or if they do, then only as an abstraction; something like:
>
> type=@CEF:CEF:0|%vendor:chart-to:|%|%product:chart-to:|%|%
> version:chart-to:|%|%signature:chart-to:|%|%name:
> chart-to:|%|%severity:chart-to:|%|%extensions:iptables%
>
> It would also be advantages to assume that the key-value pair might not
> always be separated by a single space, or have values encased in standard
> brackets (ie. ", '). Having advanced options to alter those would increase
> its flexibility.
>
> I also do not necessarily agree with the the whole "until end of line"
> portion. I see that as something that should only ever apply when using the
> data type "rest" as it is feasible that a portion contain key-values (or
> other type) but does is proceed by something else.
>
> At present I do not have any examples but could probably dig up some vendor
> sources/docs that might. I have seen in the past custom client applications
> do awkward things as well as seeing key-value pair submitted encased in
> other formats; tho i cant really supply those as examples :(
>
>
> Thanks
>
> ~Regards
>
>
>
>
>
> -
> ~Regards
>
> Matthew Gaetano
> --
> View this message in context: http://rsyslog-users.1305293.
> n2.nabble.com/Liblognorm-Usage-of-field-type-iptables-
> tp7591441p7592021.html
> Sent from the rsyslog-users mailing list archive at Nabble.com.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, 

Re: [rsyslog] Deleting already "processed" files

[mostolog via rsyslog]

On Wednesday, December 21, 2016, David Lang <da...@lang.hm> wrote:

> On Wed, 21 Dec 2016, mostolog--- via rsyslog wrote:
>
> To contextualize:
>>
>> A server is generating multiple log files and sending them using RELP to
>> a Rsyslog relay. I would love to automatically delete older logs files if
>> they are already sent/acknowledged.
>>
>>
>> Is there a way to make Rsyslog delete a file after successfully sending
>> it via RELP? (and not modified in the last...day?)
>>
>> Is there an easy way to know, outside from Rsyslog, if a file has already
>> been processed/sent, in order to delete the local file?
>>
>> Maybe an script which gets "reading offset" from state file and compares
>> with file size...
>>
>
> Rsyslog can't really know when a file isn't going to be written to any
> longer. It doesn't even know if another program has the file open to write
> to now.

Ok


> In your situation, I would use a script with logger to write the messages
> to /dev/log for rsyslog to process.

If not sure if I understood. Actually I'm using docker containers and logs
are redirected to host syslog. How would you config (eg: apache) to do this?


> By spec, the write to /dev/log will not complete until rsyslog has the
> message in it's queue. So your script can feed the log to rsyslog and
> remove it immediatly after it's completed.

I'm not sure if I understood this. Could you elaborate?


> I'd put in an enhancement request for an option to imfile to delete files
> when it's finished reading them. It was created with the assumption that
> it's reading the file as it's being written, but I can see useful cases
> where the files are made available to rsyslog (mv into a watched directory)
> after they are complete.

I'll create the issue.

Thanks!
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Deleting already "processed" files

[mostolog--- via rsyslog]

Hi


To contextualize:

A server is generating multiple log files and sending them using RELP to 
a Rsyslog relay. I would love to automatically delete older logs files 
if they are already sent/acknowledged.



Is there a way to make Rsyslog delete a file after successfully sending 
it via RELP? (and not modified in the last...day?)


Is there an easy way to know, outside from Rsyslog, if a file has 
already been processed/sent, in order to delete the local file?


Maybe an script which gets "reading offset" from state file and compares 
with file size...



Regards




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] About reloading configuration

[mostolog--- via rsyslog]

Someone called Rainer is going to have a busy January :P

David/Rainer review that docs I pulled so I can write more!


El 21/12/16 a las 17:48, Rainer Gerhards escribió:

I don't have the time to check now, but it may be easier than it looks. I
think we have pdag handles, which means we can easily load in the
background and apply when ready. However, we need a rwlock to guard this
operation (action level).

If it works that way, we could do it on HUP (for starters).

Rainer

2016-12-21 17:37 GMT+01:00 David Lang <da...@lang.hm>:


I think the headache with doing this in liblognorm is the overhead that
can be there in reading/parsing/optimizing the ruleset.

if you have multiple worker threads, having two of them hit the
mmnormalize action while one is rebuilding the ruleset leads to problems :-)

what would be needed is for all existing parsing to continue to use the
old ruleset until the new ruleset is fully parsed, optimized, and ready for
use.

We have a similar issue with the reload of a lookup_table and should be
able to use a similar solution.

One complicating factor is that while a lookup_table has a global
existance and is something the rsyslog core knows about, mmnormalize
rulesets are handled only by the mmnormalize module code, and there's no
clean way to refer to a specific ruleset outside of that code.

David Lang


On Wed, 21 Dec 2016, Benoit DOLEZ wrote:

Yes, I think you are right about reloading only liblognorm. It's probably

my first step.

In most case I think we need a simple rainer-script if/then/else using a
liblognorm app specific rule file. Then adding a new parser implies
reloading rsyslog...

Thank you for responses. I'm continuing to analyse source...

Regards

Benoit

Le 21/12/2016 à 10:33, mostolog--- via rsyslog a écrit :


You may also find interesting to be able to reload liblognorm rules,
which might be an smaller change.

The question here is the needed amount of work.


El 21/12/16 a las 10:29, Rainer Gerhards escribió:


Possible of course, but considerable work (gut feeling: 6+ strong
weeks).

Rainer

Sent from phone, thus brief.

Am 21.12.2016 09:42 schrieb "Benoit DOLEZ" <bdo...@pom-monitoring.com>:

Hi

I saw with previous posts that it seems to be complicated to reload
configuration; this is well explained here :
http://www.rsyslog.com/doc/v8-stable/compatibility/v4compati
bility.html#hup-processing.

I wonder if the architecture of rulesets/queues could be flexible
enough
to implement a reload operation limited to rulesets. Do you think it is
possible to create new queues/rulesets/actions from a new config and
mark
the old ones for removal once idle/empty ?

Regards

Benoit
--
Benoit DOLEZ, POM Monitoring, http://www.pom-monitoring.com/
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

___

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.



___

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a m

Re: [rsyslog] saving liblognorm literals

[mostolog--- via rsyslog]

El 21/12/16 a las 17:46, Rainer Gerhards escribió:

see https://github.com/rsyslog/liblognorm/pull/238

wow! that was fast!




2016-12-21 16:54 GMT+01:00 David Lang :

Can you explain your ruleset where you need to store literal as a 
value in the json?


I think the original thinking was that since this is a fixed value, 
storing it as a variable doesn't help.


slapd [1] messages were processed in grok with the following expression:

   MYEXPR ^(.*?)conn=(?\d+) (fd=(?\d+)|op=(?\d+))
   ((?ACCEPT) from IP=%{IPORHOST:ip}|(%{WORD} )?RESULT
   (?.*)|%{WORD} (?attr=.*)|%{WORD:cmd}( (?.*))?)

The relevant part of that is that messages can have 5 formats:

   ACCEPT from IP=...
   FOO RESULT...  # FOO can be different words
   RESULT...
   BAR attr=... # BAR can be different words
   FOOBAR anything # FOOBAR can be different words
   # anything is the rest (not one of the above)

And with that grok we were indexing them "properly"...

[1] http://www.openldap.org/software/man.cgi?query=slapd
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] omelasticsearch index warning

[mostolog--- via rsyslog]

Thanks for your answers.

I'll also try to have a look on this.

El 21/12/16 a las 17:48, matthew.gaetano escribió:

Though the examples given are specific modifications for aspects of
Elasticsearch 5.X i believe the modification method would be valid for any
version of Elasticsearch supported by omelasticsearch. To be honest I am not
familiar with how omelasticsearch interfaces with ES (which is using a json
body), where as I am more familiar with the direct head requests to ES.

  It is through assumptions and trial/error that this works. And I assume any
index related settings could be used.

ES version is based on node, so the cluster doesn't have an overall ES
version listed (as far as I can tell), so one would have to query the node
that the request is being sent to. Using curl it would be simply:

curl -XGET 'localhost:9200'  # where localhost is the ip or hostname of the
target ES node.

the output looks like:
{
   "name" : "node_name",
   "cluster_name" : "cluster_name",
   "cluster_uuid" : "dhB2-TUeRN2LyWmhhN1WJQ",
   "version" : {
 "number" : "5.0.2",
 "build_hash" : "f6b4951",
 "build_date" : "2016-11-24T10:07:18.101Z",
 "build_snapshot" : false,
 "lucene_version" : "6.2.1"
   },
   "tagline" : "You Know, for Search"
}

Too filter out the version number use:
curl -XGET 'localhost:9200/?filter_path=version.number'

output:
{
   "version" : {
 "number" : "5.0.2"
   }
}


Another way to possible check would be to use the node api.
curl -XGET
'localhost:9200/_nodes/_local/info?filter_path=nodes.*.version'

output:
{
   "nodes" : {
 "F6UMqjbmQCO5-IqFeKzfDQ" : {
   "version" : "5.0.2"
 }
   }
}


Note that "_local" equates to the node the request was sent to. Though there
are other ways to lookup the nodes based on other attributes this seems
easiest. Since its restricted to local we wont need to know the node id and
can just use a wildcard. Note that the "" at the end is to make the
output readable and is not required.

Thanks

~Regards





-
~Regards

Matthew Gaetano
--
View this message in context: 
http://rsyslog-users.1305293.n2.nabble.com/omelasticsearch-index-warning-tp7591961p7592008.html
Sent from the rsyslog-users mailing list archive at Nabble.com.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] omelasticsearch index warning

[mostolog--- via rsyslog]

ping!


El 15/12/16 a las 12:37, mosto...@gmail.com escribió:

Hi

Indexing on elasticsearch 5.1.1 we're getting:

[2016-12-15T12:35:27,844][WARN ][o.e.d.r.a.a.i.RestTypesExistsAction] 
[HEAD /{index}/{type}] is deprecated! Use [HEAD /{index}/_mapping/{type}]


Is that a omelasticsearch issue or are we missing something?



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] saving liblognorm literals

[mostolog--- via rsyslog]

Hi


(Moving from grok to liblognorm) We are trying to store a literal as a 
variable, as we were doing using grok. eg: (?ACCEPT)

We aren't sure if this is correct:

   rule=:%{"type":"literal", "text":"a", "name":"var"}%

As stated in 
http://www.liblognorm.com/files/manual/configuration.html#full-json-format


   /*the literal text shall not be stored inside an output variable*;
   for this reason no name attribute is given (we could also have used
   "name":"-" which achives the same effect but is more verbose)./

Does this means we shouldn't store the literal as variable? What If we 
need to?


   /*using the “literal” parser in JSON should be avoided currently*;
   the experimental version does have some rough edges where conflicts
   in literal processing will not be properly handled. This should not
   be an issue in “closed environments”, like “repeat”, where no such
   conflict can occur./

Is that still recommended? How could I "parse and store" a literal using 
condensed format?


Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] About reloading configuration

[mostolog--- via rsyslog]

You may also find interesting to be able to reload liblognorm rules, 
which might be an smaller change.


The question here is the needed amount of work.


El 21/12/16 a las 10:29, Rainer Gerhards escribió:

Possible of course, but considerable work (gut feeling: 6+ strong weeks).

Rainer

Sent from phone, thus brief.

Am 21.12.2016 09:42 schrieb "Benoit DOLEZ" :


Hi

I saw with previous posts that it seems to be complicated to reload
configuration; this is well explained here :
http://www.rsyslog.com/doc/v8-stable/compatibility/v4compati
bility.html#hup-processing.

I wonder if the architecture of rulesets/queues could be flexible enough
to implement a reload operation limited to rulesets. Do you think it is
possible to create new queues/rulesets/actions from a new config and mark
the old ones for removal once idle/empty ?

Regards

Benoit
--
Benoit DOLEZ, POM Monitoring, http://www.pom-monitoring.com/
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Assert Failure in imtcp.c?

[mostolog--- via rsyslog]

It will be great if you provide a replication use case:

 * rsyslog version (may be 8.23)
 * configuration file
 * log example to replicate the issue.

If you are able to replicate the issue with a "not too old" syslog 
version, please file an issue on github



El 20/12/16 a las 21:14, Andrew Griffin via rsyslog escribió:

I’m working setting up a syslog pipeline with rsyslog at the front, and I’m 
running in to an issue with rsyslogd core dumping when I try to start it.  
Debug log shows everything loading fine, then this:

4473.240773513:imtcp.c: nspoll.c:147 ISOBJ assert failure: invalid 
object type, expected 'netstrms' actual 'nspoll', cookie: BADEFEE
rsyslogd: nspoll.c:147: SetDrvrName: Assertion `0' failed.

Then it core dumps.  Is this a known issue?  I can provide a full debug log if 
necessary

Andrew Griffin
  ETS / Integration Services
☏ 408-783-8348



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] omfile.rst documentation

[mostolog--- via rsyslog]

Hi


Reviewing omfile documentation 
https://github.com/rsyslog/rsyslog-doc/blob/master/source/configuration/modules/omfile.rst


It states:

-  **$F$OMFileForceCHOwn** equivalent to the "ForceChOwn" parameter
-  **$ActionFileEnableSync** equivalent to the "enableSync" parameter

There's a typo in /$F$OMFileForceCHOwn/ ?

I'm not able to find /forceChOwn/ or /enableSync/ parameters.

Thanks.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm vs grok

[mostolog--- via rsyslog]

Just created https://github.com/rsyslog/liblognorm/issues/236


El 20/12/16 a las 11:58, mosto...@gmail.com escribió:

El 20/12/16 a las 11:55, Rainer Gerhards escribió:

2016-12-20 11:54 GMT+01:00 mostolog--- via rsyslog<rsyslog@lists.adiscon.com>:

Must first line be...

"version=2" (v lowercase)

this, 
seehttp://www.liblognorm.com/files/manual/configuration.html#rulebase-versions

Already did, but it's still failing, that's why I'm asking

version=2

rule=:%[
{"type":"alternative","parser":[
{"type":"literal", "text":"a"}
]},
{"type":"literal", "text":"a"}
]%

echo "a" | /usr/lib/lognorm/lognormalizer -r
/etc/rsyslog.d/apps/rb/_a.rb
liblognorm error: rulebase file /etc/rsyslog.d/apps/rb/_a.rb[8]:
invalid record type detected: ']%'
{ "originalmsg": "a", "unparsed-data": "a" }




Rainer


or

"Version=2" (V uppercase)

?

El 14/12/16 a las 10:44,mosto...@gmail.com  escribió:


El 07/12/16 a las 21:00, Rainer Gerhards escribió:

I'm getting /invalid field type 'alternative'/ when using it. Any ideas?

rule=test:%[
{"type":"alternative","parser":[
{"type":"literal","text":"-"},
{"type":"word","name":"identd"}
 ]}
]%

no idea
Did you Set Version=2 in the First line?

Yes.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.




___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm segfault ?

[mostolog--- via rsyslog]

ping?



Hi

Having more problems with liblognorm. Let me now if I should open an 
issue.


echo "a" | /usr/lib/lognorm/lognormalizer -r a.rb

Segmentation fault (core dumped)

File:

version=2

#foo

type=@rfc3164pri:<%priority:number%>
type=@rfc3164header:%date:date-rfc3164% %hostname:word%
type=@rfc3164tag:%syslogtag:char-to{"extradata":":"}%:
type=@rfc3164:%.:@rfc3164pri%%.:@rfc3164header% %.:@rfc3164tag%
type=@rfc3164:%.:@rfc3164header% %.:@rfc3164tag%
type=@syslog:%.:@rfc3164%

#bar
#
#it complains liblognorm error: rulebase file a.rb[23]: invalid
record type detected: ']%'
#if written this way:
#  {"type":"rest","name":"message"}
#]%
# Theres a blank line at the end of file
rule=:%[
  {"type":"@syslog","name":"a"},
  {"type":"rest","name":"message"}]%

Regards



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] liblognorm vs grok

[mostolog--- via rsyslog]

Must first line be...

"version=2" (v lowercase)

or

"Version=2" (V uppercase)

?

El 14/12/16 a las 10:44, mosto...@gmail.com escribió:

El 07/12/16 a las 21:00, Rainer Gerhards escribió:


I'm getting /invalid field type 'alternative'/ when using it. Any 
ideas?


   rule=test:%[
   {"type":"alternative","parser":[
   {"type":"literal","text":"-"},
   {"type":"word","name":"identd"}
]}
   ]%

no idea
Did you Set Version=2 in the First line?

Yes.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mostolog--- via rsyslog]

This is exactly why we have $. variables as well as $! variables. They 
work exactly the same, but by convention, $! variables are where you 
put things that you are going to want to send elsewhere, and $. 
variables are where you put things that you need to create for your 
internal logic, templates, etc but don't want to send to the 
destinatino as part of your log content


if you get something that you don't want to send, you can unset $!foo; 
to remove it from the $! set of data.

I didn't know that (if ever read, I forgot).
I'll document that on filters.rst
:P

Still, I'm having some issues with @timestamp. I'll let you know if we 
found any problem.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] liblognorm segfault ?

[mostolog--- via rsyslog]

Hi

Having more problems with liblognorm. Let me now if I should open an issue.

   echo "a" | /usr/lib/lognorm/lognormalizer -r a.rb

   Segmentation fault (core dumped)

File:

   version=2

   #foo

   type=@rfc3164pri:<%priority:number%>
   type=@rfc3164header:%date:date-rfc3164% %hostname:word%
   type=@rfc3164tag:%syslogtag:char-to{"extradata":":"}%:
   type=@rfc3164:%.:@rfc3164pri%%.:@rfc3164header% %.:@rfc3164tag%
   type=@rfc3164:%.:@rfc3164header% %.:@rfc3164tag%
   type=@syslog:%.:@rfc3164%

   #bar
   #
   #it complains liblognorm error: rulebase file a.rb[23]: invalid
   record type detected: ']%'
   #if written this way:
   #  {"type":"rest","name":"message"}
   #]%
   # Theres a blank line at the end of file
   rule=:%[
  {"type":"@syslog","name":"a"},
  {"type":"rest","name":"message"}]%

Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mostolog--- via rsyslog]

Solved using json template (code blindness).

Is there any way to set fields and use them (like @timestamp) but not 
indexing them on elastic? (hidden fields)


Just tried with @timestamp, but it's being indexed :(



El 15/12/16 a las 12:32, mosto...@gmail.com escribió:

Hi


At this moment we are frowarding RELP messages to Elasticsearch using 
omelasticsearch plugin, but sadly message appears as json instead of 
storing each properties. eg: message is { "app": "app1"... instead of 
indexed document having a app property.


Should we specify an especial param on rsyslog, a setting on elastic...?

Regards


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] omelasticsearch index warning

[mostolog--- via rsyslog]

Hi

Indexing on elasticsearch 5.1.1 we're getting:

[2016-12-15T12:35:27,844][WARN ][o.e.d.r.a.a.i.RestTypesExistsAction] 
[HEAD /{index}/{type}] is deprecated! Use [HEAD /{index}/_mapping/{type}]


Is that a omelasticsearch issue or are we missing something?

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mostolog--- via rsyslog]

Hi


At this moment we are frowarding RELP messages to Elasticsearch using 
omelasticsearch plugin, but sadly message appears as json instead of 
storing each properties. eg: message is { "app": "app1"... instead of 
indexed document having a app property.


Should we specify an especial param on rsyslog, a setting on elastic...?

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] global queue configuration?

[mostolog--- via rsyslog]

Probably, carrying multiple conversations at the same time didn't help 
clarifying our concerns. My fault on that. Really sorry.



We know 0% loss is "impossible", but starting from there we'll like to 
have the /best/ system possible.


Considering we found some issues with RELP+full queues we still have to 
investigate, we thought it may worth using a file as an intermediate 
step. Is that such a BAD idea?



> yes, but to fill the queue, you need to fill the disk.

For reliability, we are using linkedlist+filename, and also setting a 
maxdiskspace, so that isn't valid for us.


Rainer suggested having an eye on "ultra-reliable message delivery", and 
seems this conversation has raised a few times...more reasons to add 
"document this" to my TODO list.


Regards

PS: As already said by David, teaching is the best way to learn. I'll 
try to worth the effort!

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog-doc: dot graphs in doc

[mostolog--- via rsyslog]

To solve this hopefully once and forever, I googled again for "sphinx
dot" and the first result seems to be fully appropriate:

http://www.sphinx-doc.org/en/1.4.8/ext/graphviz.html

As I said two weeks or so ago, we need to check if we have
sufficiently current sphinx on the involved systems, but the first
step is to read and use what is describe, to see if it actually fits
the need.
Yes, but my point is that doesn't properly render on github. I'll love 
to have documentation (with graphs) on github, but I'll have to stick to 
sphinx+rsyslog.com if that doesn't work. That's why I tried gravizzo (or 
whatever the name is)



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic

[mostolog--- via rsyslog]

   set $!data=$msg;
you are getting confused over the difference between a string that 
looks like json and an actual json structure.

Sorry, I was.


action(type="mmnormalize" path="data")

this would populate $!data with the structure parsed out of $msg.

mmjsonparse puts the parsed data at $! (not configurable), and there 
are currently bugs in using $! in a set statement, so you would need 
to change your config to work with $! instead of $!data if you use 
mmjsonparse to parse the message.

Actually using set $!foo="foo"; solved the issue.

May I know what bugs could cause problems if we do:

   action(type="mmjsonparse"...)
   set $!foo="bar";


Another questions:

When using RELP, we have noticed errors could be propagated to origin. 
If queue is full and not discarding any messages, it may cause troubles 
on origin rsyslog. According to some list comments, using an 
intermediate file could solve the issue. So:


   imrelp->mmjson+mmnormalize+changes->imfile->omelasticsearch


We are using mmjson upon reception, and because elastic is expecting 
json, we are ALSO using mmjson after imfile read.


Is there any way to prevent this redundancy?(AKA: parsing to json twice)

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] global queue configuration?

[mostolog--- via rsyslog]

On Wed, 14 Dec 2016, David Lang wrote:

this will slow rsyslog down by a factor of >1000x. I ran tests on 
this using a very high end PCI based SSD and had a 2000x - 8000x 
slowdown, and this was with v5, the added improvements in performance 
since wukk cause an even bigger slowdown as they would reduce the 
amount of time rsyslog spends processing the message outside of the 
queue manipulation, I would expect the slowdown to be significantly 
above 10,000x nowdays.

you also have to set batch size = 1

The problem is that in a crash you can't count on anything in memory 
making it to disk, so you have to write each message to disk, do a 
fsync on the data, possibly do a fsync on the directory [1], and only 
then respond with the ack


doing anything else exposes you to a window where the message could be 
lost.


But the performance of filesystems is such that doing a fsync for 
every write to the disk is crippling, even with SSDs [2]. You also 
need to make sure that the SSDs don't buffer the write at all (many 
do), and that you are writing to a RAID set (so that the failure of s 
single SSD doesn't loose the data), which then means you need to make 
sure your RAID controller either doesn't do any buffering, or has a 
battery backed cache.


If you are willing to loose messages in the case of a complete system 
crash/power outage/hardware failure, then things get much simpler, you 
don't need to use a disk type queue, just use a disk assisted queue 
(type FixedArray or LinkedList with a filename) and saveonshutdown and 
you get very good performance (except for the known performace issues 
with retrieving data from a disk queue)

LinkedList and saveonshutdown will be.

About the "creating temporary queues" if ruleset doesn't have a queue 
defined, I'll wait rgerhards for a deep explanation, full of examples, 
colors and music all around.


would imrelp->file->mmjson+mmnorm->elastic be a better approach?


[1] it's actually even worse than this with filesystems newer than ext2

you write the data to the file and do a fsync on the data.
if this changes the size of the file in disk blocks, you then need to 
do a fsync on the directory the file is in so that the size change is 
written
each of these writes are written to the filesystem journal as a 
temporary thing, and then the filesystem needs to write a copy of the 
data to the final location and (once it knows the data is safe there, 
possibly involving another fsync for both the data and metadata), do 
another write to the journal to know that the data was safely saved in 
the final location


so processing one log message can require 6 writes, all of which can 
require a fsync


[2] without SSDs you are limited to simple physics of a write requireing the disk to rotate once, and 
since you aren't infinantly fast, you end up a fair bit below that.


ext3 had a bug where a fsync required writing all pending data to the 
disk, including data that was written after the fsync started. People 
documented a single fsync taking >30 seconds.
These are enough reasons to definitively totally absolutely use fsync as 
much as we can

:P

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Back to work with relp -> mmjsonparse -> mmnormalize -> file -> elastic

[mostolog--- via rsyslog]

Here you go. This is what we have so far.

   global(
MaxMessageSize="32k"
workDirectory="/data"
parser.escapeControlCharactersOnReceive="off"
   )

   module(load="imrelp")
   input(
port="20514"
type="imrelp"
name="imrelp"
ruleset="json"
   )

   module(load="builtin:omfile")
   ruleset(name="error"){
action(
type="omfile"
file="/data/rsyslog-errors.log"
)
   }
   ruleset(name="unknown"){
action(
type="omfile"
file="/data/rsyslog-unknown.log"
)
   }

   template(name="ts" type="string" string="%timestamp:::date-rfc3339%")
   ruleset(name="to-index"){
set $!data=$msg;
set $!data!host_forwarded=$hostname;
set $!data!time_processed=exec_template("ts");
#FIXME This line fails. isn't myhostname set?
#set $!data!host_received=$myhostname;
action(
action.reportSuspension="on"
action.resumeRetryCount="-1"
type="omfile"
file="/data/to-index.log"
template="json"
)
   }

   module(load="mmjsonparse")
   module(load="mmnormalize")
   ruleset(name="json"){
#FIXME seems ruleset workers need a queue or they create a temp
   queue (performance impact)
# considering this pipeline: relp->file->elastic, what should
   be the best approach?
queue.filename="relp.qi"
queue.maxdiskspace="1G"
queue.SaveOnShutdown="on"
queue.type="Disk"

action(
cookie=""
type="mmjsonparse"
)
if $parsesuccess == "FAIL" then {
call error
stop
}
# start script combines /etc/rsyslog.d/apps/*.rb into
   /etc/rsyslog.rb
#   rule=app1:app1 whatever1
#   rule=app2:app2 whatever2
# Due to how liblognorm works, seems to be much faster than
#   each app.conf file like:
#   else if $!app == "popimap" then {
#   # Here's an example on when to use inline rules
#   # https://github.com/rsyslog/rsyslog/issues/625
#   # Inline rules would make it possible to have
#   # just 1 config file per app, instead of 2
#   action(
#   #rule="<%pri%>%time_received:date% %hostname%
   %tag% %msg%"
#   rulebase="/etc/rsyslog.d/apps/app1.rb"
#   type="mmnormalize"
#   )
#   if $!user != "" then {
#   #FIXME now also fails (not set?)
#   set $!data!index="myindex-" & $now;
#   set
   $!data!type="this_msg_type_is_known_by_this_app";
#   call to-index
#   } else {
#   call error
#   }
#   }
   #TODO set $.line= app & " " & msg;?
action(
type="mmnormalize"
variable="$!msg"
rulebase="/etc/rsyslog.d/rsyslog.rb"
)
if $!user == "" then {
call unknown
stop
}
# Each app.conf defines/calls their own pipeline steps
#   at the end: call to-index
$IncludeConfig /etc/rsyslog.d/apps/*.conf
   }

   module(load="imfile")
   input(type="imfile"
file="/data/to-index.log"
tag="rsyslog"
ruleset="elastic"
   )
   template(name="json" type="string" string="%$!data%\n")
   template(name="index" type="string" string="$!data!index")
   template(name="type" type="string" string="$!data!type")
   module(load="omelasticsearch")
   ruleset(name="elastic"){
set $!data=$rawmsg;
set $!data!@timestamp=exec_template("ts");
action(
action.resumeRetryCount="-1"
type="omelasticsearch"
server="server"
serverport="9200"
searchIndex="index"
dynSearchIndex="on"
searchType="type"
dynSearchType="on"
template="json"
)
   }


Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog-doc: dot graphs in doc

[mostolog--- via rsyslog]

Trying to keep the discussion alive and separated from quite 
long-long-long github issue


IMHO it's better to have embedded graphs in doc, rather than separate 
images files, cause it will ease doc maintenance.


Said so, I agree with you using a third-party web provider is not a 
valid choice.


Do we agree on this "dot within rst" approach?


Any help on embedding dot into rst within github would be much 
appreciated. Radu?



El 11/12/16 a las 14:41, David Lang escribió:

On Sun, 11 Dec 2016, Rainer Gerhards wrote:


Hi all,

please have a look at how the graphs are generated in this PR:

https://github.com/rsyslog/rsyslog-doc/pull/281

My gut feeling is that we should not use this method, because IMHO it
is unmaintainable. However, I would like to get feedback from others.
Feel free to comment directly to the github issue tracker.


Going to the Internet is not the right thing to do.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] global queue configuration?

[mostolog--- via rsyslog]

the parameters you listed are all viable.

But listing them in global() would have no effect. You need to specify them
in an action(), ruleset() or main-queue() so they are used there.

Disk queues always use the disk and do not buffer anything in memory, which
means they are ultra-reliable, but slow.
For the best result you should put the queue in front of a reliable
transport action such as relp.

we are using RELP.
according to some comments in 
https://github.com/rsyslog/rsyslog-doc/pull/281, seems rulesets should 
have a queue to avoid "creating a temporary queue"


as we are looking for 0% loss...

1. imrelp -> ruleset+main_queue(disk) is there any chance a message
   being lost between RELP reception and queueing?
2. ruleset+main_queue(disk) -> mmjsonparse -> mmnormalize -> omfile is
   there any chance a message being lost anywhere?
3. imfile -> omelasticsearch is there any change a message being lost?

PS: using RELP, when there are connection issues we noticed some 
problems, bottlenecks and so. hence, we decided to use a file between 
RELP and elastic



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.