Re: [rsyslog] Are we building an ERK stack?

[chenlin rao]

https://github.com/rsyslog/rsyslog/pull/1099

2016-11-25 16:46 GMT+08:00 mosto...@gmail.com :

> Thanks!
>
> It's your mmdblookup opensourced?
>
>
> El 25/11/16 a las 03:46, chenlin rao escribió:
>
> re-upload an english version. The content was a little old though.
>>
>> 2016-11-23 22:39 GMT+08:00 mosto...@gmail.com :
>>
>> http://www.slideshare.net/chenryn/elk-stack-at-weibocom
>>>
>>> I NEED the english version :P
>>>
>>> ___
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] omriemann Re: Are we building an ERK stack?

[Adam Williams]

On Wed, Nov 23, 2016 at 1:32 PM, Bob Gregory  wrote:

> I can easily enough knock together an omriemann - it's protobuf over TCP or
> UDP.  TCP allows for message ack.
>
> There are a couple of C clients that are useful as prior art, and I've
> worked with a bunch of clients in python, haskell and golang.


That would be pretty great!

We have been for a couple of years sending messages to Riemann by having
omprog start up a Ruby script that basically looks like this:

```
  def process_log_entries(io, )
until io.eof?
  process_log_entry(io.gets.chomp, )
end
  end

  require 'riemann/client'
  riemann = Riemann::Client.new(host: 'localhost', port: , timeout: 5)

  process_log_entries($stdin) do |event|
riemann << event
  end
```

```
action(type="omprog"
   binary="/usr/sbin/omriemann"
   template="omriemann-json"
   queue.type="linkedlist"
   queue.size="5"
   queue.dequeuebatchsize="100"
   queue.filename="riemannqueue"
   queue.highwatermark="4"
   queue.lowwatermark="2"
   queue.maxdiskspace="5g"
   queue.saveonshutdown="on")
```

My understanding is that omprog will create a few of these processes if
necessary to keep queues happy. I have certainly seen times when there are
a couple of omriemann.rb processes owned by rsyslog!

- Adam Williams
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

I may be confused about which part is on the sender and which part is 
on the receiver.

sender: a bunch of imfiles forwarded using RELP

receiver: receives a JSON with msg=plain/original message, normalize and 
extract fields (that should be added to JSON)
each "application" should define his own rules, and sometimes even 
transform the JSON after that (seem's that the hard part)







  *# Is addMetadata="on" needed in order to use $!metadata!filename?*



the easiest thing is to try it :-)
I'm going to create an issue for double checking...can't handle this 
while editing documentation!


adding fields you may be able to do with the ammend= capabilities in 
the ruleset


if you don't want a field to be reported, give it the name '-' in the 
ruleset.


unfortunantly, you can't rename fields or copy fields in the ruleset.
So: each application having one .conf file copied to rsyslog.d/ with the 
required steps it's the only way? Perhaps something like:

*app1.conf*

   if $!group == "group" and $!app == "app1" then {
# and here's an example on when to use inline rules
# https://github.com/rsyslog/rsyslog/issues/625
# an inline rule here will make it possible to have
# just 1 config file per app, instead of 2
action(type="mmnormalize" rulebase=:/rule-for-app1.fb")
if $parsesuccess then {
# do additional steps, transforms and whatever you want
# call foo
}
call index
stop
   }


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Fri, 25 Nov 2016, mosto...@gmail.com wrote:

string="<%pri%>%timestamp:::date-rfc3339% %hostname% 
logs/$!data!group/$!data!app: %$!data%") to be fully correct

Done.

also watch out, the programname is limited to 32 characters, don't let your 
group and app names get too long.
Wasn't it possible to change that? IIRC we had some issues with 
hostnames/tags being too long and were able to handle longer.

Anyway, we'll try to stay within boundaries.

since you changed the programname to be logs/group/app this would be field 
3

Are you sure?
*ruleset apps* is invoked for each input using TAG=group/app, and AFTER that 
ruleset relp uses template json, which prefix "logs/"


I may be confused about which part is on the sender and which part is on the 
receiver.





  *# Is addMetadata="on" needed in order to use $!metadata!filename?*

I think so.
As I'm double checking everything while updating docs, I would love to have a 
more confident statement on this. Rainer?


the easiest thing is to try it :-)

correct, although mmjsonparse defaults to needing @cee: in front of the 
json, so the line below needs to be changed to:


module(load="mmjsonparse" cookie="")
I forgot! Nice catch (...I'll have to check if cookie goes in module or 
action...)


it should be action, sorry


  *# Once all operations have ended, it should be indexed**
  **# Is there any way apps not only define rules, but aditional
  transformations?**
  **# I guess having a .conf file with if+ruleset could work...*
no, the mmnormalize ruleset cannot apply any transformations. I would 
probably try to do that on the sending side if I could.

That's why I played with rulesets to make something like

  a.conf
 normalize
 add some fields
  b.conf
 normalize
  c.conf
 normalize
 remove some fields

That would make the combination script behave differently


adding fields you may be able to do with the ammend= capabilities in the ruleset

if you don't want a field to be reported, give it the name '-' in the ruleset.

unfortunantly, you can't rename fields or copy fields in the ruleset.




  *# It is possible to use $!index here? Workaround?**

this is what dynsearchindex and dynparent are for.
so: dynSearchIndex="on" searchIndex="mytemplate" and template="$!index", 
right?


yep.




  **# How could EACH app specify his own index pattern?**
they can't directly, but the template can be "%$.manual%" and you use 
rainerscript commands to set $.manual to whatever you want it to me 
(another good use for a lookup table if it's complex enough ;-)

I didn't understand this...but having each app.conf file could also work.


you can do

set ...
set ...
action()

but not

action(set)

set is a statement, action is a statement. Action takes parameters, but not 
statements inside the ()


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

string="<%pri%>%timestamp:::date-rfc3339% %hostname% 
logs/$!data!group/$!data!app: %$!data%") to be fully correct

Done.

also watch out, the programname is limited to 32 characters, don't let 
your group and app names get too long.
Wasn't it possible to change that? IIRC we had some issues with 
hostnames/tags being too long and were able to handle longer.

Anyway, we'll try to stay within boundaries.

since you changed the programname to be logs/group/app this would be 
field 3

Are you sure?
*ruleset apps* is invoked for each input using TAG=group/app, and AFTER 
that ruleset relp uses template json, which prefix "logs/"





  *# Is addMetadata="on" needed in order to use $!metadata!filename?*

I think so.
As I'm double checking everything while updating docs, I would love to 
have a more confident statement on this. Rainer?


it's probably a good idea to put stop here to make it clear that you 
don't intend for there to be any other processing of the log message. 
With the input bound to a ruleset, I don't think it makes a 
difference, but better to be explicit.

Done

correct, although mmjsonparse defaults to needing @cee: in front of 
the json, so the line below needs to be changed to:


module(load="mmjsonparse" cookie="")
I forgot! Nice catch (...I'll have to check if cookie goes in module or 
action...)


yes, the script can either populate the rules file with includes, or 
just combine them into one file (probably faster at startup to have 
them combined, but it may not be measureable)

I'll combine them (if able)




  *# Once all operations have ended, it should be indexed**
  **# Is there any way apps not only define rules, but aditional
  transformations?**
  **# I guess having a .conf file with if+ruleset could work...*
no, the mmnormalize ruleset cannot apply any transformations. I would 
probably try to do that on the sending side if I could.

That's why I played with rulesets to make something like

   a.conf
  normalize
  add some fields
   b.conf
  normalize
   c.conf
  normalize
  remove some fields

That would make the combination script behave differently




  *# It is possible to use $!index here? Workaround?**

this is what dynsearchindex and dynparent are for.
so: dynSearchIndex="on" searchIndex="mytemplate" and template="$!index", 
right?





  **# How could EACH app specify his own index pattern?**
they can't directly, but the template can be "%$.manual%" and you use 
rainerscript commands to set $.manual to whatever you want it to me 
(another good use for a lookup table if it's complex enough ;-)

I didn't understand this...but having each app.conf file could also work.

The background idea is to combine this with "dynamic configuration 
reload" to be able to change "an application pipeline"


This is looking pretty good now.

Thanks. I think the same!

Seems next pending issue is to solve "each application can do different 
things, like adding hiw own index pattern or additional 
transforms/steps" in his config file.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Fri, 25 Nov 2016, mosto...@gmail.com wrote:


Date: Fri, 25 Nov 2016 11:22:08 +0100
From: "mosto...@gmail.com" 
Reply-To: rsyslog-users 
To: rsyslog-users 
Subject: Re: [rsyslog] mmnormalize with mutiple input: conditionals?

What about...?

*remote.conf*(Please, notice there are commented questions)

  global(
  MaxMessageSize="32k"
   )

  template(name="json" type="string"
  string="%timestamp:::date-rfc3339% %hostname%
  logs/$!data!group/$!data!app %$!data%")


change this to:

string="<%pri%>%timestamp:::date-rfc3339% %hostname% logs/$!data!group/$!data!app: 
%$!data%")

to be fully correct

also watch out, the programname is limited to 32 characters, don't let your 
group and app names get too long.



  module(load="omrelp")
  ruleset(name="relp"){
  action(
  port="20514"
  *# It is possible to use $!server here? Workaround?*
  target="server"
  template="json"
  type="omrelp"
   )
   }

  ruleset(name="apps") {
   set $!data!app=field($programname,"/",2);


since you changed the programname to be logs/group/app this would be field 3


  *# Is addMetadata="on" needed in order to use $!metadata!filename?*


I think so.


   set $!data!file="$!metadata!filename";
   set $!data!group=field($programname,"/",1);


as per above, field 2


   set $!data!msg=$msg;
  call relp


it's probably a good idea to put stop here to make it clear that you don't 
intend for there to be any other processing of the log message. With the input 
bound to a ruleset, I don't think it makes a difference, but better to be 
explicit.



   }

  input(type="imfile" file="/logs/apps/app1/app1.log"
  tag="mygroup/myapp1" addMetadata="on" ruleset="apps"
  startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
  readTimeout="5" PersistStateInterval="1")

   ...

  input(type="imfile" file="/logs/apps/anotherapp/file.log"
  tag="anothergroup/anotherapp" addMetadata="on" ruleset="apps"
  readTimeout="5" PersistStateInterval="1")


*rsyslog.conf*(Please, notice there are commented questions)

  global(
  MaxMessageSize="32k"
  parser.escapeControlCharactersOnReceive="off"
   )

  *# Message is parsed as json on receive, to be able to use
  $!whatever field, right?*


correct, although mmjsonparse defaults to needing @cee: in front of the json, so 
the line below needs to be changed to:


module(load="mmjsonparse" cookie="")


  module(load="mmjsonparse")
  ruleset(name="json"){
  action(
  type="mmjsonparse"
   )
   }
  module(load="imrelp")
   input(
  name="imrelp"
  port="20514"
  type="imrelp"
  ruleset="json"
   )

   set $.line = $!group $!app + " " + $!msg;
   action(
  type="mmnormalize"
  variable="$.line"
  *# As I don't know the list of apps, **
  **# the only way to combine all rules is an script**
  **# isnt it?*


yes, the script can either populate the rules file with includes, or just 
combine them into one file (probably faster at startup to have them combined, 
but it may not be measureable)



  rulebase=:/path/to/combined/rules.fb"
   )

  *# IIUC, messages will be procesed by above rule**
  **# AFTER that, they will be processed by the following, right?*


yes


   if message contains "ip" field then {
   # TODO lookup_table
   }

  *# Once all operations have ended, it should be indexed**
  **# Is there any way apps not only define rules, but aditional
  transformations?**
  **# I guess having a .conf file with if+ruleset could work...*


no, the mmnormalize ruleset cannot apply any transformations. I would probably 
try to do that on the sending side if I could.



  template(name="json" type="string" string="%$!%")
  module(load="omelasticsearch")
   action(
  template="json"
  type="omelasticsearch"
  *# It is possible to use $!index here? Workaround?**


this is what dynsearchindex and dynparent are for.


  **# How could EACH app specify his own index pattern?**


they can't directly, but the template can be "%$.manual%" and you use 
rainerscript commands to set $.manual to whatever you want it to me (another 
good use for a lookup table if it's complex enough ;-)



  **#   set $!index="$!app2_$$year-$$month-$$day"**
  **#   set $!index="$!app2_$$year-$$month-$$day-$$hour"*


set statements cant' be inside an action() statement, they would go before it.

also, set statements end in ';'



  searchIndex="$!index"
   )


The background idea is to combine this with "dynamic configuration reload" to 
be able to change "an application pipeline"


This is looking pretty good now.


Thanks a lot for your help. I'll contribute as much as I can in exchange ;)


that's how I got into this.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow 

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

What about...?

*remote.conf*(Please, notice there are commented questions)

   global(
   MaxMessageSize="32k"
)

   template(name="json" type="string"
   string="%timestamp:::date-rfc3339% %hostname%
   logs/$!data!group/$!data!app %$!data%")

   module(load="omrelp")
   ruleset(name="relp"){
   action(
   port="20514"
   *# It is possible to use $!server here? Workaround?*
   target="server"
   template="json"
   type="omrelp"
)
}

   ruleset(name="apps") {
set $!data!app=field($programname,"/",2);
   *# Is addMetadata="on" needed in order to use $!metadata!filename?*
set $!data!file="$!metadata!filename";
set $!data!group=field($programname,"/",1);
set $!data!msg=$msg;
   call relp
}

   input(type="imfile" file="/logs/apps/app1/app1.log"
   tag="mygroup/myapp1" addMetadata="on" ruleset="apps"
   startmsg.regex="^[[:digit:]]{2} [[:alpha:]]{3} [[:digit:]]{4}"
   readTimeout="5" PersistStateInterval="1")

...

   input(type="imfile" file="/logs/apps/anotherapp/file.log"
   tag="anothergroup/anotherapp" addMetadata="on" ruleset="apps"
   readTimeout="5" PersistStateInterval="1")


*rsyslog.conf*(Please, notice there are commented questions)

   global(
   MaxMessageSize="32k"
   parser.escapeControlCharactersOnReceive="off"
)

   *# Message is parsed as json on receive, to be able to use
   $!whatever field, right?*
   module(load="mmjsonparse")
   ruleset(name="json"){
   action(
   type="mmjsonparse"
)
}
   module(load="imrelp")
input(
   name="imrelp"
   port="20514"
   type="imrelp"
   ruleset="json"
)

set $.line = $!group $!app + " " + $!msg;
action(
   type="mmnormalize"
   variable="$.line"
   *# As I don't know the list of apps, **
   **# the only way to combine all rules is an script**
   **# isnt it?*
   rulebase=:/path/to/combined/rules.fb"
)

   *# IIUC, messages will be procesed by above rule**
   **# AFTER that, they will be processed by the following, right?*
if message contains "ip" field then {
# TODO lookup_table
}

   *# Once all operations have ended, it should be indexed**
   **# Is there any way apps not only define rules, but aditional
   transformations?**
   **# I guess having a .conf file with if+ruleset could work...*

   template(name="json" type="string" string="%$!%")
   module(load="omelasticsearch")
action(
   template="json"
   type="omelasticsearch"
   *# It is possible to use $!index here? Workaround?**
   **# How could EACH app specify his own index pattern?**
   **#   set $!index="$!app2_$$year-$$month-$$day"**
   **#   set $!index="$!app2_$$year-$$month-$$day-$$hour"*
   searchIndex="$!index"
)


The background idea is to combine this with "dynamic configuration 
reload" to be able to change "an application pipeline"

Thanks a lot for your help. I'll contribute as much as I can in exchange ;)


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[David Lang]

I will point out that no matter what software you run, you will eventually run 
into a case like this where the distro packages break for you, and you are faced 
with the need to run non-standard packages to work around a bug.


As Rainer points out, if you aren't willing to compile your own version, even if 
he were to find that the bug was not fixed in the latest version, and create a 
fix for it, you still wouldn't be able to use the fix.


David Lang


On Fri, 25 Nov 2016, Kosta Psimoulis wrote:


Thank you for much guys for your support, right now I have a much better
picture of what is going on. I am aware how to build from source but this
would be something difficult to maintain, I would probably need to create a
custom repo. I think I have enough information right now to reconsider and
evaluate a business decision whether to use rsyslog or look for another
solution. I have some queue files that I can recover and rebuild with the
information you have given but I am still not sure of how they got
corrupted and what happened to the ones that were in memory, was everything
saved on the hard drive or was there information that was lost.

Kind Regards,
Kosta

On Fri, Nov 25, 2016 at 3:42 AM, David Lang  wrote:


On Fri, 25 Nov 2016, mosto...@gmail.com wrote:

TBH, it depends if you prefer building from source or instability

Kidding away, I know there are problems in older versions, they are


Isn't adiscon repo valid?



It looks like we don't have a repo for Jessie, at least it's not included
in the scripts/config.sh for rsyslog-pkg-debian

This probably also means we don't have a Travis test box running Jessie.

Packages for Wheezy will probably work (the only thing I can think of that
would be likely to break is gnutls related dependencies)

David Lang

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Fri, 25 Nov 2016, David Lang wrote:


On Fri, 25 Nov 2016, mosto...@gmail.com wrote:

After sleeping on it, I'm still thinking about "defining a separate 
pipeline for each application". To sum up, each application could do his 
own thing and return the message to the queue, in order to be processed by 
other modules...until it's done, and indexed into ES
Is there a way to put the already processed message into the input queue to 
be processed again by another *sibling* modules?


  input-> queue -> app
  <-
  -> geoip
  <-
  -> index


that's exactly what a message modification module (mm*) does, it changes the 
messages in the queue so that things after them in the config see the 
modified version.


you can also simulate it by having separate rulesets all call one common 
ruleset, but that's really not merging the work back into one flow, it's just 
executing the same code in many different flows.


Also, rsyslog really isn't structured to support per-application pipelines the 
way you are thinking of them. While you could simulate them, they are a bad fit 
for how rsyslog actually works, and as a result, your performance and resource 
usage will be substantially sub-optimal.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[David Lang]

On Fri, 25 Nov 2016, mosto...@gmail.com wrote:

After sleeping on it, I'm still thinking about "defining a separate pipeline 
for each application". To sum up, each application could do his own thing and 
return the message to the queue, in order to be processed by other 
modules...until it's done, and indexed into ES
Is there a way to put the already processed message into the input queue to 
be processed again by another *sibling* modules?


  input-> queue -> app
  <-
  -> geoip
  <-
  -> index


that's exactly what a message modification module (mm*) does, it changes the 
messages in the queue so that things after them in the config see the modified 
version.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[mosto...@gmail.com]

Agains't Rainer's advice, we are using adiscon repos and we're quite 
happy with them...



El 25/11/16 a las 09:59, Kosta Psimoulis escribió:

Thank you for much guys for your support, right now I have a much better
picture of what is going on. I am aware how to build from source but this
would be something difficult to maintain, I would probably need to create a
custom repo. I think I have enough information right now to reconsider and
evaluate a business decision whether to use rsyslog or look for another
solution. I have some queue files that I can recover and rebuild with the
information you have given but I am still not sure of how they got
corrupted and what happened to the ones that were in memory, was everything
saved on the hard drive or was there information that was lost.

Kind Regards,
Kosta

On Fri, Nov 25, 2016 at 3:42 AM, David Lang  wrote:


On Fri, 25 Nov 2016, mosto...@gmail.com wrote:

TBH, it depends if you prefer building from source or instability

Kidding away, I know there are problems in older versions, they are


Isn't adiscon repo valid?


It looks like we don't have a repo for Jessie, at least it's not included
in the scripts/config.sh for rsyslog-pkg-debian

This probably also means we don't have a Travis test box running Jessie.

Packages for Wheezy will probably work (the only thing I can think of that
would be likely to break is gnutls related dependencies)

David Lang

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[mosto...@gmail.com]

After sleeping on it, I'm still thinking about "defining a separate 
pipeline for each application". To sum up, each application could do his 
own thing and return the message to the queue, in order to be processed 
by other modules...until it's done, and indexed into ES
Is there a way to put the already processed message into the input queue 
to be processed again by another *sibling* modules?


   input-> queue -> app
   <-
   -> geoip
   <-
   -> index

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[Kosta Psimoulis]

Thank you for much guys for your support, right now I have a much better
picture of what is going on. I am aware how to build from source but this
would be something difficult to maintain, I would probably need to create a
custom repo. I think I have enough information right now to reconsider and
evaluate a business decision whether to use rsyslog or look for another
solution. I have some queue files that I can recover and rebuild with the
information you have given but I am still not sure of how they got
corrupted and what happened to the ones that were in memory, was everything
saved on the hard drive or was there information that was lost.

Kind Regards,
Kosta

On Fri, Nov 25, 2016 at 3:42 AM, David Lang  wrote:

> On Fri, 25 Nov 2016, mosto...@gmail.com wrote:
>
> TBH, it depends if you prefer building from source or instability
>>> Kidding away, I know there are problems in older versions, they are
>>>
>> Isn't adiscon repo valid?
>>
>
> It looks like we don't have a repo for Jessie, at least it's not included
> in the scripts/config.sh for rsyslog-pkg-debian
>
> This probably also means we don't have a Travis test box running Jessie.
>
> Packages for Wheezy will probably work (the only thing I can think of that
> would be likely to break is gnutls related dependencies)
>
> David Lang
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Are we building an ERK stack?

[mosto...@gmail.com]

Thanks!

It's your mmdblookup opensourced?


El 25/11/16 a las 03:46, chenlin rao escribió:

re-upload an english version. The content was a little old though.

2016-11-23 22:39 GMT+08:00 mosto...@gmail.com :


http://www.slideshare.net/chenryn/elk-stack-at-weibocom

I NEED the english version :P

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[David Lang]

On Fri, 25 Nov 2016, mosto...@gmail.com wrote:


TBH, it depends if you prefer building from source or instability
Kidding away, I know there are problems in older versions, they are

Isn't adiscon repo valid?


It looks like we don't have a repo for Jessie, at least it's not included in the 
scripts/config.sh for rsyslog-pkg-debian


This probably also means we don't have a Travis test box running Jessie.

Packages for Wheezy will probably work (the only thing I can think of that would 
be likely to break is gnutls related dependencies)


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[Rainer Gerhards]

2016-11-25 9:38 GMT+01:00 mosto...@gmail.com :
>
>> TBH, it depends if you prefer building from source or instability
>> Kidding away, I know there are problems in older versions, they are
>
> Isn't adiscon repo valid?

We don't have everything. Too few contributors :-(

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[mosto...@gmail.com]

TBH, it depends if you prefer building from source or instability
Kidding away, I know there are problems in older versions, they are

Isn't adiscon repo valid?
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[David Lang]

On Fri, 25 Nov 2016, Kosta Psimoulis wrote:


Hello Rainer,

Thank you for your honest answer, so you are basically saying that the
queue will be lost ? I thought that my current version of rsyslog was
stable enough, I guess I was wrong, in that case I will try a different
queue system. I am sending a little bit extra debugging information, please
let me know if you see anything useful and if you believe that this issue
was fixed in 8.23


we do believe that it is fixed in the current version. Unfortunantly, you aren't 
running the current version. 8.4 is rather old, and even 8.16 is prior to some 
very substantial reliability fixes, including several that caused segfaults.


In the short term, move the queue files to another directory and see if rsyslog 
starts. If so, we can look at trying to fix the queue files (there is a command 
that will rebuild the .qi file from the raw queue files, that may solve your 
issues.


with the exception of encryption, the packages for wheezy should have no 
problems running on Jessie (although I agree that after a year we should be 
providing a Jessie repo, nobody called it to our attention, so we missed it)






Unfortunantly, due to manpower limits, we (the community) can only support the 
most recent release of rsyslog, we can't do backports of fixes (although that is 
an option for people who get paid support). We tried to do that sort of thing 
and the project almost died under the load.


When you rely on the distro provided versions instead of the upstream releases, 
it's on the basis that the distro is providing support for them. If the distro 
isn't providing support for a bug that you are running into, you need to look at 
shifting to the upstream version.


Sometimes this is going to require that you maintain your own version of the 
package, built from source (to apply a fix, or because you need a compile-time 
option that's not in the default packages, or to use a new feature)


The number of fixes since 8.16 (let alone 8.4) is huge, trying to backport them 
all, without backporting the other features added would probably result in a 
less reliable system than just running the latest 8.24 version.


IIRC, 8.16 was the last release before we started using libfastjson, and part of 
the reason why Debian hasn't picked up the newer versions, even into backports, 
is concerns over long-term maintinance of libfastjson. But it fixed so many 
errors that we killing people's systems that we really need to shift.




In theory, you should be able to clone the source git repos, and the 
rsyslog-pkg-debian repo and in the rsyslog-pkg-debian repo under 
scripts/config.sh add Jessie to the distro list and everything should work.


We just haven't tested it yet

(and yes, there is a bit more to getting this going, contact me off-list and 
I'll pass along the build scripts I put together for ubuntu, they should work 
almost the same on Debian)


David Lang



Kind Regards,
Kosta

1388.716086990:action 3 queue[DA]:Reg/w0: in destructor: sendbuf 0xada9030
1388.716159868:action 3 queue[DA]:Reg/w0: relp engine is dispatching frame
with command 'rsp'
1388.716530324:action 3 queue[DA]:Reg/w0: relpSessWaitState returns 10019
1388.718355599:action 3 queue[DA]:Reg/w0: in destructor: sendbuf 0xada9570
1388.719729919:action 3 queue[DA]:Reg/w0: relp engine created new client
0xadae250
1388.719865043:action 3 queue[DA]:Reg/w0: omrelp: endTransaction
==16816== Thread 2 rs:action 3 queue[DA]:R:
==16816== Invalid read of size 8
==16816==at 0x76FEE94: relpCltHintBurstEnd (in
/usr/lib/x86_64-linux-gnu/librelp.so.0.1.0)
==16816==by 0x74EFB7A: ??? (in /usr/lib/rsyslog/omrelp.so)
==16816==by 0x4529ED: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x453517: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x448F8A: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x4457BD: wtiWorker (in /usr/sbin/rsyslogd)
==16816==by 0x4447CB: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x50580A3: start_thread (pthread_create.c:309)
==16816==by 0x5F7A62C: clone (clone.S:111)
==16816==  Address 0x20 is not stack'd, malloc'd or (recently) free'd
==16816==
==16816==
==16816== Process terminating with default action of signal 11 (SIGSEGV)
==16816==  Access not within mapped region at address 0x20
==16816==at 0x76FEE94: relpCltHintBurstEnd (in
/usr/lib/x86_64-linux-gnu/librelp.so.0.1.0)
==16816==by 0x74EFB7A: ??? (in /usr/lib/rsyslog/omrelp.so)
==16816==by 0x4529ED: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x453517: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x448F8A: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x4457BD: wtiWorker (in /usr/sbin/rsyslogd)
==16816==by 0x4447CB: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x50580A3: start_thread (pthread_create.c:309)
==16816==by 0x5F7A62C: clone (clone.S:111)
==16816==  If you believe this happened as a result of a stack
==16816==  overflow in your program's main thread (unlikely but
==16816==  possible), you can try to increase the 

Re: [rsyslog] rsyslog fails to start due to high queue

[Rainer Gerhards]

2016-11-25 9:16 GMT+01:00 Kosta Psimoulis :
> Thank you for your honest answer, so you are basically saying that the
> queue will be lost ?

There is this little tool:

https://github.com/rsyslog/rsyslog/blob/master/tools/recover_qi.pl

It works for your version as well and, depending on what is wrong, it
can fix the queue files. Worth a try.

> I thought that my current version of rsyslog was
> stable enough, I guess I was wrong, in that case I will try a different
> queue system.

Well, as always in life: things are deemed stable unless someone finds
it isn't the case. Then those things are fixed and we go back to think
"we are stable" unless... That's a large part of why new releases
happen, and that's part of the frequent release philosophy in open
source.

> I am sending a little bit extra debugging information, please
> let me know if you see anything useful and if you believe that this issue
> was fixed in 8.23

It's indeed interesting, as it may point into a bug that was fixed in
librelp (I barely remember there was something). Unfortunately, I have
made it my policy to NEVER AGAN look at old versions for fixing bugs.
I have done so in the past, and wasted month of my life, because in
95% of the cases it turned out I hunt for a bug that was long fixed.
So I only do this if I am actually paid for the pain, via an Adiscon
support contract.

I am NOT trying to get money out of you, and I understand that you do
not like to upgrade or are even unable to do so. But again, time is
very limited, I have a large TODO list and year-long experience has
prooven that hunting bugs in old code bases is fruitless. Even more so
if after hours I discover it's already fixed AND you need to apply a
code patch, what you don't want to do/ are not permitted to do in the
first place. Which means the effort was fruitless and wasteful for
everyone involved (yes, this *is* *practical* *experience*, more than
once).

So the first filter in open source development for me is: if the user
(or his org) is unwilling to update to current, there is no point in
looking at something that might even remotely be a bug.

I hope for your understanding.

Rainer

>
> Kind Regards,
> Kosta
>
> 1388.716086990:action 3 queue[DA]:Reg/w0: in destructor: sendbuf 0xada9030
> 1388.716159868:action 3 queue[DA]:Reg/w0: relp engine is dispatching frame
> with command 'rsp'
> 1388.716530324:action 3 queue[DA]:Reg/w0: relpSessWaitState returns 10019
> 1388.718355599:action 3 queue[DA]:Reg/w0: in destructor: sendbuf 0xada9570
> 1388.719729919:action 3 queue[DA]:Reg/w0: relp engine created new client
> 0xadae250
> 1388.719865043:action 3 queue[DA]:Reg/w0: omrelp: endTransaction
> ==16816== Thread 2 rs:action 3 queue[DA]:R:
> ==16816== Invalid read of size 8
> ==16816==at 0x76FEE94: relpCltHintBurstEnd (in
> /usr/lib/x86_64-linux-gnu/librelp.so.0.1.0)
> ==16816==by 0x74EFB7A: ??? (in /usr/lib/rsyslog/omrelp.so)
> ==16816==by 0x4529ED: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x453517: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x448F8A: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x4457BD: wtiWorker (in /usr/sbin/rsyslogd)
> ==16816==by 0x4447CB: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x50580A3: start_thread (pthread_create.c:309)
> ==16816==by 0x5F7A62C: clone (clone.S:111)
> ==16816==  Address 0x20 is not stack'd, malloc'd or (recently) free'd
> ==16816==
> ==16816==
> ==16816== Process terminating with default action of signal 11 (SIGSEGV)
> ==16816==  Access not within mapped region at address 0x20
> ==16816==at 0x76FEE94: relpCltHintBurstEnd (in
> /usr/lib/x86_64-linux-gnu/librelp.so.0.1.0)
> ==16816==by 0x74EFB7A: ??? (in /usr/lib/rsyslog/omrelp.so)
> ==16816==by 0x4529ED: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x453517: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x448F8A: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x4457BD: wtiWorker (in /usr/sbin/rsyslogd)
> ==16816==by 0x4447CB: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x50580A3: start_thread (pthread_create.c:309)
> ==16816==by 0x5F7A62C: clone (clone.S:111)
> ==16816==  If you believe this happened as a result of a stack
> ==16816==  overflow in your program's main thread (unlikely but
> ==16816==  possible), you can try to increase the size of the
> ==16816==  main thread stack using the --main-stacksize= flag.
> ==16816==  The main thread stack size used in this run was 8388608.
> ==16816==
> ==16816== HEAP SUMMARY:
> ==16816== in use at exit: 1,136,880 bytes in 2,685 blocks
> ==16816==   total heap usage: 15,434 allocs, 12,749 frees, 3,325,304 bytes
> allocated
> ==16816==
> ==16816== LEAK SUMMARY:
> ==16816==definitely lost: 0 bytes in 0 blocks
> ==16816==indirectly lost: 0 bytes in 0 blocks
> ==16816==  possibly lost: 1,152 bytes in 4 blocks
> ==16816==still reachable: 1,135,728 bytes in 2,681 blocks
> ==16816== suppressed: 0 bytes in 0 blocks
> 

Re: [rsyslog] Are we building an ERK stack?

[chenlin rao]

No, I don't detect them, just capture them with a special regexp because I
only need to process PHP slowlog. which memory addr appear in the beginning
`\[0x\w+\]`... The use case for slow functions stack without mem addrs can
be found at the slide 25 (pie charts for nested sub terms aggs).

2016-11-25 15:39 GMT+08:00 Rainer Gerhards :

> 2016-11-25 8:26 GMT+01:00 chenlin rao :
> > - rewrite most of mmgrok into mmnormalize+rainerscript. Except PHP
> slowlog
> > only. We want to translate the memory address of each line into "x",
> > but seems can't be done in rsyslog, so a mmexternal here.
>
> focussed question: how exactly do you detect memory address? I ask
> because there is mmanon, which does something similiar to IP
> addresses, and I *think* it could be extended to other objects if only
> we know pricesely what to look for and how to transform it.
>
> Rainer
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[Kosta Psimoulis]

Hello Rainer,

Thank you for your honest answer, so you are basically saying that the
queue will be lost ? I thought that my current version of rsyslog was
stable enough, I guess I was wrong, in that case I will try a different
queue system. I am sending a little bit extra debugging information, please
let me know if you see anything useful and if you believe that this issue
was fixed in 8.23

Kind Regards,
Kosta

1388.716086990:action 3 queue[DA]:Reg/w0: in destructor: sendbuf 0xada9030
1388.716159868:action 3 queue[DA]:Reg/w0: relp engine is dispatching frame
with command 'rsp'
1388.716530324:action 3 queue[DA]:Reg/w0: relpSessWaitState returns 10019
1388.718355599:action 3 queue[DA]:Reg/w0: in destructor: sendbuf 0xada9570
1388.719729919:action 3 queue[DA]:Reg/w0: relp engine created new client
0xadae250
1388.719865043:action 3 queue[DA]:Reg/w0: omrelp: endTransaction
==16816== Thread 2 rs:action 3 queue[DA]:R:
==16816== Invalid read of size 8
==16816==at 0x76FEE94: relpCltHintBurstEnd (in
/usr/lib/x86_64-linux-gnu/librelp.so.0.1.0)
==16816==by 0x74EFB7A: ??? (in /usr/lib/rsyslog/omrelp.so)
==16816==by 0x4529ED: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x453517: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x448F8A: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x4457BD: wtiWorker (in /usr/sbin/rsyslogd)
==16816==by 0x4447CB: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x50580A3: start_thread (pthread_create.c:309)
==16816==by 0x5F7A62C: clone (clone.S:111)
==16816==  Address 0x20 is not stack'd, malloc'd or (recently) free'd
==16816==
==16816==
==16816== Process terminating with default action of signal 11 (SIGSEGV)
==16816==  Access not within mapped region at address 0x20
==16816==at 0x76FEE94: relpCltHintBurstEnd (in
/usr/lib/x86_64-linux-gnu/librelp.so.0.1.0)
==16816==by 0x74EFB7A: ??? (in /usr/lib/rsyslog/omrelp.so)
==16816==by 0x4529ED: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x453517: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x448F8A: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x4457BD: wtiWorker (in /usr/sbin/rsyslogd)
==16816==by 0x4447CB: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x50580A3: start_thread (pthread_create.c:309)
==16816==by 0x5F7A62C: clone (clone.S:111)
==16816==  If you believe this happened as a result of a stack
==16816==  overflow in your program's main thread (unlikely but
==16816==  possible), you can try to increase the size of the
==16816==  main thread stack using the --main-stacksize= flag.
==16816==  The main thread stack size used in this run was 8388608.
==16816==
==16816== HEAP SUMMARY:
==16816== in use at exit: 1,136,880 bytes in 2,685 blocks
==16816==   total heap usage: 15,434 allocs, 12,749 frees, 3,325,304 bytes
allocated
==16816==
==16816== LEAK SUMMARY:
==16816==definitely lost: 0 bytes in 0 blocks
==16816==indirectly lost: 0 bytes in 0 blocks
==16816==  possibly lost: 1,152 bytes in 4 blocks
==16816==still reachable: 1,135,728 bytes in 2,681 blocks
==16816== suppressed: 0 bytes in 0 blocks
==16816== Rerun with --leak-check=full to see details of leaked memory
==16816==
==16816== For counts of detected and suppressed errors, rerun with: -v
==16816== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Killed


On Fri, Nov 25, 2016 at 3:03 AM, Rainer Gerhards 
wrote:

> 2016-11-25 8:59 GMT+01:00 Kosta Psimoulis  sourceknowledge.com>:
> > Hello,
> >
> > Perhaps I wasn't clear on the first message but I am getting Segmentation
> > fault.
> >
> > this error was on syslog:
> > Nov 24 20:18:25 ip-172-16-1-152 systemd[1]: rsyslog.service: main process
> > exited, code=killed, status=11/SEGV
> >
> > and when I run it debug mode (-dn), it is trying to process the queue and
> > gives the following error:
> > .
> > 9771.929711621:action 3 queue[DA]:Reg/w0: relp engine is dispatching
> frame
> > with command 'rsp'
> > 9771.929715606:action 3 queue[DA]:Reg/w0: in rsp command handler, txnr
> 201,
> > code 200, text 'OK'
> > 9771.929723831:action 3 queue[DA]:Reg/w0: DEL sess 0x7f21a40023b0 unacked
> > 1, sessState 6
> > 9771.929727757:action 3 queue[DA]:Reg/w0: in destructor: sendbuf
> > 0x7f21a401de00
> > 9771.929733344:action 3 queue[DA]:Reg/w0: relp engine is dispatching
> frame
> > with command 'rsp'
> > 9771.929739980:action 3 queue[DA]:Reg/w0: relpSessWaitState returns 10019
> > 9771.929754387:action 3 queue[DA]:Reg/w0: in destructor: sendbuf
> > 0x7f21a4025730
> > 9771.929771799:action 3 queue[DA]:Reg/w0: relp engine created new client
> > 0x7f21a40023b0
> > 9771.929780641:action 3 queue[DA]:Reg/w0: omrelp: endTransaction
> > Segmentation fault
> >
> > 8.4.2 is the latest version available as a package on Debian Jessie 8.6
> and
> > 8.16 if I use the backports. There are no repositories for Debian Jessie,
> > only for Wheezy, are you suggesting to install 8.23 from source on a
> > production server ?
>
> TBH, it depends if you 

Re: [rsyslog] rsyslog fails to start due to high queue

[Rainer Gerhards]

2016-11-25 8:59 GMT+01:00 Kosta Psimoulis :
> Hello,
>
> Perhaps I wasn't clear on the first message but I am getting Segmentation
> fault.
>
> this error was on syslog:
> Nov 24 20:18:25 ip-172-16-1-152 systemd[1]: rsyslog.service: main process
> exited, code=killed, status=11/SEGV
>
> and when I run it debug mode (-dn), it is trying to process the queue and
> gives the following error:
> .
> 9771.929711621:action 3 queue[DA]:Reg/w0: relp engine is dispatching frame
> with command 'rsp'
> 9771.929715606:action 3 queue[DA]:Reg/w0: in rsp command handler, txnr 201,
> code 200, text 'OK'
> 9771.929723831:action 3 queue[DA]:Reg/w0: DEL sess 0x7f21a40023b0 unacked
> 1, sessState 6
> 9771.929727757:action 3 queue[DA]:Reg/w0: in destructor: sendbuf
> 0x7f21a401de00
> 9771.929733344:action 3 queue[DA]:Reg/w0: relp engine is dispatching frame
> with command 'rsp'
> 9771.929739980:action 3 queue[DA]:Reg/w0: relpSessWaitState returns 10019
> 9771.929754387:action 3 queue[DA]:Reg/w0: in destructor: sendbuf
> 0x7f21a4025730
> 9771.929771799:action 3 queue[DA]:Reg/w0: relp engine created new client
> 0x7f21a40023b0
> 9771.929780641:action 3 queue[DA]:Reg/w0: omrelp: endTransaction
> Segmentation fault
>
> 8.4.2 is the latest version available as a package on Debian Jessie 8.6 and
> 8.16 if I use the backports. There are no repositories for Debian Jessie,
> only for Wheezy, are you suggesting to install 8.23 from source on a
> production server ?

TBH, it depends if you prefer building from source or instability ;-)
Kidding away, I know there are problems in older versions, they are
fixed, and so I do not know how *I* can help you without you making
sure you run the fixes. If you don't like to apply them yourself (I
can understand that), I think you should coordinate with the Debian
folks (Michael who packages rsyslog for deb is also on the list) and
ask them for update or to apply the relevant patches.

For the version you have, I think the solution is to remove the queue
files, restart rsyslog and hope for the best. Might work.

Sorry I have no better answer,
Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

[Rainer Gerhards]

2016-11-24 18:18 GMT+01:00 David Lang :
> On Thu, 24 Nov 2016, Rainer Gerhards wrote:
>
>> 2016-11-24 17:21 GMT+01:00 mosto...@gmail.com :

 # I would consider adding a section here to look for parsing failures
 and
 log them to someplace for later investigation, probably in raw format
>>>
>>>
>>> That's mandatory. is there any easy way to catch norm failures?
>>
>>
>> Have a look here:
>>
>> http://www.rsyslog.com/using-mongodb-with-rsyslog-and-loganalyzer/
>>
>> $parsesuccess is your friend... and looking at it, it doesn't seem
>> documented. Would you like to document it or add an rsyslog-doc issue
>> tracker?
>
>
> It was documented in the past, but when I tried to use it with mmnormalize,

Unfortunately the change of doc to RST lost quite some content (that's
why I insist so much this doesn't happen again when restructuring).
Should be documented anyhow.

> it didn't work, I had to fall back to looking for unparsed-data existing.

That's definitely a bug. I have co-incidently looked at the code
yesterday, the status is set, so it *should* work. Bug Tracker?

Rainer
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.