Re: [rsyslog] Feedback request: minimal log shipper project

[Rainer Gerhards]

2016-11-22 18:14 GMT+01:00 David Lang :
> On Tue, 22 Nov 2016, Rainer Gerhards wrote:
>
>> The current packages are very granular. You don't gain much by
>> re-packaging, at least this is what I think. If I am wrong, I am all
>> ears, as it should be fairly easy to change that at least for the
>> project-provided packages.
>
>
> The reasons I'm thinking of a diffeent package are:
>
> 1. compile out everything possible (let it use the rsyslog packages if the
> rsyslog package is included and the config is changed to need them)
>
> 2. change the default config file to something other than /etc/rsyslog.conf
>
> 3. include a very different default config file
>
> 4. changed binary name (something like logsend instead of rsyslogd)
>
> looking at these.
>
> #1 may not be that useful, I'll have to play around with things.
>
> #2 is needed to reduce complexity and support #3
>
> #3 is where a lot of the change comes in. I don't know if we change the main
> queue type to direct or just shrink it down to something on the order of 100
> entries, but changes like that that will make it use FAR less memory.
>
> #4 is a pure social engineering play

interesting idea...

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project

[David Lang]

On Tue, 22 Nov 2016, Rainer Gerhards wrote:


The current packages are very granular. You don't gain much by
re-packaging, at least this is what I think. If I am wrong, I am all
ears, as it should be fairly easy to change that at least for the
project-provided packages.


The reasons I'm thinking of a diffeent package are:

1. compile out everything possible (let it use the rsyslog packages if the 
rsyslog package is included and the config is changed to need them)


2. change the default config file to something other than /etc/rsyslog.conf

3. include a very different default config file

4. changed binary name (something like logsend instead of rsyslogd)

looking at these.

#1 may not be that useful, I'll have to play around with things.

#2 is needed to reduce complexity and support #3

#3 is where a lot of the change comes in. I don't know if we change the main 
queue type to direct or just shrink it down to something on the order of 100 
entries, but changes like that that will make it use FAR less memory.


#4 is a pure social engineering play

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project

[David Lang]

On Tue, 22 Nov 2016, Rich Megginson wrote:


On 11/22/2016 09:27 AM, David Lang wrote:

On Tue, 22 Nov 2016, Rich Megginson wrote:

Then can it be made a dynamically loadable module?  Same with every module 
- is there a way to build them and provide them so that they are not part 
of the base rsyslog executable/library?


All of this can be done, it's just a matter of how the packages are 
compiled and the command line used to start it


But it's a bunch of stuff that makes me think it's a bad idea to try to do 
this in the main rsyslog package.


what modules to include by default

what config file to open by default

all these things can be set at compile time, and that's why I'm saying that 
a new package with a slightly different name that has different defaults 
may be the answer.


Having done a lot of package maintenance for Fedora/EPEL/RHEL, I would rather 
not have two slightly different versions of rsyslog packages . . .


I understand, but I'm trying to make something that can be looked at as if it 
was a completely separate program, and operates independently from any syslog 
daemon on the system.


Ideally something that could have a trival command line like

logsend /path/to/file destination

where destination only needs to be specified the first time, and must be the 
same for all files.


This is a combination of simpler configs and possibly a wrapper around rsyslog. 
The idea is to make it trivial to use for the simple use-case, while still 
having the power available under the covers as needs grow more complex.


IMHO, The biggest problem with using rsyslog to do this is the same 
problem we have with using rsyslog to create /dev/log in containers, 
the fact that the config is fixed at startup time.


Can you explain more about what you mean by this?


rsyslog can create additional /dev/log equivalent sockets, and as such, 
you could create them in each container so that there is no need for a 
copy of rsyslog in each container.


What if the container could mount the /dev/log from the host?


You could do a bind mount in each container (something I hadn't thought 
of), but then rsyslog can't tell which container the message comes from. 
IMHO, that's valuable metadata that I want to know.


So if there were a way to get container metadata this way, it would be a 
viable solution?


possibly, I'm having to think hard about this and don't see how you would get 
the metadata.


the path of the socket you get it from wouldn't help, because rsyslog would see 
them all as the /dev/log in the host OS.


SCM_Credentials may be able to do something here, but it would have to get info 
on the container namespaces, which starts getting ugly.


It seems much easier to have the ability to do something like

rsyslog-ctl add --name "container1" /containers/foo/dev/log

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project

[Rainer Gerhards]

>>> +1 - rsyslog is already much more scalable than most other log
>>> collectors. Why "especially no imjournal"?
>>
>>
>> As per a discussion here a couple months ago, adding imjournal more than
>> doubles the memory footprint of a running rsyslog instance, even if it's not
>> used.
>
>
> Then can it be made a dynamically loadable module?  Same with every module -
> is there a way to build them and provide them so that they are not part of
> the base rsyslog executable/library?

That's what they are. If you don't load a module, it'll have no
footprint. I think David was refering to that we often simply need
imjournal to get hold of the regular syslog.

For a file content shipper, that's totally unnecessary.

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project

[David Lang]

On Tue, 22 Nov 2016, Rich Megginson wrote:


On 11/22/2016 06:47 AM, David Lang wrote:

On Tue, 22 Nov 2016, Rainer Gerhards wrote:


if you are interested in a very lightweight log shipper for files, I
would appreciate your feedback on this:

http://blog.gerhards.net/2016/11/would-creating-simple-linux-log-file.html 


posted to the blog, but repeating here:

I have not found a good tool yet (I've written or seen written a couple 
over the years)


The problem tends to be that there is not really such a thing as 'no 
additional processing needed', any one use requires a tiny fraction of the 
capabilities that rsyslog offers, but each setup requires a different 
combination of capabilities.


What you would have in this case (and this is the use case of other 
lightweight collectors such as filebeat and fluentbit) is the collector agent 
running on the node would do "practically" no log processing, and instead 
ship the logs to a cluster of "heavyweight" rsyslogs doing additional 
processing.  This allows you to

* make the node as small and fast as possible
* use all of the capabilities and plugins of rsyslog to do your additional 
processing

* scale up the additional processing


my problem has been that I find you tend to loose metadata that you need to have 
to properly do the additional processing (exactly what directory did this file 
come from, what is the timezone on the sending machine, etc)


just look at all the capabilities that imfile has been growing over the 
last few years.


I suspect that a stripped down compile of rsyslog (no input modules other 
than imfile, especially no imjournal, etc) would end up being competitive 
to just about any special-purpose program.


+1 - rsyslog is already much more scalable than most other log collectors. 
Why "especially no imjournal"?


As per a discussion here a couple months ago, adding imjournal more than doubles 
the memory footprint of a running rsyslog instance, even if it's not used.


IMHO, The biggest problem with using rsyslog to do this is the same problem 
we have with using rsyslog to create /dev/log in containers, the fact that 
the config is fixed at startup time.


Can you explain more about what you mean by this?


rsyslog can create additional /dev/log equivalent sockets, and as such, you 
could create them in each container so that there is no need for a copy of 
rsyslog in each container.


The problem is that since the config of rsyslog if fixed, the directories the 
containers are going to be in would all have to exist at the time rsyslog 
starts. And those directories could not be removed until rsyslog stops (and 
closes the files it has open)


In the world where you spin up containers as load changes, and shift them from 
machine to machine to balance the load, this is a problem.


Similarly with reading files, you frequently want to start a new process that 
will create files, and not have to change the rsyslog config and restart it 
(affecting the processing of other logs) just to start grabbing a new file.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project

[David Lang]

On Tue, 22 Nov 2016, mosto...@gmail.com wrote:

http://blog.gerhards.net/2016/11/would-creating-simple-linux-log-file.html 
IMHO: the "issue" we're having with rsyslog is understanding the code rather 
than performance. Why you'll invest time doing a "faster reader" if you 
already have a "fast-enough processor"? Has anyone complained about speed? 
What would be the benefit from such development? Will it increase reading 
speed by 2x? 4x? 10x?


I suggest you taking this approach:

* Read file line by line (consider also it handles multiline)
* Measure speed (lines/sec, MB/sec, chars/sec...
* Read the same file with imfile with similar reading configuration
* Measure speed (lines/sec, MB/sec, chars/sec...
* Compare both times and think what you can do to make it faster :P


I have not found a good tool yet (I've written or seen written a couple 
over the years)
What are the issues with imfile? I don't know when rsyslog development 
started, but perhaps time for refactoring has come.


well, the last time I had to do this, the problem was that the company didn't 
want to spend the money for rsyslog on Windows. The time before that it was that 
rsyslog wasn't available on AIX


The time before that I created readmode in imfile to address some of the issues.

:-)

That said, here is where I still see issues just using imfile

1. it's hard to extract the proper metadata out of imfile and use it in the 
resulting log.


imfile was designed for the case where the file has no meaningful structure. But 
if there is structure, it's hard for imfile to take advantage of it (requiring 
custom templates to replicate the existing log formats) having the ability to 
use mmnormalize to populate the default properties, or an easy way to override 
the default properties in an output would help a lot


2. you can't add a file to be monitored without having to modify the rsyslog 
config and do a full stop/start, which impacts other uses


I suspect that a stripped down compile of rsyslog (no input modules other 
than imfile, especially no imjournal, etc) would end up being competitive 
to just about any special-purpose program.
I don't know if I understood properly: is there such a thing as 
--disable-imjournal? will this make rsyslog faster?


it would make it a lot smaller

Is there any place where I can see "everything than can be disabled in order 
to make rsyslog even faster"?


it's less a matter of speed than it is size, rsyslog is getting larger with all 
the libraries it's using, which makes it arguably less appropriate for such a 
simple task.


The default queue size is also probably not appropriate for this simple 
use-case. Why read so much from the file and put it in RAM?


./configure --help shows these options. I'm wondering if having a minimal build 
that created a different binary, used a different config and stripped everything 
down would be useful


IMHO, The biggest problem with using rsyslog to do this is the same problem 
we have with using rsyslog to create /dev/log in containers, the fact that 
the config is fixed at startup time.

That's somehow why I was asking about imhiredis.
I don't know yet how rsyslog works internally, but I'll love being able to 
reload/add/remove configuration/pipelines during execution.


Rsyslog does a lot of things as it starts up, this includes loading modules, 
which each provide code for interpreting the config file, and then running the 
parsed configuration through an optimizer.


This makes a pretty significant difference (a lot of the config statements that 
make a lot of sense to a person are slow to interpret, and things that are fast 
to process are harder to read)


so adding/removing large chunks of config dynamically is probably not a 
reasonable thing to do.


Adding/removing inputs where the rest of the processing/config does not change 
is much less disruptive.


Having a command socket that rsyslog listened to that would let you 
add/remove inputs (files or unix sockets), but didn't allow you to change 
anything else in the config would let you easily tell rsyslog to start 
watching a new container or file as needed, and then stop watching so that 
it doesn't prevent the container or directory from going away when the 
app/container is removed.


It's all this because rsyslogs polls for new files/dir changes or something 
like that?


yes, rsyslog maintains an open filehandle (/dev/log) for unix sockets, and holds 
the directory open to see file changes. These prevent removing/unmounting these 
directories.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project

[Rainer Gerhards]

2016-11-22 15:35 GMT+01:00 mosto...@gmail.com :
>>> http://blog.gerhards.net/2016/11/would-creating-simple-linux-log-file.html
>
> IMHO: the "issue" we're having with rsyslog is understanding the code rather
> than performance. Why you'll invest time doing a "faster reader" if you
> already have a "fast-enough processor"? Has anyone complained about speed?
> What would be the benefit from such development? Will it increase reading
> speed by 2x? 4x? 10x?

I need to clarify: IMO "lightweight" is not directly related to
"fast". In fact, a lightweight agent can be somewhat slower, which is
fully acceptable if uttermost speed is not the prime concern. A
"lightweight" agent IMO is one that

* uses as few as possible main memory
* uses as few as possible CPU cycles
* (boils down to "uses as few ressources as possible")
* but is *sufficiently* fast for the task at hand

Lightweight IMO often means throtelling down or running an agent at
lower priority to not change the system footprint at all. That's quite
different from rsyslog's design goal, at least since systemd journal
entered the picture.

An example: running on multiple threads, if done correctly, helps
improve the speed. However, at the same time it often increases memory
requirements and puts larger burden during burst times (simply because
it handles so much more than a single-threaded one).

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Would imhiredis make sense?

[David Lang]

On Tue, 22 Nov 2016, mosto...@gmail.com wrote:

What sort of log volume are you talking about here? (logs/sec type of 
thing)

From 0 to thousand-thousands/sec

Logstash needs something like redis because it can't do any queueing 
itself. Rsyslog is built around queues, and has the ability to create 
multiple queues and piplines internally, you don't need to run multiple 
instances.

I want multiples instances in order to:

* Being able to process pipelines on different containers/hosts


much less needed on rsyslog due to the higher effiency. I've had rsyslog 
handling over a hundred thousand logs/sec on a single host.



* Isolate pipelines to prevent problems on one affecting others


rulesets with queues on each ruleset solvs this for you.


* (others)


that's hard to answer :-)



What you would do is create a ruleset for each application (pipeline) and 
give that ruleset it's own queue.
I know it can be done, but not what I'm looking for. Moreover, I would love 
to be a "dynamic" configuration


As new logs arrive, you then sort them by application, and for each 
application (or application category), you call the appropriate ruleset.


And, if there are a lot of evt/sec, you may have a bottleneck. I'll probably 
have a rsyslog cluster based on docker swarm mode


This is unlikly to be a bottleneck. The overhead of recieving a log message, 
parsing it, and looking up what ruleset to call is very cheap. At anything under 
several hundred thousand logs/sec it's unlikly to max out a single core.


All processing from that point on will take place in different threads 
working on different queues for each category.
Will I be able to "reload" rsyslog configuration to add/delete new 
rulesets/pipelines?


you can stop/start rsyslog, but there is not a way to change the config on the 
fly.


However, if you really want to go this way, one thing you can do is to make use 
of the multicast mac feature in ethernet to distribute the same logs to multiple 
systems/containers and have each container throw away all logs except what it's 
configured to handle.


This lets you add/remove log processing at any time and even have multiple 
systems processing the same logs in different ways


https://www.usenix.org/conference/lisa12/technical-sessions/presentation/lang_david


Give it a try, I'll bet that you find the result much simpler and faster.

I expecting your reply ;)


KISS, start simple and only add complexity when you find it's actually needed. 
Have plans for how to scale out when you hit limits, but you usually find that 
you hit limits far later than expected. Yes, you may have to eventually do the 
same work, but by having a solid system now with less work, you can spend the 
time saved now to improve other things.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project

[Rich Megginson]

On 11/22/2016 06:47 AM, David Lang wrote:

On Tue, 22 Nov 2016, Rainer Gerhards wrote:


if you are interested in a very lightweight log shipper for files, I
would appreciate your feedback on this:

http://blog.gerhards.net/2016/11/would-creating-simple-linux-log-file.html 



posted to the blog, but repeating here:

I have not found a good tool yet (I've written or seen written a 
couple over the years)


The problem tends to be that there is not really such a thing as 'no 
additional processing needed', any one use requires a tiny fraction of 
the capabilities that rsyslog offers, but each setup requires a 
different combination of capabilities.


What you would have in this case (and this is the use case of other 
lightweight collectors such as filebeat and fluentbit) is the collector 
agent running on the node would do "practically" no log processing, and 
instead ship the logs to a cluster of "heavyweight" rsyslogs doing 
additional processing.  This allows you to

* make the node as small and fast as possible
* use all of the capabilities and plugins of rsyslog to do your 
additional processing

* scale up the additional processing




just look at all the capabilities that imfile has been growing over 
the last few years.


I suspect that a stripped down compile of rsyslog (no input modules 
other than imfile, especially no imjournal, etc) would end up being 
competitive to just about any special-purpose program.


+1 - rsyslog is already much more scalable than most other log 
collectors.  Why "especially no imjournal"?




IMHO, The biggest problem with using rsyslog to do this is the same 
problem we have with using rsyslog to create /dev/log in containers, 
the fact that the config is fixed at startup time.


Can you explain more about what you mean by this?



Having a command socket that rsyslog listened to that would let you 
add/remove inputs (files or unix sockets), but didn't allow you to 
change anything else in the config would let you easily tell rsyslog 
to start watching a new container or file as needed, and then stop 
watching so that it doesn't prevent the container or directory from 
going away when the app/container is removed.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Would imhiredis make sense?

[mosto...@gmail.com]

What sort of log volume are you talking about here? (logs/sec type of 
thing)

From 0 to thousand-thousands/sec

Logstash needs something like redis because it can't do any queueing 
itself. Rsyslog is built around queues, and has the ability to create 
multiple queues and piplines internally, you don't need to run 
multiple instances.

I want multiples instances in order to:

 * Being able to process pipelines on different containers/hosts
 * Isolate pipelines to prevent problems on one affecting others
 * (others)


What you would do is create a ruleset for each application (pipeline) 
and give that ruleset it's own queue.
I know it can be done, but not what I'm looking for. Moreover, I would 
love to be a "dynamic" configuration


As new logs arrive, you then sort them by application, and for each 
application (or application category), you call the appropriate ruleset.
And, if there are a lot of evt/sec, you may have a bottleneck. I'll 
probably have a rsyslog cluster based on docker swarm mode


All processing from that point on will take place in different threads 
working on different queues for each category.
Will I be able to "reload" rsyslog configuration to add/delete new 
rulesets/pipelines?



Give it a try, I'll bet that you find the result much simpler and faster.

I expecting your reply ;)

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project

[mosto...@gmail.com]

Quite interested on this thread


http://blog.gerhards.net/2016/11/would-creating-simple-linux-log-file.html 

IMHO: the "issue" we're having with rsyslog is understanding the code 
rather than performance. Why you'll invest time doing a "faster reader" 
if you already have a "fast-enough processor"? Has anyone complained 
about speed? What would be the benefit from such development? Will it 
increase reading speed by 2x? 4x? 10x?


I suggest you taking this approach:

 * Read file line by line (consider also it handles multiline)
 * Measure speed (lines/sec, MB/sec, chars/sec...
 * Read the same file with imfile with similar reading configuration
 * Measure speed (lines/sec, MB/sec, chars/sec...
 * Compare both times and think what you can do to make it faster :P


I have not found a good tool yet (I've written or seen written a 
couple over the years)
What are the issues with imfile? I don't know when rsyslog development 
started, but perhaps time for refactoring has come.


I suspect that a stripped down compile of rsyslog (no input modules 
other than imfile, especially no imjournal, etc) would end up being 
competitive to just about any special-purpose program.
I don't know if I understood properly: is there such a thing as 
--disable-imjournal? will this make rsyslog faster?
Is there any place where I can see "everything than can be disabled in 
order to make rsyslog even faster"?



IMHO, The biggest problem with using rsyslog to do this is the same 
problem we have with using rsyslog to create /dev/log in containers, 
the fact that the config is fixed at startup time.

That's somehow why I was asking about imhiredis.
I don't know yet how rsyslog works internally, but I'll love being able 
to reload/add/remove configuration/pipelines during execution.


Having a command socket that rsyslog listened to that would let you 
add/remove inputs (files or unix sockets), but didn't allow you to 
change anything else in the config would let you easily tell rsyslog 
to start watching a new container or file as needed, and then stop 
watching so that it doesn't prevent the container or directory from 
going away when the app/container is removed.
It's all this because rsyslogs polls for new files/dir changes or 
something like that?


TBH, I'll love having more expertise to better contribute.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Feedback request: minimal log shipper project

[David Lang]

On Tue, 22 Nov 2016, Rainer Gerhards wrote:


if you are interested in a very lightweight log shipper for files, I
would appreciate your feedback on this:

http://blog.gerhards.net/2016/11/would-creating-simple-linux-log-file.html


posted to the blog, but repeating here:

I have not found a good tool yet (I've written or seen written a couple over the 
years)


The problem tends to be that there is not really such a thing as 'no additional 
processing needed', any one use requires a tiny fraction of the capabilities 
that rsyslog offers, but each setup requires a different combination of 
capabilities.


just look at all the capabilities that imfile has been growing over the last few 
years.


I suspect that a stripped down compile of rsyslog (no input modules other than 
imfile, especially no imjournal, etc) would end up being competitive to just 
about any special-purpose program.


IMHO, The biggest problem with using rsyslog to do this is the same problem we 
have with using rsyslog to create /dev/log in containers, the fact that the 
config is fixed at startup time.


Having a command socket that rsyslog listened to that would let you add/remove 
inputs (files or unix sockets), but didn't allow you to change anything else in 
the config would let you easily tell rsyslog to start watching a new container 
or file as needed, and then stop watching so that it doesn't prevent the 
container or directory from going away when the app/container is removed.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Would imhiredis make sense?

[David Lang]

On Tue, 22 Nov 2016, mosto...@gmail.com wrote:

We've been playing with logstash, rsyslog and redis for a while in order to 
*index into elasticsearch a bunch of application logs*. Briefly: 
app1-file1.log, app1-file2.log...appN-fileX.log -> pipeline -> elasticsearch.


So far, we are using *redis queues and _each application_ processing was made 
by one logstash instance* (docker container). Of course, this works with 5-10 
applications, but it doesn't when you plan to deploy 100 apps cause each 
logstash instance requires ~512MB of RAM.


We've been thinking about rsyslog since the beginning, because it takes fewer 
RAM, but just noticed it doesn't have a *redis input module (aka: imhiredis)*


We still plan to have independent instances (one rsyslog for each 
application), but we're wondering if you'll consider it makes sense to 
implement this module.


What sort of log volume are you talking about here? (logs/sec type of thing)

Logstash needs something like redis because it can't do any queueing itself. 
Rsyslog is built around queues, and has the ability to create multiple queues 
and piplines internally, you don't need to run multiple instances.


What you would do is create a ruleset for each application (pipeline) and give 
that ruleset it's own queue.


As new logs arrive, you then sort them by application, and for each application 
(or application category), you call the appropriate ruleset. All processing from 
that point on will take place in different threads working on different queues 
for each category.


Give it a try, I'll bet that you find the result much simpler and faster.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Feedback request: minimal log shipper project

[Rainer Gerhards]

Hi all,

if you are interested in a very lightweight log shipper for files, I
would appreciate your feedback on this:

http://blog.gerhards.net/2016/11/would-creating-simple-linux-log-file.html

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Fwd: rsyslog user

[Rainer Gerhards]

Do you use imjournal? We have seen cases where the journal library
uses vast amounts of memory. Not sure if the journal team has a fix
for this by now.

Rainer

2016-11-22 11:53 GMT+01:00 rizwana shaik :
> -- Forwarded message --
> From: rizwana shaik 
> Date: Tue, Nov 22, 2016 at 3:43 PM
> Subject: rsyslog user
> To: rsyslog@lists.adiscon.com
>
>
> Hello ,
> Am using rsyslog log for forwarding messages..
> But am found the memory usage of rsyslog continuously increasing .
>
> can u help me out how can i reduce the memory usage of rsyslog
>
> thank u
> rizwana
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Would imhiredis make sense?

[mosto...@gmail.com]

Hi


We've been playing with logstash, rsyslog and redis for a while in order 
to *index into elasticsearch a bunch of application logs*. Briefly: 
app1-file1.log, app1-file2.log...appN-fileX.log -> pipeline -> 
elasticsearch.


So far, we are using *redis queues and _each application_ processing was 
made by one logstash instance* (docker container). Of course, this works 
with 5-10 applications, but it doesn't when you plan to deploy 100 apps 
cause each logstash instance requires ~512MB of RAM.


We've been thinking about rsyslog since the beginning, because it takes 
fewer RAM, but just noticed it doesn't have a *redis input module (aka: 
imhiredis)*


We still plan to have independent instances (one rsyslog for each 
application), but we're wondering if you'll consider it makes sense to 
implement this module.


Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.