ignore this given the other thread on the topic.
David Lang
On Wed, 22 Jan 2014, David Lang wrote:
Date: Wed, 22 Jan 2014 12:53:15 -0800 (PST)
From: David Lang <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: [rsyslog] modules in other languages Re: rsyslog mission statement -
was: logo
On Wed, 22 Jan 2014, Rainer Gerhards wrote:
On Wed, Jan 22, 2014 at 2:22 PM, Boylan, James
<[email protected]>wrote:
I agree with many of the points about the mission statement. I would
really like to see Rsyslog continue on the path of becoming one of the
defacto unified data transport solutions. To me, a true unified data
transport needs to not just be able to send the messages to a destination,
but control how it arrives there as well. I need to dig into it more, but
things like the log normalization functionality appears to play a strong
role in that part as well.
I would note that I think the idea of trying to add the ability to use
another language as part of the process to build modules or plugins to be
a
bad idea. I know this wasn't specifically what Radu was suggesting, but it
has been suggested before. One of the fundamental strengths, in my
opinion,
of Rsyslog is the speed it is able to process the materials being passed
to
it. Like it or not, no other language is going to be able to parse at the
levels that C can handle. That is just a simple reality.
I (think I ;)) have a relaxed approach and I guess this is what Radu means.
Performance is critical, but probably not for every case. If we find a way
to support non-C modules in a way that *does not slow down* the native
code, I think this would be a big improvement.
Let's say someone needs to output to an xyz destination. He doesn't even
need a lot of performance, just a couple of thousand messages per second.
He doesn't know C but python and he knows how to connect to the xyz source
via python. If we now provide an easy enough way to connect that python to
rsyslog, and do so without affecting other users AND other output sources,
that user will probably write the small connector and hopefully contribute
it to the project. His problem is solved, and rsyslog got a new
second-class (performance wise) output module.
Now think someone else comes in and needs that exact functionality, but at
a much higher pace. Then there are basically two options: a) port it (or
have it port) to C or b) use some other tool. From what I have read and
heard in the past month, b) does not seem to be a vital option. So we may
get a), and thus get a first-class output plugin (except he uses option c
and invests that money in more hardware ;)). In any case, the second-class
plugin may be *the* tool to motivate someone to actually carry out the
port.
I think this is doable - already via omprog. A related improg is also not
hard to write and we could also create a generic message modification
module interface (though probably with a somewhat higher performance
impact).
I was thinking about this, and the biggest problem I see in getting the data
back and forth from these external modules is the serialization and parsing
of the messages at the boundries. I'm thinking that it may be worthwhile to
use something like google Protocol Buffers or apache Thrift for the
serialization, they are written to be fast. Thrift comes with the extra
baggage of an entire network service stack that we don't want, but both
support many languages. Since I started at Google I've been exposed more to
Protocol Buffers, but I know many people like Thrift, so we would need to
look at both.
David Lang
Rainer
That being said, I do agree the documentation needs work. That was one of
the primary reasons I agreed to help Rainer with the Rsyslog-doc project.
My plan is that once the existing documentation is converted over to the
new configuration I can work on updating, clarifying and adding to the
documentation to help resolve this. It was why I had been asking people to
email this distro with a [doc] or [docs] in the subject line detailing
issues or concerns they have with the documentation. It will be extremely
helpful in trying to work on improving the quality of it.
Do you have a process that you had to fight through to get working? Write
up instructions on how you did it and email it with the [docs] tag to the
Distro. Found an error? Email it to the distro if that documentation set
hasn't yet been completed in the Git repo. If it is done in the Git repo,
feel free to submit a PR.
-- James
-----Original Message-----
From: [email protected] [mailto:
[email protected]] On Behalf Of Rainer Gerhards
Sent: Wednesday, January 22, 2014 4:26 AM
To: rsyslog-users
Subject: Re: [rsyslog] rsyslog mission statement - was: logo
just for the records: I've read this posting, like it very much but need
some time to digest before I can craft a useful response.
Rainer
On Wed, Jan 22, 2014 at 7:48 AM, Radu Gheorghe <[email protected]
wrote:
Hi,
I think I need to make my thoughts clear regarding the flexibility
topic, because I saw the same position by Otis and I saw some
reactions to my earlier Email.
Let's imagine logging tools are cars. Rsyslog seems to be one that's
resource-efficient (light) and has a good top speed (fast). Handling
(configuration) is not that easy and all-terrain abilities (adding new
modules) are also difficult. Those two things are the flexibility I
was talking about.
My journal + mongodb example was intentionally chosen as something
that rsyslog *can* do, but it's not that easy. We can look at the causes:
documentation is far from where we want it, names like "imuxsock" or
"mmnormalize" are far from being intuitive. In car terms, this is what
I mean by "handling is difficult": it can take a turn, but requires
some skill.
Writing a new module is possible, or course, but again: documentation
can be more detailed and writing this C code is nowhere near as easy
as, say, Ruby. In car terms, this is what I mean by "all terrain
ability": it can go on some gravel, but I wouldn't cross a river with it.
Can we improve on flexibility? Of course we can and it's a good thing
to do. If a car can't take a turn without flipping over it's useless
and if it can only go on track only a handful of people will buy it.
But we can't take a Ferrari and make it a VW Touareg. We can improve
it in lots of ways, make it handle better (improve docs) or even add
4-wheel drive (improve omprog, add more modules) but ultimately there
are some design decisions that limit how far we [want to] go in that
direction, especially when there are good SUVs out there on one hand
and people need what rsyslog offers on the other.
Making rsyslog (more) flexible by helping users configure and extend
it easier is definitely something that we should do. But making it a
mission
(ie: a top priority) out of this? I think it's not credible, nor is it
realistic. I'd rather rsyslog as a tool that's light, fast and
reliable
(mission!) and that can also do a lot of stuff.
Like a McLaren F1: can still smoke supercars after 25 years, with a
couple of buddies and their backpacks on-board. Its mission was to be
the ultimate driver's car, and not to be a usable/flexible
grand-tourer. Flexibility comes in as P2, and it's important so you can
actually use the thing.
I hope I didn't bore you with my comparison, I tried my best to make
myself clear. On the upside, the designer guy I talked to gave me the
first logo.
I'll post it on a different thread.
Best regards,
Radu
2014/1/21 Rainer Gerhards <[email protected]>
Sorry folks, i had some very time critical things on my agenda... i
overlooked a dependency ;) will rejoin this great discussion tomorrow.
Just so that you know i am very interested. Actually, its kind of a
reality check for me. Flexibility was always high on ny agenda, it
probably
has slipped for performance without me noticing. I'd like to get
most of both. Maybe with the upcoming non-c interface...
Keep the thoughts flowing!
Rainer
Sent from phone, thus brief.
Am 21.01.2014 19:35 schrieb "Dave Caplinger" <
[email protected]
:
On Jan 20, 2014, at 4:24 PM, David Lang <[email protected]> wrote:
On Mon, 20 Jan 2014, Radu Gheorghe wrote:
I'm not saying rsyslog shouldn't do flexibility. I think it's
uberimportant
and it's worth investing in. I'm saying we should go with one
of the directions where rsyslog is pushed to that:
- is already partially accomplished (so it's credible)
- has the potential to go
- last but not least, where we want it to go :)
I thought something that includes the words fast, light and
processing
will
do, given the history of rsyslog and where it seems to go now
with
v8.
I see rsyslog as being the core of a logging system, it gathers
logs
from
(almost) anything, and delivers them to (almost) anything. It
can
modify
and
filter the messages along the way.
This is similar to my own "customer testimonial." The three main
reasons
I
switched to rsyslog are:
1) Much higher performance.
2) It has DAQ, detailed pstats, TLS, RELP, and now log-signing
support
so
it's reliable in the sense that logs that get in are not going to
get
lost
someplace mysteriously. (Even drops outside your control become
manageable/correctable.)
3) Property replacement, JSON, and filtering, allow you to modify
and route logs as you like.
Going with the R-theme (since 'R' initially meant Reliable) that
gives
me:
* Reliable
* Rapid
* Routing
(Back to the logo: borrowing from another of Rainer's interests
that I happen to share, maybe R is for Rocket [with apologies to
Ray
Bradbury].)
Contrast this with logstash, which is extremely flexible: it can
connect
just about any input to just about any output, like
pipe/grep/awk/etc.
for
log streams. It's a "log format translator", but not necessarily
a high-performance "log router".
- Dave
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
