Rainer Gerhards wrote:
I agree there is inconsistency - this is default on while e.g.
mmjsontransform is not.
Overall: I try to keep brief, pls ask further questions when needed.
Maybe this is even something for a git hub discussion.
Overall, rsyslog has been in use in log/data pipelines since the early
2000's, even when these terms were not yet coined. We see increasing
adoption in especially JSON logging, with a preference of
metadata-enriched logs. At the same time, many folks do not even
remember that rsyslog can do this for 15+ years now. Probably because
it is often complicated or some minor but important things are
missing.
One of our core projects is to address this and make log pipeline use
not only efficient but easy to do.
One way would be to put everything in liblognorm only. We have instead
decided to do it by dedicated smaller modules for reasons.
one reason why so many things have been in different modules is to reduce the
footprint of rsyslog. As all systems (including embedded) have had storage grow
a lot, we may want to consider including more by default, even if they bring in
some additional dependencies
We would need to proceed carefully, pulling in liblognorm by default is not that
big a deal, but we don't want to accidently trigger the install of PostgreSQL
accidently by including a module that can write to it :-)
I know Debian has a pretty flexible set of dependency options, but some other
distros are more in the all-or-nothing camp
The new mmleefparse, mmsnareparese, existing e.g. mmjsonparse and the
important mmjsontransform all address this need. We envision that over
time almost everyone will need them - log enrichment at the edge is a
hot topic, and so for a very good reason.
At the same time, all these module binaries are pretty small. So it
may make sense to include them directly in the base package. Pro is
also that it enhances usability because it is always present (a
typical complaint from rsyslog users is that they need to install
additional component packages).
what is the difference in both size and performance in having these as separate
module binaries vs a thin shim layer over liblognorm/mmnormalize? (my guess is
tht the direct implementation is faster, and not significantly larger, if at
all)
An alternative packaging approach is to put all of them into an e.g.
rsyslog-pipeline package. From my PoV reduces usability. I would also
not be surprised if the binary size is not more than the metadata size
itself.
This seems significantly worse.
I am undecided and appreciate feedback.
As a lightly related side note, yaml config-type artefacts are also
being much requested. We are thinking hard to support this, at least
in some places (upcoming enhancements to mmjsonparse for schema
transformation might be such a place).
more info on this please?
How to do this packaging-wise
is actually a larger concern for me (libyaml is almost on any system,
but not necessarily on embedded - so disable yaml by compilation or
make it a loadable functionality which requires internal interface and
mock overhead).
Bottom line: the IT logging landscape changes continuously, and
rsyslog follows or leads these changes.To remain relevant, adaptation
to new tech and trends is necessary, no matter of personal likes (e.g.
I do not like yaml that much, but being forced to getting used to it,
I see value).
yes, things change, and the easier it is to use rsyslog out-of-the-box without
having to figure out a bunch of other packages to load the better. I think we
may have gone slightly overboard in our efforts to keep the core small and not
include any modules that needed other libraries.
We should keep compile time options to turn off modules (and their dependencies)
for embedded use, but for more powerful systems, the overhead of dealing with
that reduces or eliminates the savings.
David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
