>> What is really cool about this approach is that it completely isolates the
>> application from how the data is going to be stored.
I find this is what is really cool about Log4j -- it (nearly) completely
isolates the application from how the logging is delivered.
Its all in the configuration. (of course, all the impl code has to exist, but
thats true already - see below )
And with Log4j2, with the added late evaluated formatters its even more
practical to defer formatting to the configuration instead of building strings
in the app specific to the log endpoint requirements.
( I'm still pre-conversion stage on one app and it really annoys me when I have
to do
mLogger.debug( "Called with : " + arg.toString() + "and " +
Type(arg2));
-- I can hear the wasted CPU cycles as much as my fingers cracking under the
strain
)
>> When you have dozens of servers that is a distinct advantage for using the
>> “agent/aggregator model” as only a few servers have to be reconfigured.
Definitely true:
But isn't that nearly the same as 'simply' changing the log4j configuration ?
(to refer to a different appended)
Where, what and when the changes get made may be different depending on how you
originally set things up, but as long as the app main code isn't implicitly
depending on it then even on a totally locked down encapsulated system like say
docker or EC2 ...
This change could be done with only an environment variable change at the OS
launch.
Providing of course that the log4j appended code all pre-exists (or can be
tossed on. The details of how easy or practical it is to do that will of
course vary ... But by how you have so well separated implementation from
application - to me it seems Your lesson was so well learned that you have
achieved the same level of isolation but now at the log4j configuration level
as well - so the range of choice points is now larger and better.
----------------------------------------
David A. Lee
[email protected]
http://www.xmlsh.org
-----Original Message-----
From: Ralph Goers [mailto:[email protected]]
Sent: Wednesday, July 22, 2015 11:24 PM
To: Log4J Users List <[email protected]>
Subject: Re: Log4J2 NoSQL query
I agree with what you are saying here, but there is one other thing I would
like to add.
One of my motivations for starting work on Log4j 2 was that my employer had a
need for a very flexible audit logging system. Most applications I am aware of
do their auditing to a database. However, we were adding new audit events with
new event details almost every time we did an enhancement or introduced a new
product. I found using a logging API with RFC 5424 messages to be an ideal
solution (which is why you will find a few examples of that on the web site).
What is really cool about this approach is that it completely isolates the
application from how the data is going to be stored. We created an API around
that (which one of these days I would like to turn into a companion project for
log4j) and used the Flume appender to get the data out of the application. We
were having some problems with one of the data repositories we were using and
my understanding is that after I left the company the data was just routed to a
different data store, without having to change any applications. When you have
dozens of servers that is a distinct advantage for using the “agent/aggregator
model” as only a few servers have to be reconfigured.
Ralph
> On Jul 22, 2015, at 8:07 PM, David Lee <[email protected]> wrote:
>
>
>>
>>> Our idea is to build a central logging system just by using Log4 and
>>> doing away with forwarders and queues.
>>
>> Don't ypu notice that your giving your own answer? Use of a forwarder
>> and a queue is (IMO) the obvious choice for a central logging system,
>> based on existing components, and with the least effort in terms of
>> custom programming.
>>
>
> It's not obvious (IMHO) that removing the requirement for a queuing system
> and central logging server is intrinsically more effort, more programming or
> worse .
> If the database is already a requirement and HA, and otherwise appropriate
> to the need, especially if the logs need to end up there at the end.
> It may be much easier etc to write directly to it.
> A robust queue system is essentially a distributed database and a HA server
> cluster you can't use for other purposes but still have to manage.
>
> The crucial consideration, IMHO, is if the database can match the
> requirements of a realtime logging store wrt the requirements of your apps,
> Tolerance for loss of data vs performance vs accessibility , competition with
> other uses of the DB. The workload characteristics of logging can be very
> different then traditional transactional DB access and likewise traditional
> RDBMS servers. Both are changing in the "NoSQL" world ...
> Apps are trading out HA and transactional guarantees for performance, ease of
> use and prevalence of streaming data , many DBs are doing the same. The
> characteristics of App data + App DB are approaching that of logging systems
> from both sides ... It may well be a good match,
> Some "NoSQL" DBs don't make you sacrifice HA and transactional
> guarantees for performance. (Instead you pay for them, not as many
> community volunteers to do the really hard parts for the years it
> takes without a paycheck It's all a balance of cost vs value - what is
> your data worth? If you can afford to loose an occasional tweet or log
> record, or extra staff the system will cost less in cash . Even with
> a very high performance HA DB if the value per IO of log records is
> less then 'regular data' it's a tough sell to use 50% of the server
> capacity for. 1% of the ROI
>
> Utility compute /cloud services stat to look very attractive in this space,
> Offload the IT management of log collection IT, pay for it are a utility rate
> instead upfront cash and iIT staff, use their aggregation and reporting tools
> then suck in the consolidated filtered data at your leisure or use their gui
> for realtime analysis ... And who wants to write a decent log analysis tool,
> or learn another set of install config,and feeding of yet another tool that s
> supposed to make things easier, but only for the guy who wrote it,
> that is way down on my list of ways to totally waste a month and give
> up at the end, . There are some foot companies out there with cost
> attractive high value services. I use Loggly for several projects.
> Tried logentities , it's good but Focused note on analytics when I
> need search AWS logging has recently come to center stage all of those
> use NoSQL databases for the logs, (bit only the logs )
>
>
> My major concern would be that I wouldn't find out soon enough in the dev
> cycle what the affects will be on the whole system. It's likely to need a
> completely functional system end to end run under high load and represent ice
> workloads for long periods to find out.
> Calclate the risk/cost of that going really bad very late in the
> cycle vs taking a well paved trail with known risk and cost upfront --
> in best scenario You have a fallback plan or low cost of failure - say
> the project is not business critical so you can absorb risk and may
> well come out quite ahead
>
>
>
> David A Lee
> [email protected]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
