> >> Our idea is to build a central logging system just by using Log4 and doing >> away with forwarders and queues. > > Don't ypu notice that your giving your own answer? Use of a forwarder > and a queue is (IMO) the obvious choice for a central logging system, > based on existing components, and with the least effort in terms of > custom programming. >
It's not obvious (IMHO) that removing the requirement for a queuing system and central logging server is intrinsically more effort, more programming or worse . If the database is already a requirement and HA, and otherwise appropriate to the need, especially if the logs need to end up there at the end. It may be much easier etc to write directly to it. A robust queue system is essentially a distributed database and a HA server cluster you can't use for other purposes but still have to manage. The crucial consideration, IMHO, is if the database can match the requirements of a realtime logging store wrt the requirements of your apps, Tolerance for loss of data vs performance vs accessibility , competition with other uses of the DB. The workload characteristics of logging can be very different then traditional transactional DB access and likewise traditional RDBMS servers. Both are changing in the "NoSQL" world ... Apps are trading out HA and transactional guarantees for performance, ease of use and prevalence of streaming data , many DBs are doing the same. The characteristics of App data + App DB are approaching that of logging systems from both sides ... It may well be a good match, Some "NoSQL" DBs don't make you sacrifice HA and transactional guarantees for performance. (Instead you pay for them, not as many community volunteers to do the really hard parts for the years it takes without a paycheck It's all a balance of cost vs value - what is your data worth? If you can afford to loose an occasional tweet or log record, or extra staff the system will cost less in cash . Even with a very high performance HA DB if the value per IO of log records is less then 'regular data' it's a tough sell to use 50% of the server capacity for. 1% of the ROI Utility compute /cloud services stat to look very attractive in this space, Offload the IT management of log collection IT, pay for it are a utility rate instead upfront cash and iIT staff, use their aggregation and reporting tools then suck in the consolidated filtered data at your leisure or use their gui for realtime analysis ... And who wants to write a decent log analysis tool, or learn another set of install config,and feeding of yet another tool that s supposed to make things easier, but only for the guy who wrote it, that is way down on my list of ways to totally waste a month and give up at the end, . There are some foot companies out there with cost attractive high value services. I use Loggly for several projects. Tried logentities , it's good but Focused note on analytics when I need search AWS logging has recently come to center stage all of those use NoSQL databases for the logs, (bit only the logs ) My major concern would be that I wouldn't find out soon enough in the dev cycle what the affects will be on the whole system. It's likely to need a completely functional system end to end run under high load and represent ice workloads for long periods to find out. Calclate the risk/cost of that going really bad very late in the cycle vs taking a well paved trail with known risk and cost upfront -- in best scenario You have a fallback plan or low cost of failure - say the project is not business critical so you can absorb risk and may well come out quite ahead David A Lee [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
