Hi Raul, This is a brillant feature that you propose here and I think that the time is perfect (with up coming Camel 3.0 release) that we discuss as we need such stats in projects. I think that the plugin strategy proposed is fine and can be included in the camel-core module for in-memory usage. Additional implementations requiring DB, NoSQL storage should be part of a new camel module (maybe) called camel-extra or camel-extend containing different plugins one by implementation (JDBC, JPA, Hazelcast, MongoDb, Cassandra, ...). Some questions come to me like : 1) What is the definition of the "stats" or more precisely which info will be part of the stats (camel JMX stats, camel exchangeId, camel bredCrumbId, id collected by Idempotent - Aggregator - Splitter processors, trace in text, XML formats ...) ? 2) Do we need one plugin by storage engine or one plugin by strategy (= what we would like to store : trace, processors exchange, ....) and storage engine ? 3) Which kind of granularity do we want - all, processors, (= idempotent, splitter, ...), idempotent, splitter, trace (aka text or XML messages that usually we send to logger, ...) ? 4) Which solution are we gonna to use to add this plugin strategy (event notifier + plugin, ....)
Remark : It could be interesting that you create a JIRA ticket to continue this discussion ;-) Regards, Charles people in charge of the operations and business users On Thu, Jan 5, 2012 at 6:58 PM, Raul Kripalani <r...@fusesource.com> wrote: > Responding to Claus' request for feedback on how to store EIP stats and > expose it in several ways... > > With the current architecture where stats are kept in memory there are a > few downsides, mainly: (1) not being able to store rich data, only counters > due to memory usage considerations and (2) stats are lost between restarts. > > A legitimate requirement is to be able to browse the list of exchange IDs > marked as duplicates by the idempotent consumer, but storing those in > memory would cause an immediate memory leak due to an ever-growing list. > Good thing is that at least we can set skipDuplicate to false and react to > the DUPLICATE_MESSAGE storing the information somewhere, but it would be > easier and less error prone if there was a centralised place to resort to > to obtain better richer visibility and governance, without the user having > to implement hooks here and there to react when things happen inside Camel > (tracer, interceptor, etc.). > > In my mind, the ideal solution is to create a generic "management engine" > to which components, EIPs, services, etc. publish their stats. This > represents an important overhaul, but in my opinion it is imperative for > Camel to advance to the next generation ;) > > Features of my proposed "management engine": > > * pluggable persistence repositories, to choose what back-end your stats > will be persisted to (JDBC, MongoDB, Redis, In-memory, etc.) > * flexible external access via pluggable management front-ends (JMX, REST, > raw API), so that any type of client can access the info from the outside > * persistence flushing policy, so that the user can control how often the > management engine will persist in-memory data to the back-end > * purge policy, so that old stats are flushed out on a schedule, i.e. it's > self-cleaning > * centralised filtering of management events, so you can turn off stats for > specific EIPs, Components, etc. from one point only. E.g. (*, > !idempotentConsumer, !camel-jpa) would enable all stats except idempotent > consumer and the camel-jpa component. > * query capabilities based on the Simple language > * async/sync persistence behaviour selection ... based on how important > your stats are to you > * etc. > > > Note that achieving blazing-fast persistence nowadays is a piece of cake > with NoSQL, Big Data and Data Grids. > > I realise the extent of the impact on the codebase to implement this > feature, but seeing that Camel 3.0 is coming up in conversations, I wanted > to drop my 2 cents ;) > > Regards, > Raúl. > > On 5 January 2012 16:14, Babak Vahdat <babak.vah...@swissonline.ch> wrote: > > > As Claus has already suggested [1] I would like to share the question > with > > you for a better possible third approach! > > > > For all the details regarding the proposed improvement please take a look > > at > > the comments and also the 2 attached patches on it and let us know of any > > possible third approach you could think of. Thanks! > > > > [1] https://issues.apache.org/jira/browse/CAMEL-4782 > > > > Babak > > > > > > -- > > View this message in context: > > > http://camel.465427.n5.nabble.com/DISCUSS-How-to-better-enlist-the-duplicate-message-count-by-the-IdempotentConsumer-on-JMX-tp5123118p5123118.html > > Sent from the Camel Development mailing list archive at Nabble.com. > > >
