This sound really interesting. Currently our legacy products are integrated with external applications using an integration platform that has been developed in-house. We are now creating an integration platform that is Camel based instead. One of the things we had support for in our old integration platform was a message repository. All messages (incoming and outgoing) were stored in a database along with information showing whether the message was processed successfully or not. We also have a rich user interface for finding and inspecting messages as well as resending messages that have failed.
I think this is required functionality for any integration platform. So far, this has not been part of Camel and I have envisioned to keep developing this functionality in house. It seems like you are approaching this from the statistics side (which is fine) but if you broaden the concept to a complete message repository along with querying and resending functionality then this could be extremely useful. /Bengt 2012/1/6 Charles Moulliard <cmoulli...@gmail.com> > Hi Raul, > > This is a brillant feature that you propose here and I think that the time > is perfect (with up coming Camel 3.0 release) > that we discuss as we need such stats in projects. I think that the plugin > strategy proposed is fine and can be included > in the camel-core module for in-memory usage. > Additional implementations requiring DB, NoSQL storage should be part of a > new camel module (maybe) called camel-extra or camel-extend containing > different plugins one by implementation (JDBC, JPA, Hazelcast, MongoDb, > Cassandra, ...). > Some questions come to me like : > 1) What is the definition of the "stats" or more precisely which info will > be part of the stats (camel JMX stats, camel exchangeId, camel bredCrumbId, > id collected by Idempotent - Aggregator - Splitter processors, trace in > text, XML formats ...) ? > 2) Do we need one plugin by storage engine or one plugin by strategy (= > what we would like to store : trace, processors exchange, ....) and storage > engine ? > 3) Which kind of granularity do we want - all, processors, (= idempotent, > splitter, ...), idempotent, splitter, trace (aka text or XML messages that > usually we send to logger, ...) ? > 4) Which solution are we gonna to use to add this plugin strategy (event > notifier + plugin, ....) > > Remark : It could be interesting that you create a JIRA ticket to continue > this discussion ;-) > > Regards, > > Charles > > people in charge of the operations and business users > On Thu, Jan 5, 2012 at 6:58 PM, Raul Kripalani <r...@fusesource.com> > wrote: > > > Responding to Claus' request for feedback on how to store EIP stats and > > expose it in several ways... > > > > With the current architecture where stats are kept in memory there are a > > few downsides, mainly: (1) not being able to store rich data, only > counters > > due to memory usage considerations and (2) stats are lost between > restarts. > > > > A legitimate requirement is to be able to browse the list of exchange IDs > > marked as duplicates by the idempotent consumer, but storing those in > > memory would cause an immediate memory leak due to an ever-growing list. > > Good thing is that at least we can set skipDuplicate to false and react > to > > the DUPLICATE_MESSAGE storing the information somewhere, but it would be > > easier and less error prone if there was a centralised place to resort to > > to obtain better richer visibility and governance, without the user > having > > to implement hooks here and there to react when things happen inside > Camel > > (tracer, interceptor, etc.). > > > > In my mind, the ideal solution is to create a generic "management engine" > > to which components, EIPs, services, etc. publish their stats. This > > represents an important overhaul, but in my opinion it is imperative for > > Camel to advance to the next generation ;) > > > > Features of my proposed "management engine": > > > > * pluggable persistence repositories, to choose what back-end your stats > > will be persisted to (JDBC, MongoDB, Redis, In-memory, etc.) > > * flexible external access via pluggable management front-ends (JMX, > REST, > > raw API), so that any type of client can access the info from the outside > > * persistence flushing policy, so that the user can control how often the > > management engine will persist in-memory data to the back-end > > * purge policy, so that old stats are flushed out on a schedule, i.e. > it's > > self-cleaning > > * centralised filtering of management events, so you can turn off stats > for > > specific EIPs, Components, etc. from one point only. E.g. (*, > > !idempotentConsumer, !camel-jpa) would enable all stats except idempotent > > consumer and the camel-jpa component. > > * query capabilities based on the Simple language > > * async/sync persistence behaviour selection ... based on how important > > your stats are to you > > * etc. > > > > > > Note that achieving blazing-fast persistence nowadays is a piece of cake > > with NoSQL, Big Data and Data Grids. > > > > I realise the extent of the impact on the codebase to implement this > > feature, but seeing that Camel 3.0 is coming up in conversations, I > wanted > > to drop my 2 cents ;) > > > > Regards, > > Raúl. > > > > On 5 January 2012 16:14, Babak Vahdat <babak.vah...@swissonline.ch> > wrote: > > > > > As Claus has already suggested [1] I would like to share the question > > with > > > you for a better possible third approach! > > > > > > For all the details regarding the proposed improvement please take a > look > > > at > > > the comments and also the 2 attached patches on it and let us know of > any > > > possible third approach you could think of. Thanks! > > > > > > [1] https://issues.apache.org/jira/browse/CAMEL-4782 > > > > > > Babak > > > > > > > > > -- > > > View this message in context: > > > > > > http://camel.465427.n5.nabble.com/DISCUSS-How-to-better-enlist-the-duplicate-message-count-by-the-IdempotentConsumer-on-JMX-tp5123118p5123118.html > > > Sent from the Camel Development mailing list archive at Nabble.com. > > > > > >
