Marcel Ruff
Mon, 19 Jun 2006 09:38:47 -0700
Thomas, Johann wrote:
<homage> I must say, as I ran across xmlblaster I found it more and more interesting as I tried to dig deeper to find out it's intention and usage. What I liked was the implemented development process, that surfaced through the reference documentation. I liked the mailing list quality, it assured me, that this product is well alive and has a goodfoundation.I was also astonished of the number of the clients and protocolsimplemented.I would have been happy to find a short introduction to the concepts
Yes, this is missing.
that base the architecture of xmlBlaster, as the reference is sorted by the requirement 'package identifier'. But I also knew, the lack of a tutorial is a guarantee to find a community of smart and skilled developers and users not needing such a introduction to hard ;-) So the mailing list archive was the way to start with for the glue between the reference pages. </homage> I am happy not to patch anything in the swapping code, as it would be a real behaviour change that I expect to have unwanted side effects. I have expected to get some more understanding of how the caching works and find where to block the message data caching. I would be happy if I could get some hints for a few API usage questions I have: 1) Lot of topics: ----------------- I have chosen to use many topics, as: a) I (partly) need to process the messages in random order and b) I have not figured out how to "get" the topic history, seems I get only the last message in the topic. I could use the iteration of the topic history at least where I introduced the splitting of the content in multiple publishes. There I have used the faster, because known, way of different topics.
http://www.xmlblaster.org/xmlBlaster/doc/requirements/interface.get.htmlshows with <history numEntries='20' newestFirst='true'/> how to get history entries.
If you are a subscriber (versus the above getter) you usually are not interested anymore in the history as you will receive all messages. Don't forget to connect the client in a fail safe mode to not loose the messages during the client is down.
2) StoragePlugin vs. QueuePlugin
------------------------------------------
I have expected to get rid of the 'meat' caching by configuring the
StoragePlugin accordingly before. I just got stock before with the
max*Cache settings. Now, from the following to configurations I expect
the first to swap the message data/meat directly to the persistent
store:
StoragePlugin[CACHE][1.0]=${JdbcStorage[HSQLDatabaseEngine]}
QueuePlugin[CACHE][1.0]=${JdbcStorage[HSQLDatabaseEngine]}
For my understanding: Is the second line suppressing the history,
callback, subject and (if client) connection queue RAM caching of only
the references into the StoragePlugin? Or is it necessary for stopping
the 'meat' caching, too?
No, this is perfectly legal as well:
StoragePlugin[CACHE][1.0]=${JdbcStorage[HSQLDatabaseEngine]}
QueuePlugin[CACHE][1.0]=org.xmlBlaster.util.queue.cache.CacheQueueInterceptorPlugin,persistentQueue=JDBC,transientQueue=RAM
Now the meat is on hard disk always, and the references from the queues
are cached (you can choose a huge RAM part to avoid swapping).
Note: All implementations 'CacheQueueInterceptorPlugin', 'RamQueuePlugin' and
'JdbcQueueCommonTablePlugin'
implement the same I_Queue interface and are therefor interchangeable, you can
provide your
own implementation of I_Queue if needed.
enjoy
Marcel
Regards, Johann Thomas -----Original Message----- Von: [EMAIL PROTECTED] [EMAIL PROTECTED] Im Auftrag von Marcel Ruff Gesendet: Montag, 19. Juni 2006 15:49 An: xmlblaster@server.xmlBlaster.org Betreff: Re: [xmlblaster] WG: Help! OutOfMemory by publishing lot big topics Thomas, Johann wrote:... - the content dumped is cached "somewhere" in the heap, but it can notbe garbage collected (each message has it' unique oid).This is the problem: The topics have no swapping mechanism so using a lot of different topics wont scale. Usually you should choose some well defined topics (typically 5-100, not more than say 1000). Each topic can hold any numbers of message instances, they will be swapped away, depending on the configuration. As you mentioned above - swapping does hard disk IO and is therefore not as performing as memory based messages - but it will prevent out of memory problems.- I have set some properties maxEntriesCache = 0, but it had no effectat publishing time. I have found some comments in the some of the code, saying first overload of queue is allowed.You can just switch off the CACHE and work directly with the persistent queue: StoragePlugin[CACHE][1.0]=${JdbcStorage[HSQLDatabaseEngine]} QueuePlugin[CACHE][1.0]=${JdbcStorage[HSQLDatabaseEngine]} Now no RAM queue is used as all, the 'CACHE' instantiates the JDBC queue implementation ... so no patching is needed.- I have patched CacheQueueInterceptorPlugin for me, so that it doesnotoverload the cache above the specified settings. I'm not sure theserveris behaving as stable as before with this settings, btw. - after the restart of the xmlBlaster server the heap was mostly free. So I suppose, there some other queues/caches were the first published message of a given oid is held in memory, but not preloaded at aserverrestart.We have one storage holding the message 'meat' (which is reference counted): StoragePlugin and several queues, which have a reference on the 'meat' only: history callback subject connection (on client side only)The "callback" queue exists once for each login session, it is the most important. The "history" queue is sort of a memory of already processed messages, and the "subject" queue is for the seldom use case of PtP messages to a login name destination.regards MarcelHere the overview of my little patch (between 2006-06-16 2101 JThpatchcomments): // .................... (cut for keeping the email short) package org.xmlBlaster.engine.msgstore.cache; public class PersistenceCachePlugin implements I_StoragePlugin, I_StorageProblemListener, I_Map, PersistenceCachePluginMBean { // .................... (cut for keeping the email short) private void assureTransientSpace(I_MapEntry mapEntry) throws XmlBlasterException { while (!spaceLeft(mapEntry, this.transientStore)) { // .................... (cut for keeping the email short) if (oldest == null) { if (log.isLoggable(Level.FINE)) log.fine("The RAM queue is full, new entry '" + mapEntry.getUniqueId() + "' seems to be the first and only one, so we accept it"); break; } // ++++ 2006-06-16 2101 JTh patch start if (oldest == mapEntry) { if (log.isLoggable(Level.FINE)) log.fine("2006-06-16 2101 JTh patch: The RAM queue will become full, new entry '" + mapEntry.getUniqueId() + "' seems to be the first and only one, westillremoved it, transientStore.numOfEntries now " + this.transientStore.getNumOfEntries()); } // ---- 2006-06-16 2101 JTh patch end if (log.isLoggable(Level.FINER)) log.finer("Swapping '" + oldest.getLogId() + "' to HD ..."); // .................... (cut for keeping the email short) } } public int put(I_MapEntry mapEntry) throws XmlBlasterException { // .................... (cut for keeping the email short) synchronized(this) { // .................... (cut for keeping the email short) // separate persistent from transient entries and store the persistents in persistence // .................... (cut for keepingtheemail short) assureTransientSpace(mapEntry);numTransientPut = this.transientStore.put(mapEntry); // ++++ 2006-06-16 2101 JTh patch startassureTransientSpace(mapEntry);// ---- 2006-06-16 2101 JTh patch end} // sync(this) // .................... (cut for keeping the email short) } }Regards, Johann Thomas-- OpenPGP: 0x224225EA (7454 D7BB A75D EA8E 5DFF EA4E 778E A85B 224225EA)