Re: [xmlblaster] WG: Help! OutOfMemory by publishing lot big topics

[Marcel Ruff]

Thomas, Johann wrote:
(after 3h still not seen in mailing list, resending)

-----Original-----

Hi,
 
I have chosen xmlBlaster to send messages (out of a Java Servlet) with
content of often few 10 or 100 Kbyte reliably, but not to often.

But now I ran into OutOfMemoryError in the server, by doing kind of a
dump through this interface. The problem seems to be that the published
messages are not swapped out to persistence, filling up the heap. 

I start with the question: 
- Should I try to keep the RAM caches empty by some settings and patches
or will it probably not work at last?

What I have done in this area:
I read some mailing list messages saying that xmlBlaster is not that
good for persisting lot of data :-( I started my testing with publishing
some 10MByte in 512KByte pieces and found:
- it helps switching away from HSQLDB to Oracle (OT: I used a 10g1 JDBC
against the 9g2 database)
- the content dumped is cached "somewhere" in the heap, but it can not
be garbage collected (each message has it' unique oid).
  
This is the problem: The topics have no swapping mechanism so using a 
lot of different topics wont
scale. Usually you should choose some well defined topics (typically 
5-100, not more than say 1000).

Each topic can hold any numbers of message instances, they will be 
swapped away, depending on the configuration.
As you mentioned above - swapping does hard disk IO and is therefore not 
as performing as memory based
messages - but it will prevent out of memory problems.

- I have set some properties maxEntriesCache = 0, but it had no effect
at publishing time. I have found some comments in the some of the code,
saying first overload of queue is allowed. 
  
You can just switch off the CACHE and work directly with the persistent 
queue:

StoragePlugin[CACHE][1.0]=${JdbcStorage[HSQLDatabaseEngine]}
QueuePlugin[CACHE][1.0]=${JdbcStorage[HSQLDatabaseEngine]}

Now no RAM queue is used as all, the 'CACHE' instantiates the JDBC queue 
implementation ...
so no patching is needed.


- I have patched CacheQueueInterceptorPlugin for me, so that it does not
overload the cache above the specified settings. I'm not sure the server
is behaving as stable as before with this settings, btw.
- after the restart of the xmlBlaster server the heap was mostly free.

So I suppose, there some other queues/caches were the first published
message of a given oid is held in memory, but not preloaded at a server
restart.
  
We have one storage holding the message 'meat' (which is reference counted):
 StoragePlugin
and several queues, which have a reference on the 'meat' only:
 history
 callback
 subject
 connection (on client side only)

The "callback" queue exists once for each login session, it is the most 
important.
The "history" queue is sort of a memory of already processed messages, 
and the
"subject" queue is for the seldom use case of PtP messages to a login 
name destination.

regards
Marcel


Here the overview of my little patch (between 2006-06-16 2101 JTh patch
comments):

// .................... (cut for keeping the email short) package
org.xmlBlaster.engine.msgstore.cache;
public class PersistenceCachePlugin implements I_StoragePlugin,
I_StorageProblemListener, I_Map, PersistenceCachePluginMBean {

// .................... (cut for keeping the email short)

   private void assureTransientSpace(I_MapEntry mapEntry) throws
XmlBlasterException {

      while (!spaceLeft(mapEntry, this.transientStore)) {

// .................... (cut for keeping the email short)

         if (oldest == null) {
            if (log.isLoggable(Level.FINE)) log.fine("The RAM queue is
full, new entry '" + mapEntry.getUniqueId() + "' seems to be the first
and only one, so we accept it");
            break;
         }
         // ++++ 2006-06-16 2101 JTh patch start
         if (oldest == mapEntry) {
            if (log.isLoggable(Level.FINE)) log.fine("2006-06-16 2101
JTh patch: The RAM queue will become full, new entry '" +
mapEntry.getUniqueId() + "' seems to be the first and only one, we still
removed it, transientStore.numOfEntries now " +
this.transientStore.getNumOfEntries());
         }
         // ---- 2006-06-16 2101 JTh patch end
         if (log.isLoggable(Level.FINER)) log.finer("Swapping '" +
oldest.getLogId() + "' to HD ...");

// .................... (cut for keeping the email short)

      }
   }

   public int put(I_MapEntry mapEntry) throws XmlBlasterException {

// .................... (cut for keeping the email short)

      synchronized(this) {

// .................... (cut for keeping the email short)

         // separate persistent from transient entries and store the
persistents in persistence // .................... (cut for keeping the
email short)


         assureTransientSpace(mapEntry);
         
         numTransientPut = this.transientStore.put(mapEntry);
         
         // ++++ 2006-06-16 2101 JTh patch start
         assureTransientSpace(mapEntry);
         
         // ---- 2006-06-16 2101 JTh patch end
      } // sync(this)
// .................... (cut for keeping the email short)
   }
}
 
Regards,
 
Johann Thomas

--
OpenPGP: 0x224225EA (7454 D7BB A75D EA8E 5DFF EA4E 778E A85B 2242 25EA)