Bugs item #664459, was opened at 2003-01-08 16:21 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=376685&aid=664459&group_id=22866
Category: JBossMQ Group: v3.0 Rabbit Hole >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Ryan Harris (rpharris) Assigned to: Adrian Brock (ejort) Summary: JMS Message lost due to synchronization in MessageReference Initial Comment: Bug and fix submitted by Ryan Harris of Sterling Commerce, Inc. http://www.sterlingcommerce.com Due to a synchronization error in a MessageReference object, the file reference to a Message object is deleted while no hard reference exists. However, the system is not finished processing the Message and when it attempts to retrieve its hard reference from the file reference on the disk, it fails with a FileNotFound exception. A fix is included and a is attached. It will be submitted in the patch section as well. The problem occurs when a series of the following three events occurs in a MessageReference object. This scenario only occurs when the heap size has passed the HighMemoryMark of the Cachemanager AND the garbage collector happens to run just after the validateSoftReferenceDepth step below. Original MessageReference.invalidate: public void invalidate() throws JMSException { clear(); // Clear out the disk copy } This is the original course of events that occurs: ************************************** validateSoftRefernceDepth (deleting our hard reference and creating a soft reference) **context switch** GC Runs and collects our hard reference. **context switch** invalidate and clear are called (we have no hard reference at this point, deleting our disk reference) **context switch** makeHard is called (we have no hard reference nor a file on the disk) Explosion. ************************************ In order to fix this error, it is required that we check within the invalidate method if we actually have a hard reference to the object, before we blow away the file reference on the disk. Modified MessageReference.invalidate: public void invalidate() throws JMSException { synchronized(messageCache) { if(hardReference != null) clear(); // Clear out the disk copy } } This forces the events into the following pattern: *************************************** validateSoftRefernceDepth (deleting our hard reference) **context switch** GC Runs and collects our hard reference. **context switch** invalidate (fails to delete disk reference because we have no hard reference) **context switch** makeHard is called (we have no hard reference but thefile is still on the disk) Everything is happy. **************************************** Operating Systems: Linux, Windows2k, HPUX, Solaris, AIX, Others most likely JDK: 1.3.1 Stack Trace: [WARN,SpyConnectionConsumer] Connection consumer closing due to error in listening thread. org.jboss.mq.SpyJMSException: Could not load message from secondary storage: ; - nested throwable: (java.io.FileNotFoundException: C:\si_1230\SterlingIntegrator\jboss\jboss\tmp\jbossmq\Message-5437 (The system cannot find the file specified)) at org.jboss.mq.pm.file.CacheStore.loadFromStorage(CacheStore.java:61) at org.jboss.mq.server.MessageCache.loadFromStorage(MessageCache.java:246) at org.jboss.mq.server.MessageReference.makeHard(MessageReference.java:207) at org.jboss.mq.server.MessageReference.getMessage(MessageReference.java:93 ) at org.jboss.mq.server.BasicQueue.setupMessageAcknowledgement(BasicQueue.ja va:380) at org.jboss.mq.server.BasicQueue.receive(BasicQueue.java:238) at org.jboss.mq.server.JMSQueue.receive(JMSQueue.java:122) at org.jboss.mq.server.ClientConsumer.receive(ClientConsumer.java:225) at org.jboss.mq.server.JMSDestinationManager.receive(JMSDestinationManager. java:669) at org.jboss.mq.server.TracingInterceptor.receive(TracingInterceptor.java:4 33) at org.jboss.mq.server.JMSServerInvoker.receive(JMSServerInvoker.java:227) at org.jboss.mq.il.jvm.JVMServerIL.receive(JVMServerIL.java:245) at org.jboss.mq.Connection.receive(Connection.java:1046) at org.jboss.mq.SpyConnectionConsumer.run(SpyConnectionConsumer.java:183) at java.lang.Thread.run(Thread.java:479) java.io.FileNotFoundException: C:\si_1230\SterlingIntegrator\jboss\jboss\tmp\jbossmq\Message-5437 (The system cannot find the file specified) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:59) at java.io.FileInputStream.<init>(FileInputStream.java:90) at org.jboss.mq.pm.file.CacheStore.loadFromStorage(CacheStore.java:54) at org.jboss.mq.server.MessageCache.loadFromStorage(MessageCache.java:246) at org.jboss.mq.server.MessageReference.makeHard(MessageReference.java:207) at org.jboss.mq.server.MessageReference.getMessage(MessageReference.java:93 ) at org.jboss.mq.server.BasicQueue.setupMessageAcknowledgement(BasicQueue.ja va:380) at org.jboss.mq.server.BasicQueue.receive(BasicQueue.java:238) at org.jboss.mq.server.JMSQueue.receive(JMSQueue.java:122) at org.jboss.mq.server.ClientConsumer.receive(ClientConsumer.java:225) at org.jboss.mq.server.JMSDestinationManager.receive(JMSDestinationManager. java:669) at org.jboss.mq.server.TracingInterceptor.receive(TracingInterceptor.java:4 33) at org.jboss.mq.server.JMSServerInvoker.receive(JMSServerInvoker.java:227) at org.jboss.mq.il.jvm.JVMServerIL.receive(JVMServerIL.java:245) at org.jboss.mq.Connection.receive(Connection.java:1046) at org.jboss.mq.SpyConnectionConsumer.run(SpyConnectionConsumer.java:183) at java.lang.Thread.run(Thread.java:479) [INFO,Default] Mon Dec 30 17:56:49 EST 2002 ->connect() ---------------------------------------------------------------------- >Comment By: Adrian Brock (ejort) Date: 2003-02-10 02:15 Message: Logged In: YES user_id=9459 The cache changes have now been backported to 3.0 Regards, Adrian ---------------------------------------------------------------------- Comment By: Steven Ostrowski (alchemista) Date: 2003-02-07 05:00 Message: Logged In: YES user_id=552047 I tried 3.2.0RC1 because JBossMQ was dying (becoming unresponsive without recovery) under peak load. As of 3.2.0RC1, it is able to handle peak load. Performance drops as expected, but it stays responsive, and it recovers once the load is reduced. I'd vote for backporting whatever changes to the 3.0.x versions. Thanks ---------------------------------------------------------------------- Comment By: Adrian Brock (ejort) Date: 2003-01-16 09:31 Message: Logged In: YES user_id=9459 The JBoss4 cache changes have been backported for the 3.2.0RC1release. If there is some positive feedback, it can also go into 3.0.x Regards, Adrian ---------------------------------------------------------------------- Comment By: Adrian Brock (ejort) Date: 2003-01-08 17:11 Message: Logged In: YES user_id=9459 This patch won't work, it fails to invalidate messages that are not in memory. The jboss4 implementation solves this problem more thoroughly. public void invalidate() throws JMSException { boolean trace = log.isTraceEnabled(); if( trace ) log.trace("invalidate lock aquire " + toString()); synchronized (this) { if (stored == STORED) { if (hardReference == null) { makeHard(); messageCache.messageReferenceUsedEvent(this, false, trace); } messageCache.removeFromStorage(this); } if( trace ) log.trace("invalidate lock relased " + toString()); } } I'm looking for somebody else to test the new cache implemention in JBoss4 before backporting to JBoss3 I spent two weeks testing it, but I doubt I thought of everything :-) It performs much better under load because there is no global lock on the message cache and some potential leaks under tight memory like the one you describe have been fixed. It also correctly invalidates persistent messages so that the redelivered flag survives a reboot of the server. Regards, Adrian ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=376685&aid=664459&group_id=22866 ------------------------------------------------------- This SF.NET email is sponsored by: SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See! http://www.vasoftware.com _______________________________________________ Jboss-development mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/jboss-development
