Hi Leander & Werner, I work with Simon here at Multicom and I just wanted to share with you our experience of HashMap's in a highly concurrent environment. I haven't ever attempted to replicate the following problem with a unit test and so it is only based on empirical evidence of running our web service under Java 5 on both linux (x86) and solaris (sun fire T1000 upto 32 simultaneous threads).
Very occasionally we have seen concurrent HashMap accesses causing 100% cpu usage in an endless tight loop. It is exrememly rare (before we worked out what it was it might have happened once a or twice a month across 6 T1000 appservers), and I'm not sure exactly how it happens, but it does happen and it can bring our application servers to their knees. As such, we have a strict policy of checking HashMap usage in our own code here at Multicom and we have even had to produce our own patched versions of a few open source tools in order to remove concurrent HashMap usage. We are a massive user of Castor as it probably accounts for 20% to 30% of our total CPU usage (we pretty much act as an ecommerce market place connecting many different vendors to many different vendee's via an XML based hub) and so we would love to take advantage of ClassDescriptor caching that the Resolver provides. However, we are in a sitatuation where many different (>30) class heirarchies will require marshalling and unmarshalling (each vendor will have their own message set and their own Castor beans Jar) and we cannot know at initialisation time which class heirarchies will be required during the lifetime of the ClassDescriptorResolver. Therefore it is inevitable that the Maps holding the class descriptors may well continue to be updated throught the lifetime of the Resolver, and therefore as I see it we have 2 options: A) Implement a Thread safe pool of Resolvers (one pool per class heirarchy to save memory) - Simon has already done this and it works fine and will be safe to use. B) Patch Castor to use Hashtable instead of HashMap (or Collections.synchronizeMap (hashmap) if null values are required) The advantage of B) is reduced memory footprint and it works out of the box. As I see it, the very slight performence hit on using a synchronized Map instead of an unsynchronized one is vastly overshadowed by the inability of Castor to bring our servers to their knees. :-) So... what do you guys think? Cheers, Adam Chesney Senior Architect Multicom Products Ltd ===================================================== Tel: (+44) 117 908 1254 (Direct) Mobile (+44) 7780 962 961 email: [EMAIL PROTECTED] This communication is intended solely for the addressee and is confidential. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Although this e-mail and any attachments are believed to be free of any virus, or any other defect which might affect any computer or IT system into which they are received and opened, it is the responsibility of the recipient to ensure that they are virus free and no responsibility is accepted by Multicom Products Limited for any loss or damage arising in any way from receipt or use thereof. ----- Original Message ----- From: "M.-Leander Reimer" <[EMAIL PROTECTED]> To: <[email protected]> Sent: Friday, March 16, 2007 11:25 AM Subject: Re: AW: AW: [castor-user] [XML] XMLClassDescriptorResolver XMLFieldDescriptorImpl cache possible memory leak > Hi Simon and Werner, > > after running Simon's tests and seeing the memory results I started to > write for my own test code using JUnitPerf to load test my stuff using > multiple concurrent threads with several iterations. > > Everything runs fine. The memory stays constant and also the time taken > to unmarshall/marshall an XML structure. No concurrency problems detected. > > HOWEVER, the main difference to Simon's code is, that I initialize the > XMLClassdescriptorResolver using a mapping and I still use Castor 1.0.5! > > > After switching to Castor 1.1, all of the tests that use IDs in the > mapping do not run anymore. (stack trace at the end of my mail) > > After looking at the release notes I saw that there have been changes to > the ID resolution. But I guess that's a different issue and I will > investigate this a little more. > > > Regards, > Leander > > > ValidationException: The following exception occured while validating > field: Locales of class: > de.adaptions.adgen.common.fonts.localemapping.LocaleMapping: The object > associated with IDREF > "[EMAIL PROTECTED]" of type class > de.adaptions.adgen.common.fonts.localemapping.Locale has no ID!; > - location of error: XPATH: /locales > The object associated with IDREF > "[EMAIL PROTECTED]" of type class > de.adaptions.adgen.common.fonts.localemapping.Locale has no ID! > at org.exolab.castor.xml.FieldValidator.validate(FieldValidator.java:270) > at > org.exolab.castor.xml.util.XMLClassDescriptorImpl.validate(XMLClassDescripto rImpl.java:931) > at org.exolab.castor.xml.Validator.validate(Validator.java:124) > at > org.exolab.castor.xml.UnmarshalHandler.endElement(UnmarshalHandler.java:913) > at > org.exolab.castor.xml.UnmarshalHandler.endElement(UnmarshalHandler.java:1108 ) > at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source) > at > org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown > Source) > at > org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatc her.dispatch(Unknown > Source) > at > org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown > Source) > at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) > at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) > at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) > at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) > at org.exolab.castor.xml.Unmarshaller.unmarshal(Unmarshaller.java:726) > at > de.adaptions.adgen.common.util.xml.LocaleMappingSerializer.deserialize(Local eMappingSerializer.java:133) > at > de.adaptions.adgen.common.util.xml.LocaleMappingSerializerTest.testPerforman ce(LocaleMappingSerializerTest.java:56) > at sun.reflect.GeneratedMethodAccessor177.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at junit.framework.TestCase.runTest(TestCase.java:154) > at junit.framework.TestCase.runBare(TestCase.java:127) > at junit.framework.TestResult$1.protect(TestResult.java:106) > at junit.framework.TestResult.runProtected(TestResult.java:124) > at junit.framework.TestResult.run(TestResult.java:109) > at junit.framework.TestCase.run(TestCase.java:118) > at junit.framework.TestSuite.runTest(TestSuite.java:208) > at junit.framework.TestSuite.run(TestSuite.java:203) > at com.clarkware.junitperf.TestFactory.run(TestFactory.java:83) > at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) > at junit.extensions.TestDecorator.run(TestDecorator.java:28) > at > com.clarkware.junitperf.TimedTest.runUntilTestCompletion(TimedTest.java:154) > at com.clarkware.junitperf.TimedTest.run(TimedTest.java:138) > at junit.extensions.TestDecorator.basicRun(TestDecorator.java:22) > at junit.extensions.TestDecorator.run(TestDecorator.java:28) > at junit.extensions.RepeatedTest.run(RepeatedTest.java:25) > at > com.clarkware.junitperf.ThreadedTest$TestRunner.run(ThreadedTest.java:75) > at java.lang.Thread.run(Unknown Source) > Caused by: ValidationException: The object associated with IDREF > "[EMAIL PROTECTED]" of type class > de.adaptions.adgen.common.fonts.localemapping.Locale has no ID!; > - location of error: XPATH: /locale > at > org.exolab.castor.xml.validators.IdRefValidator.validate(IdRefValidator.java :70) > at org.exolab.castor.xml.FieldValidator.validate(FieldValidator.java:207) > at > org.exolab.castor.xml.util.XMLClassDescriptorImpl.validate(XMLClassDescripto rImpl.java:938) > at org.exolab.castor.xml.Validator.validate(Validator.java:124) > at > org.exolab.castor.xml.FieldValidator.validateInstance(FieldValidator.java:31 8) > at org.exolab.castor.xml.FieldValidator.validate(FieldValidator.java:248) > ... 36 more > > --------------------------------------------------------------------- > To unsubscribe from this list please visit: > > http://xircles.codehaus.org/manage_email > --------------------------------------------------------------------- To unsubscribe from this list please visit: http://xircles.codehaus.org/manage_email
