AbstractCachingProcessingPipeline: Two requests can deadlock each other -----------------------------------------------------------------------
Key: COCOON-2173 URL: https://issues.apache.org/jira/browse/COCOON-2173 Project: Cocoon Issue Type: Bug Components: - Components: Sitemap Affects Versions: 2.1.11, 2.1.10, 2.1.9, 2.2-dev (Current SVN) Reporter: Alexander Daniel Attachments: reproduceMultipleThreads.tar.gz Two requests can deadlock each other when they depend on the same resources which they acquire in a different order. I can reproduce that in Cocoon 2.1.11 and Cocoon 2.2-RC3-SNAPSHOT: * request A: generating lock for 55933 * request B: generating lock for 58840 * request B: waiting for lock 55933 which is hold by request A * request A: waiting for lock 58840 which is hold by request B I can reproduce this behaviour with Apache Bench and following pipeline: * terminal 1: Apache Bench request A (ab -k -n 10000 -c 25 http://localhost:8888/samples/reproduceMultipleThreads/productOfferForDevice/55933/) * terminal 2: Apache Bench request B (ab -k -n 10000 -c 25 http://localhost:8888/samples/reproduceMultipleThreads/productOfferForDevice/58840/) * terminal 3: touching the two data files every second to invalidate the cache (while true; do echo -n "."; touch 55933.xml 58840.xml; sleep 1; done) * pipeline: <map:pipeline type="caching"> <map:match pattern="productOfferForDevice*/*/"> <map:generate src="cocoon:/exists/{2}.xml" label="a"/> <map:transform type="xsltc" src="productOfferIncludeDevice.xsl" label="b"> <map:parameter name="noInc" value="{1}"/> </map:transform> <map:transform type="include" label="c"/> <map:serialize type="xml"/> </map:match> <map:match pattern="exists/**"> <map:act type="resource-exists"> <map:parameter name="url" value="{1}" /> <map:generate src="{../1}" /> <map:serialize type="xml" /> </map:act> <!-- not found --> <map:generate src="dummy.xml" /> <map:serialize type="xml" /> </map:match> </map:pipeline> After some seconds the deadlock occurs ==> * Apache Bench requests run into a timeout * I can see following pipe locks in the default transient store: PIPELOCK:PK_G-file-cocoon://samples/reproduceMultipleThreads/exists/55933.xml?pipelinehash=-910770960103935149_T-xsltc-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/productOfferIncludeDevice.xsl;noInc=_T-include-I_S-xml-1 (class: org.mortbay.util.ThreadPool$PoolThread) PIPELOCK:PK_G-file-cocoon://samples/reproduceMultipleThreads/exists/58840.xml?pipelinehash=-4996088883111986478_T-xsltc-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/productOfferIncludeDevice.xsl;noInc=_T-include-I_S-xml-1 (class: org.mortbay.util.ThreadPool$PoolThread) PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/55933.xml (class: org.mortbay.util.ThreadPool$PoolThread) PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/58840.xml (class: org.mortbay.util.ThreadPool$PoolThread) I added some logging to AbstractCachingProcessingPipeline.java which reconfirms the explanations above: INFO (2008-03-13) 13:50.16:072 [sitemap] (/samples/reproduceMultipleThreads/productOfferForDevice/55933/) PoolThread-47/AbstractCachingProcessingPipeline: generating lock PIPELOCK:PK_G-file-cocoon://samples/reproduceMultipleThreads/exists/55933.xml?pipelinehash=-910770960103935149_T-xsltc-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/productOfferIncludeDevice.xsl;noInc=_T-include-I_S-xml-1 INFO (2008-03-13) 13:50.16:074 [sitemap] (/samples/reproduceMultipleThreads/productOfferForDevice/55933/) PoolThread-47/AbstractCachingProcessingPipeline: generating lock PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/55933.xml INFO (2008-03-13) 13:50.16:075 [sitemap] (/samples/reproduceMultipleThreads/productOfferForDevice/58840/) PoolThread-6/AbstractCachingProcessingPipeline: generating lock PIPELOCK:PK_G-file-cocoon://samples/reproduceMultipleThreads/exists/58840.xml?pipelinehash=-4996088883111986478_T-xsltc-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/productOfferIncludeDevice.xsl;noInc=_T-include-I_S-xml-1 INFO (2008-03-13) 13:50.16:075 [sitemap] (/samples/reproduceMultipleThreads/productOfferForDevice/58840/) PoolThread-6/AbstractCachingProcessingPipeline: generating lock PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/58840.xml INFO (2008-03-13) 13:50.16:281 [sitemap] (/samples/reproduceMultipleThreads/productOfferForDevice/58840/) PoolThread-6/AbstractCachingProcessingPipeline: waiting for lock PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/55933.xml INFO (2008-03-13) 13:50.16:304 [sitemap] (/samples/reproduceMultipleThreads/productOfferForDevice/55933/) PoolThread-47/AbstractCachingProcessingPipeline: waiting for lock PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/58840.xml Reproduce yourself with Cocoon 2.1.11: * download and extract Cocoon 2.1.11 * cd $CocoonHome * ./build.sh * cd build/webapp/samples * tar -xzf $DownloadFolder/reproduceMultipleThreads.tar.gz * cd ../../.. * ./cocoon.sh * open 3 terminals and cd into $CocoonHome/build/webapp/samples/reproduceMultipleThreads in each * dry run without invalidating the cache to see that everything is working: - terminal 1: ./terminal1.sh - terminal 2: ./terminal2.sh * run with invalidating the cache every seconds: - terminal 1: ./terminal1.sh - terminal 2: ./terminal2.sh - terminal 3: ./terminal3.sh * When Apache Bench has run into a timeout you can view the pipelocks with http://localhost:8888/samples/reproduceMultipleThreads/pipelocks Reproduce yourself with Cocoon 2.2RC3-SNAPSHOT: * svn checkout http://svn.apache.org/repos/asf/cocoon/trunk * mvn clean install * ./cocoon.sh * cd core/cocoon-webapp/target/work/blocks/cocoon-core-main-sample * tar -xzf $DownloadFolder/reproduceMultipleThreads2.2RC3-SNAPSHOT.tar.gz * open 3 terminals and cd into $CocoonTrunk/core/cocoon-webapp/target/work/blocks/cocoon-core-main-sample/reproduceMultipleThreads/ in each * dry run without invalidating the cache to see that everything is working: - terminal 1: ./terminal1.sh - terminal 2: ./terminal2.sh * run with invalidating the cache every second: - terminal 1: ./terminal1.sh - terminal 2: ./terminal2.sh - terminal 3: ./terminal3.sh * When Apache Bench has run into a timeout you can view the pipelocks with http://localhost:8888/samples/core/reproduceMultipleThreads/pipelocks Since we are currently facing this issue on our production servers I would appreciate a hint on how to fix that issue in a good way. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.