AbstractCachingProcessingPipeline: Two requests can deadlock each other
-----------------------------------------------------------------------
Key: COCOON-2173
URL: https://issues.apache.org/jira/browse/COCOON-2173
Project: Cocoon
Issue Type: Bug
Components: - Components: Sitemap
Affects Versions: 2.1.11, 2.1.10, 2.1.9, 2.2-dev (Current SVN)
Reporter: Alexander Daniel
Attachments: reproduceMultipleThreads.tar.gz
Two requests can deadlock each other when they depend on the same resources
which they acquire in a different order. I can reproduce that in Cocoon 2.1.11
and Cocoon 2.2-RC3-SNAPSHOT:
* request A: generating lock for 55933
* request B: generating lock for 58840
* request B: waiting for lock 55933 which is hold by request A
* request A: waiting for lock 58840 which is hold by request B
I can reproduce this behaviour with Apache Bench and following pipeline:
* terminal 1: Apache Bench request A (ab -k -n 10000 -c 25
http://localhost:8888/samples/reproduceMultipleThreads/productOfferForDevice/55933/)
* terminal 2: Apache Bench request B (ab -k -n 10000 -c 25
http://localhost:8888/samples/reproduceMultipleThreads/productOfferForDevice/58840/)
* terminal 3: touching the two data files every second to invalidate the cache
(while true; do echo -n "."; touch 55933.xml 58840.xml; sleep 1; done)
* pipeline:
<map:pipeline type="caching">
<map:match pattern="productOfferForDevice*/*/">
<map:generate src="cocoon:/exists/{2}.xml" label="a"/>
<map:transform type="xsltc" src="productOfferIncludeDevice.xsl" label="b">
<map:parameter name="noInc" value="{1}"/>
</map:transform>
<map:transform type="include" label="c"/>
<map:serialize type="xml"/>
</map:match>
<map:match pattern="exists/**">
<map:act type="resource-exists">
<map:parameter name="url" value="{1}" />
<map:generate src="{../1}" />
<map:serialize type="xml" />
</map:act>
<!-- not found -->
<map:generate src="dummy.xml" />
<map:serialize type="xml" />
</map:match>
</map:pipeline>
After some seconds the deadlock occurs ==>
* Apache Bench requests run into a timeout
* I can see following pipe locks in the default transient store:
PIPELOCK:PK_G-file-cocoon://samples/reproduceMultipleThreads/exists/55933.xml?pipelinehash=-910770960103935149_T-xsltc-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/productOfferIncludeDevice.xsl;noInc=_T-include-I_S-xml-1
(class: org.mortbay.util.ThreadPool$PoolThread)
PIPELOCK:PK_G-file-cocoon://samples/reproduceMultipleThreads/exists/58840.xml?pipelinehash=-4996088883111986478_T-xsltc-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/productOfferIncludeDevice.xsl;noInc=_T-include-I_S-xml-1
(class: org.mortbay.util.ThreadPool$PoolThread)
PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/55933.xml
(class: org.mortbay.util.ThreadPool$PoolThread)
PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/58840.xml
(class: org.mortbay.util.ThreadPool$PoolThread)
I added some logging to AbstractCachingProcessingPipeline.java which reconfirms
the explanations above:
INFO (2008-03-13) 13:50.16:072 [sitemap]
(/samples/reproduceMultipleThreads/productOfferForDevice/55933/)
PoolThread-47/AbstractCachingProcessingPipeline: generating lock
PIPELOCK:PK_G-file-cocoon://samples/reproduceMultipleThreads/exists/55933.xml?pipelinehash=-910770960103935149_T-xsltc-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/productOfferIncludeDevice.xsl;noInc=_T-include-I_S-xml-1
INFO (2008-03-13) 13:50.16:074 [sitemap]
(/samples/reproduceMultipleThreads/productOfferForDevice/55933/)
PoolThread-47/AbstractCachingProcessingPipeline: generating lock
PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/55933.xml
INFO (2008-03-13) 13:50.16:075 [sitemap]
(/samples/reproduceMultipleThreads/productOfferForDevice/58840/)
PoolThread-6/AbstractCachingProcessingPipeline: generating lock
PIPELOCK:PK_G-file-cocoon://samples/reproduceMultipleThreads/exists/58840.xml?pipelinehash=-4996088883111986478_T-xsltc-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/productOfferIncludeDevice.xsl;noInc=_T-include-I_S-xml-1
INFO (2008-03-13) 13:50.16:075 [sitemap]
(/samples/reproduceMultipleThreads/productOfferForDevice/58840/)
PoolThread-6/AbstractCachingProcessingPipeline: generating lock
PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/58840.xml
INFO (2008-03-13) 13:50.16:281 [sitemap]
(/samples/reproduceMultipleThreads/productOfferForDevice/58840/)
PoolThread-6/AbstractCachingProcessingPipeline: waiting for lock
PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/55933.xml
INFO (2008-03-13) 13:50.16:304 [sitemap]
(/samples/reproduceMultipleThreads/productOfferForDevice/55933/)
PoolThread-47/AbstractCachingProcessingPipeline: waiting for lock
PIPELOCK:PK_G-file-file:///Users/alex/dev/cocoon/cocoon-2.1.11/build/webapp/samples/reproduceMultipleThreads/58840.xml
Reproduce yourself with Cocoon 2.1.11:
* download and extract Cocoon 2.1.11
* cd $CocoonHome
* ./build.sh
* cd build/webapp/samples
* tar -xzf $DownloadFolder/reproduceMultipleThreads.tar.gz
* cd ../../..
* ./cocoon.sh
* open 3 terminals and cd into
$CocoonHome/build/webapp/samples/reproduceMultipleThreads in each
* dry run without invalidating the cache to see that everything is working:
- terminal 1: ./terminal1.sh
- terminal 2: ./terminal2.sh
* run with invalidating the cache every seconds:
- terminal 1: ./terminal1.sh
- terminal 2: ./terminal2.sh
- terminal 3: ./terminal3.sh
* When Apache Bench has run into a timeout you can view the pipelocks with
http://localhost:8888/samples/reproduceMultipleThreads/pipelocks
Reproduce yourself with Cocoon 2.2RC3-SNAPSHOT:
* svn checkout http://svn.apache.org/repos/asf/cocoon/trunk
* mvn clean install
* ./cocoon.sh
* cd core/cocoon-webapp/target/work/blocks/cocoon-core-main-sample
* tar -xzf $DownloadFolder/reproduceMultipleThreads2.2RC3-SNAPSHOT.tar.gz
* open 3 terminals and cd into
$CocoonTrunk/core/cocoon-webapp/target/work/blocks/cocoon-core-main-sample/reproduceMultipleThreads/
in each
* dry run without invalidating the cache to see that everything is working:
- terminal 1: ./terminal1.sh
- terminal 2: ./terminal2.sh
* run with invalidating the cache every second:
- terminal 1: ./terminal1.sh
- terminal 2: ./terminal2.sh
- terminal 3: ./terminal3.sh
* When Apache Bench has run into a timeout you can view the pipelocks with
http://localhost:8888/samples/core/reproduceMultipleThreads/pipelocks
Since we are currently facing this issue on our production servers I would
appreciate a hint on how to fix that issue in a good way.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
