RE: XSLT Thread Safety Issue
The route endpoint was
xslt:classpath:xslt/orch-encryptionkey-flowRemoveNamespace.xsl. You may have
been asking for the actual XSLT which I included but it looks like it got
pruned by the listserver. I will include it here xml encoded:
lt;xsl:stylesheet version=quot;1.0quot;
xmlns:xsl=quot;http://www.w3.org/1999/XSL/Transformquot;gt;
lt;xsl:output standalone=quot;yesquot;
omit-xml-declaration=quot;yesquot; /gt;
lt;xsl:template match=quot;processing-instruction()quot;gt;
lt;xsl:copygt;lt;xsl:apply-templates/gt;lt;/xsl:copygt;
lt;/xsl:templategt;
lt;xsl:template match=quot;*quot;gt;
lt;xsl:element
name=quot;{local-name()}quot;gt;lt;xsl:apply-templates
select=quot;@*|node()quot;/gt;lt;/xsl:elementgt;
lt;/xsl:templategt;
lt;xsl:template match=quot;@*quot;gt;
lt;xsl:attribute name=quot;{local-name()}quot;gt;lt;xsl:value-of
select=quot;.quot;/gt;lt;/xsl:attributegt;
lt;/xsl:templategt;
lt;/xsl:stylesheetgt;
I don't believe we are using either of those options, a full text search of our
entire codebase didn't yield any references.
Sam
-Original Message-
From: Willem Jiang [mailto:willem.ji...@gmail.com]
Sent: Monday, October 13, 2014 11:07 PM
To: users@camel.apache.org
Subject: Re: XSLT Thread Safety Issue
Can I have a look at you XSLT endpoint uri?
Did you use the options of transformerCacheSize or contentCache?
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English) http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On October 14, 2014 at 9:45:39 AM, Hendley, Sam (sam.hend...@sensus.com) wrote:
We are having a very strange (and intermittent) issue with some of our camel
routes.
We are in the process of upgrading to a more up-to-date version but
are still on camel 2.10.7 and running on java 1.6. I have looked
through the forums and tickets but haven't found anything that looks like it
could cause this issue.
What appears to be occurring is this:
* An XML document is received from a JMS queue (as a text message)
o The xml is correctly formatted and is what we expect to see
* It is then routed through an XSLT transformation (to strip off
excess namespacing)
* 99% of the time this works, but very occasionally we get Xpath
errors when we go onto the next step
* We were able to do a heap dump of a system that was having this
problem, when we dug through the dump we were able to analyze an
instance of the failure (same exchangeId)
o It turns out that when the issue occurs the enclosing tags (document
root) have disappeared rendering the document un-xpathable
Since this works nearly all of the time and there is nothing special
about the failing documents we suspect it could be a threading or race
condition in the XSLT processing.
Are there any known issues that might be related? This route is
running in a relatively large and busy camel application that makes
heavy use of XSLT transforms. We believe similar issues have occurred
in other instances but haven't debugged to this level so that is just a hunch.
I have included some details below, let me know if there is more I
should include. I still have the heap/thread dump if there is anything else
that might be worth digging into.
Thanks for any help or advise,
Sam Hendley
Jms Timeout message (notice it has xml header and wrapping of
EncryptionKeyFlow) [cid:image001.png@01CFD7FC.D4F23FD0]
2193-49984781-keymanager-1411163761368-126275
orch.encryptionkey.encryptionkeyTimeout.queue
FAIL
timeout
Request timed out while waiting for response(s).
Camel Body (stripped off top and bottom elements)
[cid:image002.png@01CFD7FC.D4F23FD0]
2193-49984781-keymanager-1411163761368-126275
orch.encryptionkey.encryptionkeyTimeout.queue
...
FAIL
timeout
Request timed out while waiting for response(s).
Route:
from(JMS_encryptionkeyTimeout).routeId(JMS_encryptionkeyTimeout)
.log(LoggingLevel.INFO, Timed out waiting for encryptionkey response)
.choice()
.when(isInBodyNullOrEmpty)
.to(SEDA_encryptionkeyOrchDead)
.otherwise()
.to(DIRECT_processBackToClient)
.end();
// Called from several places - when a response needs to be sent back to the
client
from(DIRECT_processBackToClient).routeId(DIRECT_processBackToClient)
.setHeader(HDR_JMS_CORRELATION_ID).xpath(//*[local-name()='transactionId'],
String.class)
.to(XSLT_toAPIResponsePreprocessing)
.choice()
// New response type may be added later
.when().xpath(not(/*[local-name()='responseQueue']))
.log(LoggingLevel.ERROR, Unable to format encryptionkey response)
.to(SEDA_encryptionkeyOrchDead)
.otherwise()
.recipientList().simple(JMS_RESPONSE_QUEUE)
.end();
XSLT that is probably at fault:
XSLT Thread Safety Issue
We are having a very strange (and intermittent) issue with some of our camel
routes.
We are in the process of upgrading to a more up-to-date version but are still
on camel 2.10.7 and running on java 1.6. I have looked through the forums and
tickets but haven't found anything that looks like it could cause this issue.
What appears to be occurring is this:
* An XML document is received from a JMS queue (as a text message)
o The xml is correctly formatted and is what we expect to see
* It is then routed through an XSLT transformation (to strip off excess
namespacing)
* 99% of the time this works, but very occasionally we get Xpath errors
when we go onto the next step
* We were able to do a heap dump of a system that was having this
problem, when we dug through the dump we were able to analyze an instance of
the failure (same exchangeId)
o It turns out that when the issue occurs the enclosing tags (document root)
have disappeared rendering the document un-xpathable
Since this works nearly all of the time and there is nothing special about the
failing documents we suspect it could be a threading or race condition in the
XSLT processing. Are there any known issues that might be related? This route
is running in a relatively large and busy camel application that makes heavy
use of XSLT transforms. We believe similar issues have occurred in other
instances but haven't debugged to this level so that is just a hunch.
I have included some details below, let me know if there is more I should
include. I still have the heap/thread dump if there is anything else that might
be worth digging into.
Thanks for any help or advise,
Sam Hendley
Jms Timeout message (notice it has xml header and wrapping of EncryptionKeyFlow)
[cid:image001.png@01CFD7FC.D4F23FD0]
?xml version=1.0 encoding=UTF-8?
EncryptionKeyFlow xmlns=http://www.example.com/messages;
transactionId2193-49984781-keymanager-1411163761368-126275/transactionId
responseQueueorch.encryptionkey.encryptionkeyTimeout.queue/responseQueue
statusFAIL/status
substatustimeout/substatus
statusMessagesRequest timed out while waiting for
response(s)./statusMessages
/EncryptionKeyFlow
Camel Body (stripped off top and bottom elements)
[cid:image002.png@01CFD7FC.D4F23FD0]
transactionId2193-49984781-keymanager-1411163761368-126275/transactionId
responseQueueorch.encryptionkey.encryptionkeyTimeout.queue/responseQueue
...
statusFAIL/status
substatustimeout/substatus
statusMessagesRequest timed out while waiting for
response(s)./statusMessages
Route:
from(JMS_encryptionkeyTimeout).routeId(JMS_encryptionkeyTimeout)
.log(LoggingLevel.INFO, Timed out waiting for encryptionkey response)
.choice()
.when(isInBodyNullOrEmpty)
.to(SEDA_encryptionkeyOrchDead)
.otherwise()
.to(DIRECT_processBackToClient)
.end();
// Called from several places - when a response needs to be sent back to the
client
from(DIRECT_processBackToClient).routeId(DIRECT_processBackToClient)
.setHeader(HDR_JMS_CORRELATION_ID).xpath(//*[local-name()='transactionId'],
String.class)
.to(XSLT_toAPIResponsePreprocessing)
.choice()
// New response type may be added later
.when().xpath(not(/*[local-name()='responseQueue']))
.log(LoggingLevel.ERROR, Unable to format encryptionkey
response)
.to(SEDA_encryptionkeyOrchDead)
.otherwise()
.recipientList().simple(JMS_RESPONSE_QUEUE)
.end();
XSLT that is probably at fault:
xsl:stylesheet version=1.0 xmlns:xsl=http://www.w3.org/1999/XSL/Transform;
xsl:output standalone=yes omit-xml-declaration=yes /
xsl:template match=processing-instruction()
xsl:copyxsl:apply-templates//xsl:copy
/xsl:template
xsl:template match=*
xsl:element name={local-name()}xsl:apply-templates
select=@*|node()//xsl:element
/xsl:template
xsl:template match=@*
xsl:attribute name={local-name()}xsl:value-of
select=.//xsl:attribute
/xsl:template
/xsl:stylesheet
RE: Failover JMS and Camel DefaultTimeoutManager failures
Hi Christian, thanks for your interest. As I said the size of the application is very large, we have around 150 route builders spread over 5 or so processes. I can't easily boil down our issue to a few configurations or route builders. I was just trying to see if issues like this have been encountered before. I am trying to make a case for upgrading ActiveMQ and Camel but would like to know if the camel community believes that will clear up this issue. Thanks Sam -Original Message- From: Christian Posta [mailto:christian.po...@gmail.com] Sent: Thursday, September 26, 2013 11:07 AM To: users@camel.apache.org Subject: Re: Failover JMS and Camel DefaultTimeoutManager failures Post your route, your configs, and a description of what it does, who are its clients, etc. Also link to the bug in the current code. On Thu, Sep 26, 2013 at 8:02 AM, Hendley, Sam sam.hend...@sensus.comwrote: Hello all. I am working with a large application written in camel. We have recently started noticing (it may have been occurring for a while) that in some cases a sort of death spiral occurs. Out setup: Camel 2.10.1 (unfortunately we can't upgrade) ActiveMQ 5.6.0 (unfortunately we can't upgrade) As far as I can tell this is what is occurring: · Something puts excess load on the system and slows down our application · Some ReplyManagerTimeoutChecks start failing · It starts doing excessive amounts of work dealing with timeouts which delays the processing of the work causing more timeouts o Using a sampler it appears most the time is spent in DefaultTimeoutMap.purge() § There is a .size() call that is very expensive which is discarded unless trace level logging is turned on (this bug still exists in the current camel code) · The activemq connections start timing out and disconnecting from the broker side · This appears to cause the camel routes to spawn a new set of TimeoutChecker and ReplyManager threads for each of the routes which are all Has anyone seen issue like this? Any advice on what else I should be looking for? Thanks Sam Hendley -- *Christian Posta* http://www.christianposta.com/blog twitter: @christianposta
Failover JMS and Camel DefaultTimeoutManager failures
Hello all. I am working with a large application written in camel. We have recently started noticing (it may have been occurring for a while) that in some cases a sort of death spiral occurs. Out setup: Camel 2.10.1 (unfortunately we can't upgrade) ActiveMQ 5.6.0 (unfortunately we can't upgrade) As far as I can tell this is what is occurring: · Something puts excess load on the system and slows down our application · Some ReplyManagerTimeoutChecks start failing · It starts doing excessive amounts of work dealing with timeouts which delays the processing of the work causing more timeouts o Using a sampler it appears most the time is spent in DefaultTimeoutMap.purge() § There is a .size() call that is very expensive which is discarded unless trace level logging is turned on (this bug still exists in the current camel code) · The activemq connections start timing out and disconnecting from the broker side · This appears to cause the camel routes to spawn a new set of TimeoutChecker and ReplyManager threads for each of the routes which are all Has anyone seen issue like this? Any advice on what else I should be looking for? Thanks Sam Hendley
