[
https://issues.apache.org/jira/browse/SYNAPSE-1015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15315591#comment-15315591
]
Senduran commented on SYNAPSE-1015:
-----------------------------------
Hi Hiranya,
First we identified, that some of the message building were failing at high
concurrency(around 400). To narrow down the issue, we added verbose println
statements to log the buffer's position and limit. We could observer that the
failing requests got flipped / compacted additionally or didn't at all. Since
before setting the input mode / output mode, the atomic boolean flag is getting
set. So the only possible we could think of is, that the value of the atomic
boolean is incorrect when it comes to set the value.(i.e the input mode is
getting set only if the previous state is output mode, so if the state is
already input mode it won't get set) The problem here is, per each request the
pipe object is created. so the atomic boolean also created per pipe. And it is
not possible to get an incorrect value. But according to the println statements
that only possible could be the read / write flag is not set in correctly. So
we created the ControlledByteBuffer with ByteBuffer and its flag.
Also recently we came across another issue. Which also validates the advantage
of ByteBuffer and its flag are being together.
The issue is when response's end of chunk character comes after a delay, the
consecutive request is not getting its response (request timeouts).
When the end of chunk flag is received after a delay and the decoder reads it
as 0 bytes so the buffer is (still) empty and when it got compacted (in the
setInputMode method) the buffer becomes full and the suspendInput() is called.
so the consecutive request is getting timeout. This issue occurs when the
response path is content aware. When the message body is received(without end
of chunk flag), it is produced to buffer successfully, and it is consumed from
the outbuffer (since the response path is content aware). Meanwhile the buffer
is cleared and released to the buffer factory (but note that the pipe object is
still holding its reference and the inputMode of the buffer as false, because
when buffer is read it calls the setOutput Mode and flag is set to false).
When the end of chunk flag is received, it is produced to the same buffer,
(which is cleared and released to the buffer factory), but it gets compacted
because the input mode is false, so setInputMode() set the flag to true and
compact the empty buffer, which eventually becomes full and suspendInput is
called. When we wrapped the ByteBuffer and its flag together, it won't happen
because, when we release it to the factory, we forcefully set its input mode to
true. so we can't again set the flag to true and it won't get compacted.
Above explanations are based on my understanding and observation. I believe it
gives some idea about the fix I tried.
Thank you
Senduran
> Message building fails at high concurrency when the outsequence is content
> aware
> ---------------------------------------------------------------------------------
>
> Key: SYNAPSE-1015
> URL: https://issues.apache.org/jira/browse/SYNAPSE-1015
> Project: Synapse
> Issue Type: Bug
> Components: Core, Transports
> Affects Versions: 2.1
> Reporter: Senduran
> Assignee: Hiranya Jayathilaka
> Priority: Blocker
> Attachments: controlled-bytebuffer-fix.diff
>
>
> When invoke the following proxy at a concurrency (around 200)
> {code:xml}
> <proxy xmlns="http://ws.apache.org/ns/synapse";
> name="HelloProxy"
> transports="https,http"
> statistics="disable"
> trace="disable"
> startOnLoad="true">
> <target>
> <inSequence>
> <send>
> <endpoint>
> <address uri="http://localhost:9765/services/HelloService/"/>
> </endpoint>
> </send>
> </inSequence>
> <outSequence>
> <enrich>
> <source type="inline" clone="true">
> <sss xmlns="">a</sss>
> </source>
> <target type="property" property="bar"/>
> </enrich>
> <send/>
> </outSequence>
> </target>
> <publishWSDL uri="http://localhost:9765/services/HelloService?wsdl"/>
> <description/>
> </proxy>
> {code}
> Following exception occurs after a while,
> {code}
> 2016-05-23 23:51:09,455 [-] [PassThroughMessageProcessor-30] ERROR RelayUtils
> Error while building PassThrough stream
> org.apache.axiom.om.DeferredParsingException:
> com.ctc.wstx.exc.WstxUnexpectedCharException: Illegal character (NULL,
> unicode 0) encountered: not valid in any content
> at [row,col {unknown-source}]: [1,1]
> at
> org.apache.axiom.om.impl.builder.StAXOMBuilder.parserNext(StAXOMBuilder.java:681)
> at
> org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:184)
> at
> org.apache.axiom.core.CoreParentNodeSupport.ajc$interMethod$org_apache_axiom_core_CoreParentNodeSupport$org_apache_axiom_core_CoreParentNode$buildNext(CoreParentNodeSupport.aj:96)
> at
> org.apache.axiom.om.impl.llom.OMDocumentImpl.buildNext(OMDocumentImpl.java:1)
> at
> org.apache.axiom.core.CoreParentNodeSupport.ajc$interMethodDispatch1$org_apache_axiom_core_CoreParentNodeSupport$org_apache_axiom_core_CoreParentNode$buildNext(CoreParentNodeSupport.aj)
> at
> org.apache.axiom.core.CoreParentNodeSupport.ajc$interMethod$org_apache_axiom_core_CoreParentNodeSupport$org_apache_axiom_core_CoreParentNode$coreGetFirstChild(CoreParentNodeSupport.aj:113)
> at
> org.apache.axiom.om.impl.llom.OMDocumentImpl.coreGetFirstChild(OMDocumentImpl.java:1)
> at
> org.apache.axiom.core.CoreParentNodeSupport.ajc$interMethodDispatch1$org_apache_axiom_core_CoreParentNodeSupport$org_apache_axiom_core_CoreParentNode$coreGetFirstChild(CoreParentNodeSupport.aj)
> at
> org.apache.axiom.core.CoreDocumentSupport.ajc$interMethod$org_apache_axiom_core_CoreDocumentSupport$org_apache_axiom_core_CoreDocument$coreGetDocumentElement(CoreDocumentSupport.aj:42)
> at
> org.apache.axiom.om.impl.llom.OMDocumentImpl.coreGetDocumentElement(OMDocumentImpl.java:1)
> at
> org.apache.axiom.om.impl.common.AxiomDocumentSupport.ajc$interMethod$org_apache_axiom_om_impl_common_AxiomDocumentSupport$org_apache_axiom_om_impl_intf_AxiomDocument$getOMDocumentElement(AxiomDocumentSupport.aj:32)
> at
> org.apache.axiom.om.impl.llom.OMDocumentImpl.getOMDocumentElement(OMDocumentImpl.java:1)
> at
> org.apache.axiom.om.impl.builder.StAXOMBuilder.getDocumentElement(StAXOMBuilder.java:557)
> at
> org.apache.axiom.om.impl.builder.StAXOMBuilder.getDocumentElement(StAXOMBuilder.java:553)
> at
> org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:53)
> at
> org.apache.synapse.transport.passthru.util.DeferredMessageBuilder.getDocument(DeferredMessageBuilder.java:114)
> at
> org.apache.synapse.transport.passthru.util.RelayUtils.buildMessage(RelayUtils.java:146)
> at
> org.apache.synapse.transport.passthru.util.RelayUtils.buildMessage(RelayUtils.java:86)
> at
> org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:62)
> at
> org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:114)
> at
> org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:230)
> at
> org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:444)
> at
> org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:221)
> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:169)
> at
> org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:221)
> at
> org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:173)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: Illegal character
> (NULL, unicode 0) encountered: not valid in any content
> at [row,col {unknown-source}]: [1,1]
> at
> com.ctc.wstx.sr.StreamScanner.constructNullCharException(StreamScanner.java:630)
> at
> com.ctc.wstx.sr.StreamScanner.throwInvalidSpace(StreamScanner.java:660)
> at
> com.ctc.wstx.sr.StreamScanner.throwInvalidSpace(StreamScanner.java:651)
> at
> com.ctc.wstx.sr.BasicStreamReader.readSpacePrimary(BasicStreamReader.java:4965)
> at
> com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:2008)
> at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1134)
> at
> org.apache.axiom.util.stax.wrapper.XMLStreamReaderWrapper.next(XMLStreamReaderWrapper.java:225)
> at
> org.apache.axiom.util.stax.dialect.DisallowDoctypeDeclStreamReaderWrapper.next(DisallowDoctypeDeclStreamReaderWrapper.java:34)
> at
> org.apache.axiom.util.stax.wrapper.XMLStreamReaderWrapper.next(XMLStreamReaderWrapper.java:225)
> at
> org.apache.axiom.om.impl.builder.StAXOMBuilder.parserNext(StAXOMBuilder.java:666)
> ... 28 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
