Hi All, I am ok as long as we are invoking a method and not copying.
thanks, Dimuthu On Tue, Jan 9, 2018 at 4:41 PM, Isuru Udana <[email protected]> wrote: > Hi Hasunie, > > As we discussed, setting the PassThroughConstants.BUFFERED_INPUT_STREAM > has no effect on the flow in this case and Passthough Sender still seek > content from the original input stream which got empty due to this cloning > logic. That's the reason for this behaviour. > > Thanks. > > > > On Tue, Jan 9, 2018 at 11:43 AM, Hasunie Adikari <[email protected]> wrote: > >> Hi Isuru, >> >> As we discussed, I cloned the input stream by consuming the passthrough >> pipe as in below. >> >> >> if (pipe != null) { >> bufferedInputStream = new BufferedInputStream(pipe.getIn >> putStream()); >> >> } >> ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream(); >> byte[] buffer = new byte[1024]; >> int len; >> while ((len = bufferedInputStream.read(buffer)) > -1 ) { >> byteArrayOutputStream.write(buffer, 0, len); >> } >> byteArrayOutputStream.flush(); >> >> >> InputStream is1 = new ByteArrayInputStream(byteArray >> OutputStream.toByteArray()); >> InputStream is2 = new ByteArrayInputStream(byteArray >> OutputStream.toByteArray()); >> >> consume the clones for the validation and set another clone as a >> buffereInputstream property in the axis2messagecontext. >> >> BufferedInputStream bufferedInputStreamOriginal = new >> BufferedInputStream(inputStreamOriginal); >> axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM, >> bufferedInputStreamOriginal); >> >> I'm still getting the stream closed issue only for correct the messages >> which have been passed through multiple validators. If the validators throw >> an exception, the request is getting build and generate the custom response >> as expected. It seems like we implemented a way that gets the inputstream >> from the passthrough pipe for the content unaware flows. Unless it uses >> to get the inputstream from the messagecontext. It was proven once I >> attached an empty content aware mediator and test the same flow. I was >> able to observe the expected behaviour for the same implementation with the >> content aware mediator. >> >> Do we have a way to define cloned input stream as an original >> inputstream in passthrough pipe? >> >> >> Regards >> Hasunie >> >> >> >> >> >> >> On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <[email protected]> wrote: >> >>> Hi Dushan, >>> >>> On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[email protected]> >>> wrote: >>> >>>> Hi Hasunie, >>>> Current PTT design would build the message whenever if there is >>>> content aware mediator available. However IIRC, I did this >>>> *message.builder.the >>>> invoked* thing to cope with the WSO2 ELB we had (a few years ago). >>>> >>> No. I think it was *force.passthrough.builder *property which you >>> introduced for ELB requirement. >>> >>> To be honest, that looks ugly isn it (in terms of overall picture). >>>> Basically, what it does; even if there are content-aware mediators, the >>>> engine would forcefully ignore that (it was ELB requirement :) ) but for >>>> APIM I don't think that would be the same, cos we may have to deal with >>>> many use cases sometimes of cause with content aware flows with API >>>> compositions etc etc. >>>> >>>> So, let's think what we can do here; regex and XML threat protectors >>>> equally important if security is priority thus, we would no longer able to >>>> achieve the same core basic aspect (content none awareness) because, such >>>> protections required you to walk through the nodes and verify some aspects >>>> (basically, you need to expand the xml node tree to get result set) in that >>>> way, it is required the message to be build. Anyway, what I would think the >>>> best approach here is not to change complete synapse content awareness >>>> logic rather I would think you may have mediator in place but only if such >>>> protection engaged that may build the message to get XML inforset (rather >>>> build through root, may be you can mark this meditor as content-aware >>>> false, then build if message not already build prior to process) >>>> >>>> IMO, lets just not complicate the what we try to build around message >>>> validation. I mean if we need such protection we may need to sacrify some >>>> aspects am I? >>>> >>>> Cheers, >>>> Dushan >>>> >>>> On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[email protected]> wrote: >>>> >>>>> Hi Hasunie, >>>>> >>>>> This is expected since the synapse engine now expecting an already >>>>> built message. If I understood your requirement correctly, one option is >>>>> to >>>>> use a Builder Mediator before using any content aware mediator. Even >>>>> though >>>>> we do not recommend the Builder mediators now, still we can use it for >>>>> your >>>>> specific use case. Or you have to revert the >>>>> *message.builder.invoked *property to *false *again*.* >>>>> >>>>> Regards, >>>>> Vinod >>>>> >>>>> On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[email protected]> >>>>> wrote: >>>>> >>>>>> I'm trying to combine SQL injection(Regex) threat protector with the >>>>>> XML threat protector. So I created a sequence[1] with XMLthreatprotector >>>>>> mediator and regex mediator consecutively and uploaded it to be able to >>>>>> validate the request message through both the xml validator and regex >>>>>> validators. If I set the *message.builder.invoked *property to *TRUE >>>>>> *in xml validator mediator to avoid sending the content in >>>>>> pass-through pipe(request message) as the response, Regex mediators is >>>>>> getting failed. The regex mediator was designed a way that the incoming >>>>>> messages are built in synapse level and eveluate the message content at >>>>>> the mediator level. It seems like we can't continue any mediators which >>>>>> are >>>>>> required to get the message content, after we manually set the >>>>>> aforementioned property to true in the previous mediator. If I set it to >>>>>> true, RelayUtill will skip building the message as in here [2]. Any >>>>>> thoughts regarding the issue. I'm currently working on the issue to be >>>>>> able >>>>>> to combine regex and XML threat protectors. >>>>>> [2] >>>>>> >>>>>> if (pipe != null >>>>>> && !Boolean.TRUE.equals(messageContext >>>>>> >>>>>> .getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && >>>>>> forcePTBuild) { >>>>>> InputStream in = pipe.getInputStream(); >>>>>> >>>>>> Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC); >>>>>> if (http_sc != null && http_sc instanceof Integer && >>>>>> http_sc.equals(202)) { >>>>>> >>>>>> messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED, >>>>>> Boolean.TRUE); >>>>>> return; >>>>>> } >>>>>> >>>>>> builldMessage(messageContext, earlyBuild, in); >>>>>> return; >>>>>> } >>>>>> >>>>>> >>>>>> >>>>>> [1] >>>>>> <sequence xmlns="http://ws.apache.org/ns/synapse"; name="validator"> >>>>>> <property name="threatType" expression="get-property('threatType')" >>>>>> value="SQL-Injection and XML validator"/> >>>>>> <property> >>>>>> <property> >>>>>> -------- >>>>>> <switch source="get-property('To')"> >>>>>> -------------- >>>>>> </switch> >>>>>> *<class >>>>>> name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>* >>>>>> <property name="enabledCheckPathParams" >>>>>> expression="get-property('enabledCheckPathParams')" value="true"/> >>>>>> <property> >>>>>> <property> >>>>>> -- >>>>>> *<class >>>>>> name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>* >>>>>> <filter source="get-property('threat')" regex=".*error.*"> >>>>>> <then> >>>>>> <sequence key="_threat_fault_"></sequence> >>>>>> </then> >>>>>> </filter> >>>>>> </sequence> >>>>>> >>>>>> Regards >>>>>> Hasunie >>>>>> >>>>>> On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I'm working on threat protector feature in APIM. We're actually >>>>>>> trying to achieve here is to protect both backend resources and gateway >>>>>>> from the XML and JSON based attacks. The Balerina based APIM 3 gateway >>>>>>> will >>>>>>> be protected by threat handlers. But In here >>>>>>> APIM 2.1.x we have implemented mediators to achieve it. >>>>>>> >>>>>>> If we allow building the request message at the synapse level, It >>>>>>> will definitely affect the gateway, All the request messages which go >>>>>>> through the mediators are built since the Abstarctemediator is designed >>>>>>> a >>>>>>> way that the isContentAware method always returns true. So we set it to >>>>>>> false in both XML and JSON validator mediators and allow to parse the >>>>>>> XML request via a third party StAX parser called woodstox hence it was >>>>>>> the >>>>>>> best option among other StAX parsers for threat protection features. It >>>>>>> will keep counting the given limits and when the limit is exceeded, It >>>>>>> will >>>>>>> terminate the process and throw a meaningful exception. I have created a >>>>>>> custom threat sequence(thrat_fault) and If a threat is detected by >>>>>>> getting >>>>>>> an exception I configured to direct the response through the custom >>>>>>> error >>>>>>> sequence. >>>>>>> I reuse the same custom sequence which was implemented for the regex >>>>>>> threat protector [1] >>>>>>> >>>>>>> Woodstox parser covers most of the vulnerabilities as in here >>>>>>> >>>>>>> *Vulnerablity:* >>>>>>> >>>>>>> *xml bomb* - DTD disabling >>>>>>> >>>>>>> *external entity attack* - disabling external entities. >>>>>>> >>>>>>> Note : >>>>>>> Apart from the mediator level, The external entity >>>>>>> reference property was disabled from the DOM parsers at the synapse >>>>>>> level >>>>>>> as well. >>>>>>> >>>>>>> import org.apache.xerces.impl.Constants; >>>>>>> >>>>>>> private static final int ENTITY_EXPANSION_LIMIT = 0; >>>>>>> private static final DocumentBuilderFactory documentBuilderFactory = >>>>>>> >>>>>>> DocumentBuilderFactory.newInstance(); >>>>>>> >>>>>>> static { >>>>>>> documentBuilderFactory.setNamespaceAware(true); >>>>>>> documentBuilderFactory.setXIncludeAware(false); >>>>>>> documentBuilderFactory.setExpandEntityReferences(false); >>>>>>> >>>>>>> try { >>>>>>> documentBuilderFactory.setFeature(Constants.SAX_FEATU >>>>>>> RE_PREFIX + >>>>>>> Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false); >>>>>>> } catch (ParserConfigurationException e) { >>>>>>> >>>>>>> *Buffer overflow attack* - by limiting the count of elements, >>>>>>> children and length of attributes/keys/values. >>>>>>> >>>>>>> *woodstox properties:* >>>>>>> dtdEnabled >>>>>>> externalEntitiesEnabled >>>>>>> maxDepth >>>>>>> maxElementCount >>>>>>> maxAttributeCount >>>>>>> maxAttributeLength >>>>>>> entityExpansionLimit >>>>>>> maxChildrenPerElement >>>>>>> >>>>>>> >>>>>>> For thwart cohesive attacks, we use both schema validator and depth >>>>>>> limits. Ideally, only the woodstox validator should detect the cohesive >>>>>>> attacks by exceeding the defined depth limit. But the schema validator >>>>>>> will >>>>>>> protect the schema poising attacks in the second step as well. >>>>>>> >>>>>>> I observed an issue when It comes to combining each other(woodstox+ >>>>>>> schema validator). We have designed the feature in such a way that gets >>>>>>> the >>>>>>> inputstream from the message context and consumes it in the woodstox >>>>>>> validator. but in here we have to consume the input stream again for the >>>>>>> schema validation just after passing through the woodstox.That was the >>>>>>> issue and I tried the following methodologies to resolved the issue >>>>>>> >>>>>>> 1. try to get the XML object from the woodstox parser to be able to >>>>>>> avoid using the input stream again. >>>>>>> 2. deep clone the inputstream and use cloned input stream for the >>>>>>> schema validation. >>>>>>> 3. reset, mark the buffered input stream(synapse engine also has >>>>>>> done rest, mark) >>>>>>> >>>>>>> 1st one was taken time and much complex to get the XML object since >>>>>>> Woodstock is based on the StAX parsers and also deep cloning was not >>>>>>> working properly and experienced the same issue after cloning the >>>>>>> inputstream. But the 3rd option makes life easy so I implemented a way >>>>>>> that >>>>>>> returning the buffered input stream, after doing the rest, mark, then >>>>>>> It >>>>>>> works properly. I went through the RelayUtil message builders [2] and It >>>>>>> also uses the mark and reset methodology and return InputStream. >>>>>>> >>>>>>> >>>>>>> I observed another issue once the validator throws an exception, the >>>>>>> server hanged and didn't get any response and getting timeout issue. I >>>>>>> was >>>>>>> able to figure it out and Issue occurred while trying to build the >>>>>>> request >>>>>>> message in Relayutil.buildmessage().But Ideally, If we get an error we >>>>>>> don't need the request message anymore. As I discussed offline with the >>>>>>> APIM team, I used the *consumeAndDiscardMessage* method to discard >>>>>>> the request message from the message context and set >>>>>>> *message.builder.invoked >>>>>>> *property to *TRUE. *It needs to be set to avoid sending the >>>>>>> content in pass-through pipe (request message) as the response. >>>>>>> >>>>>>> >>>>>>> [1] https://docs.wso2.com/display/AM210/Regular+Expression+T >>>>>>> hreat+Protection+for+API+Gateway >>>>>>> [2] https://github.com/wso2/wso2-synapse/blob/master/modules >>>>>>> /transports/core/nhttp/src/main/java/org/apache/synapse/tran >>>>>>> sport/passthru/util/RelayUtils.java#L121 >>>>>>> >>>>>>> >>>>>>> Regards >>>>>>> Hasunie >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Hasunie Adikari* >>>>>>> Senior Software Engineer >>>>>>> WSO2 Inc.; http://wso2.com >>>>>>> lean.enterprise.middleware >>>>>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/ >>>>>>> Mobile:+94713095876 <+94%2071%20309%205876> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Hasunie Adikari* >>>>>> Senior Software Engineer >>>>>> WSO2 Inc.; http://wso2.com >>>>>> lean.enterprise.middleware >>>>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/ >>>>>> Mobile:+94713095876 <+94%2071%20309%205876> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Vinod Kavinda >>>>> Senior Software Engineer >>>>> *WSO2 Inc. - lean . enterprise . middleware <http://www.wso2.com>.* >>>>> Mobile : +94 (0) 712 415544 >>>>> Blog : http://soatechflicks.blogspot.com/ >>>>> [image: http://wso2.com/signature] >>>>> <http://wso2.com/signature> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Dushan Abeyruwan | Architect >>>> Technical Support,MV >>>> PMC Member Apache Synpase >>>> WSO2 Inc. http://wso2.com/ >>>> Blog:*http://www.dushantech.com/ <http://www.dushantech.com/>* >>>> LinkedIn:*https://www.linkedin.com/in/dushanabeyruwan >>>> <https://www.linkedin.com/in/dushanabeyruwan>* >>>> Mobile:(001)408-791-9312 <+1%20408-791-9312> >>>> >>>> >>> >>> >>> -- >>> *Isuru Udana* >>> Senior Technical Lead >>> WSO2 Inc.; http://wso2.com >>> email: [email protected] cell: +94 77 3791887 <+94%2077%20379%201887> >>> blog: http://mytecheye.blogspot.com/ >>> >> >> >> >> -- >> *Hasunie Adikari* >> Senior Software Engineer >> WSO2 Inc.; http://wso2.com >> lean.enterprise.middleware >> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/ >> Mobile:+94713095876 <071%20309%205876> >> >> > > > -- > *Isuru Udana* > Senior Technical Lead > WSO2 Inc.; http://wso2.com > email: [email protected] cell: +94 77 3791887 <077%20379%201887> > blog: http://mytecheye.blogspot.com/ > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Dimuthu Leelarathne Director, Solutions Architecture WSO2, Inc. (http://wso2.com) email: [email protected] Mobile: +94773661935 Blog: http://muthulee.blogspot.com Lean . Enterprise . Middleware
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
