Hi Dushan, It is an optional mediator. We haven't touched the core (not a single line)
We will provide the performance results on Monday. thanks, Dimuthu On Mon, Jan 15, 2018 at 10:26 PM, Dushan Abeyruwan <[email protected]> wrote: > Hi > Pls provide the diff of the changes you have done. > > @ESB Team / PPT experts, since there are PPT level changes you need keep > watch on performance impact, memory blueprint impact, how the heap usage > varies per message size since (smallest to the largest) + per how the > behavior for complex interaction which has clone/iterators etc etc. > > Cheers, > Dushan > > On Tue, Jan 9, 2018 at 9:38 AM, Hasunie Adikari <[email protected]> wrote: > >> Hi all, >> >> As I discussed with Isuru, There are some possible approaches to overcome >> the issue. >> >> 1. Create a new pass through pipe. >> - The data will be written to the pipe by a spawned thread and current >> thread will be consuming the data and continuing the message flow. We went >> through the pipe creation logic and seemed it tightly coupled with >> encoding, decoding methodologies so that it can't be >> implemented at APIM level. >> >> 2. Change the synapse level logic to get the ByteArrayInputstream and >> write it into the response. >> - It can be but have to thoroughly go through and do it carefully >> unless the default message flow would be affected. >> >> 3. Build the message by invoking RelayUtils.buildMessage() at the >> validator mediator after successfully parsing all the validators. >> - It will be slightly affected the performance but this is the >> straightforward solution at this moment. >> >> I have improved the code by applying the 3rd option as we discussed. >> Setting the PassThroughConstants.BUFFERED_INPUT_STREAM has an effect now >> onwards since we changed a way that building the message to achieve the >> content aware behavior which seeks the >> inputream from the axis2 message context instead of the original >> inputstream. >> >> >> Regards >> Hasunie >> >> >> >> On Tue, Jan 9, 2018 at 4:41 PM, Isuru Udana <[email protected]> wrote: >> >>> Hi Hasunie, >>> >>> As we discussed, setting the PassThroughConstants.BUFFERED_INPUT_STREAM >>> has no effect on the flow in this case and Passthough Sender still seek >>> content from the original input stream which got empty due to this cloning >>> logic. That's the reason for this behaviour. >>> >>> Thanks. >>> >>> >>> >>> On Tue, Jan 9, 2018 at 11:43 AM, Hasunie Adikari <[email protected]> >>> wrote: >>> >>>> Hi Isuru, >>>> >>>> As we discussed, I cloned the input stream by consuming the passthrough >>>> pipe as in below. >>>> >>>> >>>> if (pipe != null) { >>>> bufferedInputStream = new BufferedInputStream(pipe.getIn >>>> putStream()); >>>> >>>> } >>>> ByteArrayOutputStream byteArrayOutputStream = new >>>> ByteArrayOutputStream(); >>>> byte[] buffer = new byte[1024]; >>>> int len; >>>> while ((len = bufferedInputStream.read(buffer)) > -1 ) { >>>> byteArrayOutputStream.write(buffer, 0, len); >>>> } >>>> byteArrayOutputStream.flush(); >>>> >>>> >>>> InputStream is1 = new ByteArrayInputStream(byteArray >>>> OutputStream.toByteArray()); >>>> InputStream is2 = new ByteArrayInputStream(byteArray >>>> OutputStream.toByteArray()); >>>> >>>> consume the clones for the validation and set another clone as a >>>> buffereInputstream property in the axis2messagecontext. >>>> >>>> BufferedInputStream bufferedInputStreamOriginal = new >>>> BufferedInputStream(inputStreamOriginal); >>>> axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM, >>>> bufferedInputStreamOriginal); >>>> >>>> I'm still getting the stream closed issue only for correct the messages >>>> which have been passed through multiple validators. If the validators throw >>>> an exception, the request is getting build and generate the custom response >>>> as expected. It seems like we implemented a way that gets the >>>> inputstream from the passthrough pipe for the content unaware flows. >>>> Unless it uses to get the inputstream from the messagecontext. It was >>>> proven once I attached an empty content aware mediator and test the >>>> same flow. I was able to observe the expected behaviour for the same >>>> implementation with the content aware mediator. >>>> >>>> Do we have a way to define cloned input stream as an original >>>> inputstream in passthrough pipe? >>>> >>>> >>>> Regards >>>> Hasunie >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <[email protected]> wrote: >>>> >>>>> Hi Dushan, >>>>> >>>>> On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Hasunie, >>>>>> Current PTT design would build the message whenever if there is >>>>>> content aware mediator available. However IIRC, I did this >>>>>> *message.builder.the >>>>>> invoked* thing to cope with the WSO2 ELB we had (a few years ago). >>>>>> >>>>> No. I think it was *force.passthrough.builder *property which you >>>>> introduced for ELB requirement. >>>>> >>>>> To be honest, that looks ugly isn it (in terms of overall picture). >>>>>> Basically, what it does; even if there are content-aware mediators, the >>>>>> engine would forcefully ignore that (it was ELB requirement :) ) but for >>>>>> APIM I don't think that would be the same, cos we may have to deal with >>>>>> many use cases sometimes of cause with content aware flows with API >>>>>> compositions etc etc. >>>>>> >>>>>> So, let's think what we can do here; regex and XML threat protectors >>>>>> equally important if security is priority thus, we would no longer able >>>>>> to >>>>>> achieve the same core basic aspect (content none awareness) because, such >>>>>> protections required you to walk through the nodes and verify some >>>>>> aspects >>>>>> (basically, you need to expand the xml node tree to get result set) in >>>>>> that >>>>>> way, it is required the message to be build. Anyway, what I would think >>>>>> the >>>>>> best approach here is not to change complete synapse content awareness >>>>>> logic rather I would think you may have mediator in place but only if >>>>>> such >>>>>> protection engaged that may build the message to get XML inforset (rather >>>>>> build through root, may be you can mark this meditor as content-aware >>>>>> false, then build if message not already build prior to process) >>>>>> >>>>>> IMO, lets just not complicate the what we try to build around message >>>>>> validation. I mean if we need such protection we may need to sacrify some >>>>>> aspects am I? >>>>>> >>>>>> Cheers, >>>>>> Dushan >>>>>> >>>>>> On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[email protected]> wrote: >>>>>> >>>>>>> Hi Hasunie, >>>>>>> >>>>>>> This is expected since the synapse engine now expecting an already >>>>>>> built message. If I understood your requirement correctly, one option >>>>>>> is to >>>>>>> use a Builder Mediator before using any content aware mediator. Even >>>>>>> though >>>>>>> we do not recommend the Builder mediators now, still we can use it for >>>>>>> your >>>>>>> specific use case. Or you have to revert the >>>>>>> *message.builder.invoked *property to *false *again*.* >>>>>>> >>>>>>> Regards, >>>>>>> Vinod >>>>>>> >>>>>>> On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> I'm trying to combine SQL injection(Regex) threat protector with >>>>>>>> the XML threat protector. So I created a sequence[1] with >>>>>>>> XMLthreatprotector mediator and regex mediator consecutively and >>>>>>>> uploaded >>>>>>>> it to be able to validate the request message through both the >>>>>>>> xml validator and regex validators. If I set the >>>>>>>> *message.builder.invoked *property to *TRUE *in xml validator >>>>>>>> mediator to avoid sending the content in pass-through pipe(request >>>>>>>> message) as the response, Regex mediators is getting failed. The regex >>>>>>>> mediator was designed a way that the incoming messages are built in >>>>>>>> synapse >>>>>>>> level and eveluate the message content at the mediator level. It >>>>>>>> seems like we can't continue any mediators which are required to get >>>>>>>> the >>>>>>>> message content, after we manually set the aforementioned property to >>>>>>>> true >>>>>>>> in the previous mediator. If I set it to true, RelayUtill will skip >>>>>>>> building the message as in here [2]. Any thoughts regarding the issue. >>>>>>>> I'm currently working on the issue to be able to combine regex and XML >>>>>>>> threat protectors. >>>>>>>> [2] >>>>>>>> >>>>>>>> if (pipe != null >>>>>>>> && !Boolean.TRUE.equals(messageContext >>>>>>>> >>>>>>>> .getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && >>>>>>>> forcePTBuild) { >>>>>>>> InputStream in = pipe.getInputStream(); >>>>>>>> >>>>>>>> Object http_sc = >>>>>>>> messageContext.getProperty(NhttpConstants.HTTP_SC); >>>>>>>> if (http_sc != null && http_sc instanceof Integer && >>>>>>>> http_sc.equals(202)) { >>>>>>>> >>>>>>>> messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED, >>>>>>>> Boolean.TRUE); >>>>>>>> return; >>>>>>>> } >>>>>>>> >>>>>>>> builldMessage(messageContext, earlyBuild, in); >>>>>>>> return; >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> [1] >>>>>>>> <sequence xmlns="http://ws.apache.org/ns/synapse"; name="validator"> >>>>>>>> <property name="threatType" expression="get-property('threatType')" >>>>>>>> value="SQL-Injection and XML validator"/> >>>>>>>> <property> >>>>>>>> <property> >>>>>>>> -------- >>>>>>>> <switch source="get-property('To')"> >>>>>>>> -------------- >>>>>>>> </switch> >>>>>>>> *<class >>>>>>>> name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>* >>>>>>>> <property name="enabledCheckPathParams" >>>>>>>> expression="get-property('enabledCheckPathParams')" value="true"/> >>>>>>>> <property> >>>>>>>> <property> >>>>>>>> -- >>>>>>>> *<class >>>>>>>> name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>* >>>>>>>> <filter source="get-property('threat')" regex=".*error.*"> >>>>>>>> <then> >>>>>>>> <sequence key="_threat_fault_"></sequence> >>>>>>>> </then> >>>>>>>> </filter> >>>>>>>> </sequence> >>>>>>>> >>>>>>>> Regards >>>>>>>> Hasunie >>>>>>>> >>>>>>>> On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I'm working on threat protector feature in APIM. We're actually >>>>>>>>> trying to achieve here is to protect both backend resources and >>>>>>>>> gateway >>>>>>>>> from the XML and JSON based attacks. The Balerina based APIM 3 >>>>>>>>> gateway will >>>>>>>>> be protected by threat handlers. But In here >>>>>>>>> APIM 2.1.x we have implemented mediators to achieve it. >>>>>>>>> >>>>>>>>> If we allow building the request message at the synapse level, It >>>>>>>>> will definitely affect the gateway, All the request messages which go >>>>>>>>> through the mediators are built since the Abstarctemediator is >>>>>>>>> designed a >>>>>>>>> way that the isContentAware method always returns true. So we set it >>>>>>>>> to >>>>>>>>> false in both XML and JSON validator mediators and allow to parse the >>>>>>>>> XML request via a third party StAX parser called woodstox hence it >>>>>>>>> was the >>>>>>>>> best option among other StAX parsers for threat protection features. >>>>>>>>> It >>>>>>>>> will keep counting the given limits and when the limit is exceeded, >>>>>>>>> It will >>>>>>>>> terminate the process and throw a meaningful exception. I have >>>>>>>>> created a >>>>>>>>> custom threat sequence(thrat_fault) and If a threat is detected by >>>>>>>>> getting >>>>>>>>> an exception I configured to direct the response through the custom >>>>>>>>> error >>>>>>>>> sequence. >>>>>>>>> I reuse the same custom sequence which was implemented for the >>>>>>>>> regex threat protector [1] >>>>>>>>> >>>>>>>>> Woodstox parser covers most of the vulnerabilities as in here >>>>>>>>> >>>>>>>>> *Vulnerablity:* >>>>>>>>> >>>>>>>>> *xml bomb* - DTD disabling >>>>>>>>> >>>>>>>>> *external entity attack* - disabling external entities. >>>>>>>>> >>>>>>>>> Note : >>>>>>>>> Apart from the mediator level, The external entity >>>>>>>>> reference property was disabled from the DOM parsers at the synapse >>>>>>>>> level >>>>>>>>> as well. >>>>>>>>> >>>>>>>>> import org.apache.xerces.impl.Constants; >>>>>>>>> >>>>>>>>> private static final int ENTITY_EXPANSION_LIMIT = 0; >>>>>>>>> private static final DocumentBuilderFactory >>>>>>>>> documentBuilderFactory = >>>>>>>>> DocumentBuilderFactory.newInstance(); >>>>>>>>> >>>>>>>>> static { >>>>>>>>> documentBuilderFactory.setNamespaceAware(true); >>>>>>>>> documentBuilderFactory.setXIncludeAware(false); >>>>>>>>> documentBuilderFactory.setExpandEntityReferences(false); >>>>>>>>> >>>>>>>>> try { >>>>>>>>> documentBuilderFactory.setFeature(Constants.SAX_FEATU >>>>>>>>> RE_PREFIX + >>>>>>>>> Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false); >>>>>>>>> } catch (ParserConfigurationException e) { >>>>>>>>> >>>>>>>>> *Buffer overflow attack* - by limiting the count of elements, >>>>>>>>> children and length of attributes/keys/values. >>>>>>>>> >>>>>>>>> *woodstox properties:* >>>>>>>>> dtdEnabled >>>>>>>>> externalEntitiesEnabled >>>>>>>>> maxDepth >>>>>>>>> maxElementCount >>>>>>>>> maxAttributeCount >>>>>>>>> maxAttributeLength >>>>>>>>> entityExpansionLimit >>>>>>>>> maxChildrenPerElement >>>>>>>>> >>>>>>>>> >>>>>>>>> For thwart cohesive attacks, we use both schema validator and >>>>>>>>> depth limits. Ideally, only the woodstox validator should detect the >>>>>>>>> cohesive attacks by exceeding the defined depth limit. But the schema >>>>>>>>> validator will protect the schema poising attacks in the second step >>>>>>>>> as >>>>>>>>> well. >>>>>>>>> >>>>>>>>> I observed an issue when It comes to combining each >>>>>>>>> other(woodstox+ schema validator). We have designed the feature in >>>>>>>>> such a >>>>>>>>> way that gets the inputstream from the message context and consumes >>>>>>>>> it in >>>>>>>>> the woodstox validator. but in here we have to consume the input >>>>>>>>> stream >>>>>>>>> again for the schema validation just after passing through the >>>>>>>>> woodstox.That was the issue and I tried the following methodologies to >>>>>>>>> resolved the issue >>>>>>>>> >>>>>>>>> 1. try to get the XML object from the woodstox parser to be able >>>>>>>>> to avoid using the input stream again. >>>>>>>>> 2. deep clone the inputstream and use cloned input stream for the >>>>>>>>> schema validation. >>>>>>>>> 3. reset, mark the buffered input stream(synapse engine also has >>>>>>>>> done rest, mark) >>>>>>>>> >>>>>>>>> 1st one was taken time and much complex to get the XML object >>>>>>>>> since Woodstock is based on the StAX parsers and also deep cloning >>>>>>>>> was not >>>>>>>>> working properly and experienced the same issue after cloning the >>>>>>>>> inputstream. But the 3rd option makes life easy so I implemented a >>>>>>>>> way that >>>>>>>>> returning the buffered input stream, after doing the rest, mark, >>>>>>>>> then It >>>>>>>>> works properly. I went through the RelayUtil message builders [2] and >>>>>>>>> It >>>>>>>>> also uses the mark and reset methodology and return InputStream. >>>>>>>>> >>>>>>>>> >>>>>>>>> I observed another issue once the validator throws an exception, >>>>>>>>> the server hanged and didn't get any response and getting timeout >>>>>>>>> issue. I >>>>>>>>> was able to figure it out and Issue occurred while trying to build the >>>>>>>>> request message in Relayutil.buildmessage().But Ideally, If we get an >>>>>>>>> error >>>>>>>>> we don't need the request message anymore. As I discussed offline >>>>>>>>> with the >>>>>>>>> APIM team, I used the *consumeAndDiscardMessage* method to >>>>>>>>> discard the request message from the message context and set >>>>>>>>> *message.builder.invoked >>>>>>>>> *property to *TRUE. *It needs to be set to avoid sending the >>>>>>>>> content in pass-through pipe (request message) as the response. >>>>>>>>> >>>>>>>>> >>>>>>>>> [1] https://docs.wso2.com/display/AM210/Regular+Expression+T >>>>>>>>> hreat+Protection+for+API+Gateway >>>>>>>>> [2] https://github.com/wso2/wso2-synapse/blob/master/modules >>>>>>>>> /transports/core/nhttp/src/main/java/org/apache/synapse/tran >>>>>>>>> sport/passthru/util/RelayUtils.java#L121 >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> Hasunie >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Hasunie Adikari* >>>>>>>>> Senior Software Engineer >>>>>>>>> WSO2 Inc.; http://wso2.com >>>>>>>>> lean.enterprise.middleware >>>>>>>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/ >>>>>>>>> Mobile:+94713095876 <+94%2071%20309%205876> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Hasunie Adikari* >>>>>>>> Senior Software Engineer >>>>>>>> WSO2 Inc.; http://wso2.com >>>>>>>> lean.enterprise.middleware >>>>>>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/ >>>>>>>> Mobile:+94713095876 <+94%2071%20309%205876> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Vinod Kavinda >>>>>>> Senior Software Engineer >>>>>>> *WSO2 Inc. - lean . enterprise . middleware <http://www.wso2.com>.* >>>>>>> Mobile : +94 (0) 712 415544 >>>>>>> Blog : http://soatechflicks.blogspot.com/ >>>>>>> [image: http://wso2.com/signature] >>>>>>> <http://wso2.com/signature> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Dushan Abeyruwan | Architect >>>>>> Technical Support,MV >>>>>> PMC Member Apache Synpase >>>>>> WSO2 Inc. http://wso2.com/ >>>>>> Blog:*http://www.dushantech.com/ <http://www.dushantech.com/>* >>>>>> LinkedIn:*https://www.linkedin.com/in/dushanabeyruwan >>>>>> <https://www.linkedin.com/in/dushanabeyruwan>* >>>>>> Mobile:(001)408-791-9312 <+1%20408-791-9312> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Isuru Udana* >>>>> Senior Technical Lead >>>>> WSO2 Inc.; http://wso2.com >>>>> email: [email protected] cell: +94 77 3791887 <+94%2077%20379%201887> >>>>> blog: http://mytecheye.blogspot.com/ >>>>> >>>> >>>> >>>> >>>> -- >>>> *Hasunie Adikari* >>>> Senior Software Engineer >>>> WSO2 Inc.; http://wso2.com >>>> lean.enterprise.middleware >>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/ >>>> Mobile:+94713095876 <071%20309%205876> >>>> >>>> >>> >>> >>> -- >>> *Isuru Udana* >>> Senior Technical Lead >>> WSO2 Inc.; http://wso2.com >>> email: [email protected] cell: +94 77 3791887 <+94%2077%20379%201887> >>> blog: http://mytecheye.blogspot.com/ >>> >> >> >> >> -- >> *Hasunie Adikari* >> Senior Software Engineer >> WSO2 Inc.; http://wso2.com >> lean.enterprise.middleware >> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/ >> Mobile:+94713095876 <+94%2071%20309%205876> >> >> > > > -- > Dushan Abeyruwan | Architect > Technical Support,MV > PMC Member Apache Synpase > WSO2 Inc. http://wso2.com/ > Blog:*http://www.dushantech.com/ <http://www.dushantech.com/>* > LinkedIn:*https://www.linkedin.com/in/dushanabeyruwan > <https://www.linkedin.com/in/dushanabeyruwan>* > Mobile:(001)408-791-9312 <+1%20408-791-9312> > > -- Dimuthu Leelarathne Director, Solutions Architecture WSO2, Inc. (http://wso2.com) email: [email protected] Mobile: +94773661935 Blog: http://muthulee.blogspot.com Lean . Enterprise . Middleware
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
