Hi Dimuthu, Understood, thanks for the prompted response Cheers, Dushan
On Tue, Jan 16, 2018 at 5:12 AM, Dimuthu Leelarathne <[email protected]> wrote: > Hi Dushan, > > It is an optional mediator. We haven't touched the core (not a single line) > > We will provide the performance results on Monday. > > thanks, > Dimuthu > > > On Mon, Jan 15, 2018 at 10:26 PM, Dushan Abeyruwan <[email protected]> > wrote: > >> Hi >> Pls provide the diff of the changes you have done. >> >> @ESB Team / PPT experts, since there are PPT level changes you need keep >> watch on performance impact, memory blueprint impact, how the heap usage >> varies per message size since (smallest to the largest) + per how the >> behavior for complex interaction which has clone/iterators etc etc. >> >> Cheers, >> Dushan >> >> On Tue, Jan 9, 2018 at 9:38 AM, Hasunie Adikari <[email protected]> wrote: >> >>> Hi all, >>> >>> As I discussed with Isuru, There are some possible approaches to >>> overcome the issue. >>> >>> 1. Create a new pass through pipe. >>> - The data will be written to the pipe by a spawned thread and >>> current thread will be consuming the data and continuing the message flow. >>> We went through the pipe creation logic and seemed it tightly coupled with >>> encoding, decoding methodologies so that it can't be >>> implemented at APIM level. >>> >>> 2. Change the synapse level logic to get the ByteArrayInputstream and >>> write it into the response. >>> - It can be but have to thoroughly go through and do it carefully >>> unless the default message flow would be affected. >>> >>> 3. Build the message by invoking RelayUtils.buildMessage() at the >>> validator mediator after successfully parsing all the validators. >>> - It will be slightly affected the performance but this is the >>> straightforward solution at this moment. >>> >>> I have improved the code by applying the 3rd option as we discussed. >>> Setting the PassThroughConstants.BUFFERED_INPUT_STREAM has an effect >>> now onwards since we changed a way that building the message to achieve the >>> content aware behavior which seeks the >>> inputream from the axis2 message context instead of the original >>> inputstream. >>> >>> >>> Regards >>> Hasunie >>> >>> >>> >>> On Tue, Jan 9, 2018 at 4:41 PM, Isuru Udana <[email protected]> wrote: >>> >>>> Hi Hasunie, >>>> >>>> As we discussed, setting the PassThroughConstants.BUFFERED_INPUT_STREAM >>>> has no effect on the flow in this case and Passthough Sender still seek >>>> content from the original input stream which got empty due to this cloning >>>> logic. That's the reason for this behaviour. >>>> >>>> Thanks. >>>> >>>> >>>> >>>> On Tue, Jan 9, 2018 at 11:43 AM, Hasunie Adikari <[email protected]> >>>> wrote: >>>> >>>>> Hi Isuru, >>>>> >>>>> As we discussed, I cloned the input stream by consuming the >>>>> passthrough pipe as in below. >>>>> >>>>> >>>>> if (pipe != null) { >>>>> bufferedInputStream = new BufferedInputStream(pipe.getIn >>>>> putStream()); >>>>> >>>>> } >>>>> ByteArrayOutputStream byteArrayOutputStream = new >>>>> ByteArrayOutputStream(); >>>>> byte[] buffer = new byte[1024]; >>>>> int len; >>>>> while ((len = bufferedInputStream.read(buffer)) > -1 ) { >>>>> byteArrayOutputStream.write(buffer, 0, len); >>>>> } >>>>> byteArrayOutputStream.flush(); >>>>> >>>>> >>>>> InputStream is1 = new ByteArrayInputStream(byteArray >>>>> OutputStream.toByteArray()); >>>>> InputStream is2 = new ByteArrayInputStream(byteArray >>>>> OutputStream.toByteArray()); >>>>> >>>>> consume the clones for the validation and set another clone as a >>>>> buffereInputstream property in the axis2messagecontext. >>>>> >>>>> BufferedInputStream bufferedInputStreamOriginal = new >>>>> BufferedInputStream(inputStreamOriginal); >>>>> axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM, >>>>> bufferedInputStreamOriginal); >>>>> >>>>> I'm still getting the stream closed issue only for correct the >>>>> messages which have been passed through multiple validators. If the >>>>> validators throw an exception, the request is getting build and generate >>>>> the custom response as expected. It seems like we implemented a way >>>>> that gets the inputstream from the passthrough pipe for the content >>>>> unaware flows. Unless it uses to get the inputstream from the >>>>> messagecontext. It was proven once I attached an empty content aware >>>>> mediator and test the same flow. I was able to observe the expected >>>>> behaviour for the same implementation with the content aware mediator. >>>>> >>>>> Do we have a way to define cloned input stream as an original >>>>> inputstream in passthrough pipe? >>>>> >>>>> >>>>> Regards >>>>> Hasunie >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <[email protected]> wrote: >>>>> >>>>>> Hi Dushan, >>>>>> >>>>>> On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Hasunie, >>>>>>> Current PTT design would build the message whenever if there is >>>>>>> content aware mediator available. However IIRC, I did this >>>>>>> *message.builder.the >>>>>>> invoked* thing to cope with the WSO2 ELB we had (a few years ago). >>>>>>> >>>>>> No. I think it was *force.passthrough.builder *property which you >>>>>> introduced for ELB requirement. >>>>>> >>>>>> To be honest, that looks ugly isn it (in terms of overall picture). >>>>>>> Basically, what it does; even if there are content-aware mediators, the >>>>>>> engine would forcefully ignore that (it was ELB requirement :) ) but for >>>>>>> APIM I don't think that would be the same, cos we may have to deal with >>>>>>> many use cases sometimes of cause with content aware flows with API >>>>>>> compositions etc etc. >>>>>>> >>>>>>> So, let's think what we can do here; regex and XML threat protectors >>>>>>> equally important if security is priority thus, we would no longer >>>>>>> able to >>>>>>> achieve the same core basic aspect (content none awareness) because, >>>>>>> such >>>>>>> protections required you to walk through the nodes and verify some >>>>>>> aspects >>>>>>> (basically, you need to expand the xml node tree to get result set) in >>>>>>> that >>>>>>> way, it is required the message to be build. Anyway, what I would think >>>>>>> the >>>>>>> best approach here is not to change complete synapse content awareness >>>>>>> logic rather I would think you may have mediator in place but only if >>>>>>> such >>>>>>> protection engaged that may build the message to get XML inforset >>>>>>> (rather >>>>>>> build through root, may be you can mark this meditor as content-aware >>>>>>> false, then build if message not already build prior to process) >>>>>>> >>>>>>> IMO, lets just not complicate the what we try to build around >>>>>>> message validation. I mean if we need such protection we may need to >>>>>>> sacrify some aspects am I? >>>>>>> >>>>>>> Cheers, >>>>>>> Dushan >>>>>>> >>>>>>> On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Hasunie, >>>>>>>> >>>>>>>> This is expected since the synapse engine now expecting an already >>>>>>>> built message. If I understood your requirement correctly, one option >>>>>>>> is to >>>>>>>> use a Builder Mediator before using any content aware mediator. Even >>>>>>>> though >>>>>>>> we do not recommend the Builder mediators now, still we can use it for >>>>>>>> your >>>>>>>> specific use case. Or you have to revert the >>>>>>>> *message.builder.invoked *property to *false *again*.* >>>>>>>> >>>>>>>> Regards, >>>>>>>> Vinod >>>>>>>> >>>>>>>> On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I'm trying to combine SQL injection(Regex) threat protector with >>>>>>>>> the XML threat protector. So I created a sequence[1] with >>>>>>>>> XMLthreatprotector mediator and regex mediator consecutively and >>>>>>>>> uploaded >>>>>>>>> it to be able to validate the request message through both the >>>>>>>>> xml validator and regex validators. If I set the >>>>>>>>> *message.builder.invoked *property to *TRUE *in xml validator >>>>>>>>> mediator to avoid sending the content in pass-through >>>>>>>>> pipe(request message) as the response, Regex mediators is getting >>>>>>>>> failed. >>>>>>>>> The regex mediator was designed a way that the incoming messages are >>>>>>>>> built >>>>>>>>> in synapse level and eveluate the message content at the mediator >>>>>>>>> level. It seems like we can't continue any mediators which are >>>>>>>>> required to >>>>>>>>> get the message content, after we manually set the aforementioned >>>>>>>>> property >>>>>>>>> to true in the previous mediator. If I set it to true, RelayUtill >>>>>>>>> will skip >>>>>>>>> building the message as in here [2]. Any thoughts regarding the issue. >>>>>>>>> I'm currently working on the issue to be able to combine regex and XML >>>>>>>>> threat protectors. >>>>>>>>> [2] >>>>>>>>> >>>>>>>>> if (pipe != null >>>>>>>>> && !Boolean.TRUE.equals(messageContext >>>>>>>>> >>>>>>>>> .getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && >>>>>>>>> forcePTBuild) { >>>>>>>>> InputStream in = pipe.getInputStream(); >>>>>>>>> >>>>>>>>> Object http_sc = >>>>>>>>> messageContext.getProperty(NhttpConstants.HTTP_SC); >>>>>>>>> if (http_sc != null && http_sc instanceof Integer && >>>>>>>>> http_sc.equals(202)) { >>>>>>>>> >>>>>>>>> messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED, >>>>>>>>> Boolean.TRUE); >>>>>>>>> return; >>>>>>>>> } >>>>>>>>> >>>>>>>>> builldMessage(messageContext, earlyBuild, in); >>>>>>>>> return; >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> <sequence xmlns="http://ws.apache.org/ns/synapse"; >>>>>>>>> name="validator"> >>>>>>>>> <property name="threatType" >>>>>>>>> expression="get-property('threatType')" >>>>>>>>> value="SQL-Injection and XML validator"/> >>>>>>>>> <property> >>>>>>>>> <property> >>>>>>>>> -------- >>>>>>>>> <switch source="get-property('To')"> >>>>>>>>> -------------- >>>>>>>>> </switch> >>>>>>>>> *<class >>>>>>>>> name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>* >>>>>>>>> <property name="enabledCheckPathParams" >>>>>>>>> expression="get-property('enabledCheckPathParams')" value="true"/> >>>>>>>>> <property> >>>>>>>>> <property> >>>>>>>>> -- >>>>>>>>> *<class >>>>>>>>> name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>* >>>>>>>>> <filter source="get-property('threat')" regex=".*error.*"> >>>>>>>>> <then> >>>>>>>>> <sequence key="_threat_fault_"></sequence> >>>>>>>>> </then> >>>>>>>>> </filter> >>>>>>>>> </sequence> >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> Hasunie >>>>>>>>> >>>>>>>>> On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[email protected] >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I'm working on threat protector feature in APIM. We're actually >>>>>>>>>> trying to achieve here is to protect both backend resources and >>>>>>>>>> gateway >>>>>>>>>> from the XML and JSON based attacks. The Balerina based APIM 3 >>>>>>>>>> gateway will >>>>>>>>>> be protected by threat handlers. But In here >>>>>>>>>> APIM 2.1.x we have implemented mediators to achieve it. >>>>>>>>>> >>>>>>>>>> If we allow building the request message at the synapse level, It >>>>>>>>>> will definitely affect the gateway, All the request messages which go >>>>>>>>>> through the mediators are built since the Abstarctemediator is >>>>>>>>>> designed a >>>>>>>>>> way that the isContentAware method always returns true. So we set it >>>>>>>>>> to >>>>>>>>>> false in both XML and JSON validator mediators and allow to parse the >>>>>>>>>> XML request via a third party StAX parser called woodstox hence it >>>>>>>>>> was the >>>>>>>>>> best option among other StAX parsers for threat protection features. >>>>>>>>>> It >>>>>>>>>> will keep counting the given limits and when the limit is exceeded, >>>>>>>>>> It will >>>>>>>>>> terminate the process and throw a meaningful exception. I have >>>>>>>>>> created a >>>>>>>>>> custom threat sequence(thrat_fault) and If a threat is detected by >>>>>>>>>> getting >>>>>>>>>> an exception I configured to direct the response through the custom >>>>>>>>>> error >>>>>>>>>> sequence. >>>>>>>>>> I reuse the same custom sequence which was implemented for the >>>>>>>>>> regex threat protector [1] >>>>>>>>>> >>>>>>>>>> Woodstox parser covers most of the vulnerabilities as in here >>>>>>>>>> >>>>>>>>>> *Vulnerablity:* >>>>>>>>>> >>>>>>>>>> *xml bomb* - DTD disabling >>>>>>>>>> >>>>>>>>>> *external entity attack* - disabling external entities. >>>>>>>>>> >>>>>>>>>> Note : >>>>>>>>>> Apart from the mediator level, The external entity >>>>>>>>>> reference property was disabled from the DOM parsers at the synapse >>>>>>>>>> level >>>>>>>>>> as well. >>>>>>>>>> >>>>>>>>>> import org.apache.xerces.impl.Constants; >>>>>>>>>> >>>>>>>>>> private static final int ENTITY_EXPANSION_LIMIT = 0; >>>>>>>>>> private static final DocumentBuilderFactory >>>>>>>>>> documentBuilderFactory = >>>>>>>>>> DocumentBuilderFactory.newInstance(); >>>>>>>>>> >>>>>>>>>> static { >>>>>>>>>> documentBuilderFactory.setNamespaceAware(true); >>>>>>>>>> documentBuilderFactory.setXIncludeAware(false); >>>>>>>>>> documentBuilderFactory.setExpandEntityReferences(false); >>>>>>>>>> >>>>>>>>>> try { >>>>>>>>>> documentBuilderFactory.setFeature(Constants.SAX_FEATU >>>>>>>>>> RE_PREFIX + >>>>>>>>>> Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false); >>>>>>>>>> } catch (ParserConfigurationException e) { >>>>>>>>>> >>>>>>>>>> *Buffer overflow attack* - by limiting the count of elements, >>>>>>>>>> children and length of attributes/keys/values. >>>>>>>>>> >>>>>>>>>> *woodstox properties:* >>>>>>>>>> dtdEnabled >>>>>>>>>> externalEntitiesEnabled >>>>>>>>>> maxDepth >>>>>>>>>> maxElementCount >>>>>>>>>> maxAttributeCount >>>>>>>>>> maxAttributeLength >>>>>>>>>> entityExpansionLimit >>>>>>>>>> maxChildrenPerElement >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> For thwart cohesive attacks, we use both schema validator and >>>>>>>>>> depth limits. Ideally, only the woodstox validator should detect the >>>>>>>>>> cohesive attacks by exceeding the defined depth limit. But the schema >>>>>>>>>> validator will protect the schema poising attacks in the second step >>>>>>>>>> as >>>>>>>>>> well. >>>>>>>>>> >>>>>>>>>> I observed an issue when It comes to combining each >>>>>>>>>> other(woodstox+ schema validator). We have designed the feature in >>>>>>>>>> such a >>>>>>>>>> way that gets the inputstream from the message context and consumes >>>>>>>>>> it in >>>>>>>>>> the woodstox validator. but in here we have to consume the input >>>>>>>>>> stream >>>>>>>>>> again for the schema validation just after passing through the >>>>>>>>>> woodstox.That was the issue and I tried the following methodologies >>>>>>>>>> to >>>>>>>>>> resolved the issue >>>>>>>>>> >>>>>>>>>> 1. try to get the XML object from the woodstox parser to be able >>>>>>>>>> to avoid using the input stream again. >>>>>>>>>> 2. deep clone the inputstream and use cloned input stream for the >>>>>>>>>> schema validation. >>>>>>>>>> 3. reset, mark the buffered input stream(synapse engine also has >>>>>>>>>> done rest, mark) >>>>>>>>>> >>>>>>>>>> 1st one was taken time and much complex to get the XML object >>>>>>>>>> since Woodstock is based on the StAX parsers and also deep cloning >>>>>>>>>> was not >>>>>>>>>> working properly and experienced the same issue after cloning the >>>>>>>>>> inputstream. But the 3rd option makes life easy so I implemented a >>>>>>>>>> way that >>>>>>>>>> returning the buffered input stream, after doing the rest, mark, >>>>>>>>>> then It >>>>>>>>>> works properly. I went through the RelayUtil message builders [2] >>>>>>>>>> and It >>>>>>>>>> also uses the mark and reset methodology and return InputStream. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I observed another issue once the validator throws an exception, >>>>>>>>>> the server hanged and didn't get any response and getting timeout >>>>>>>>>> issue. I >>>>>>>>>> was able to figure it out and Issue occurred while trying to build >>>>>>>>>> the >>>>>>>>>> request message in Relayutil.buildmessage().But Ideally, If we get >>>>>>>>>> an error >>>>>>>>>> we don't need the request message anymore. As I discussed offline >>>>>>>>>> with the >>>>>>>>>> APIM team, I used the *consumeAndDiscardMessage* method to >>>>>>>>>> discard the request message from the message context and set >>>>>>>>>> *message.builder.invoked >>>>>>>>>> *property to *TRUE. *It needs to be set to avoid sending the >>>>>>>>>> content in pass-through pipe (request message) as the response. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> [1] https://docs.wso2.com/display/AM210/Regular+Expression+T >>>>>>>>>> hreat+Protection+for+API+Gateway >>>>>>>>>> [2] https://github.com/wso2/wso2-synapse/blob/master/modules >>>>>>>>>> /transports/core/nhttp/src/main/java/org/apache/synapse/tran >>>>>>>>>> sport/passthru/util/RelayUtils.java#L121 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>> Hasunie >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Hasunie Adikari* >>>>>>>>>> Senior Software Engineer >>>>>>>>>> WSO2 Inc.; http://wso2.com >>>>>>>>>> lean.enterprise.middleware >>>>>>>>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/ >>>>>>>>>> Mobile:+94713095876 <+94%2071%20309%205876> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Hasunie Adikari* >>>>>>>>> Senior Software Engineer >>>>>>>>> WSO2 Inc.; http://wso2.com >>>>>>>>> lean.enterprise.middleware >>>>>>>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/ >>>>>>>>> Mobile:+94713095876 <+94%2071%20309%205876> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Vinod Kavinda >>>>>>>> Senior Software Engineer >>>>>>>> *WSO2 Inc. - lean . enterprise . middleware <http://www.wso2.com>.* >>>>>>>> Mobile : +94 (0) 712 415544 >>>>>>>> Blog : http://soatechflicks.blogspot.com/ >>>>>>>> [image: http://wso2.com/signature] >>>>>>>> <http://wso2.com/signature> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Dushan Abeyruwan | Architect >>>>>>> Technical Support,MV >>>>>>> PMC Member Apache Synpase >>>>>>> WSO2 Inc. http://wso2.com/ >>>>>>> Blog:*http://www.dushantech.com/ <http://www.dushantech.com/>* >>>>>>> LinkedIn:*https://www.linkedin.com/in/dushanabeyruwan >>>>>>> <https://www.linkedin.com/in/dushanabeyruwan>* >>>>>>> Mobile:(001)408-791-9312 <+1%20408-791-9312> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Isuru Udana* >>>>>> Senior Technical Lead >>>>>> WSO2 Inc.; http://wso2.com >>>>>> email: [email protected] cell: +94 77 3791887 <+94%2077%20379%201887> >>>>>> blog: http://mytecheye.blogspot.com/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Hasunie Adikari* >>>>> Senior Software Engineer >>>>> WSO2 Inc.; http://wso2.com >>>>> lean.enterprise.middleware >>>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/ >>>>> Mobile:+94713095876 <071%20309%205876> >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Isuru Udana* >>>> Senior Technical Lead >>>> WSO2 Inc.; http://wso2.com >>>> email: [email protected] cell: +94 77 3791887 <+94%2077%20379%201887> >>>> blog: http://mytecheye.blogspot.com/ >>>> >>> >>> >>> >>> -- >>> *Hasunie Adikari* >>> Senior Software Engineer >>> WSO2 Inc.; http://wso2.com >>> lean.enterprise.middleware >>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/ >>> Mobile:+94713095876 <+94%2071%20309%205876> >>> >>> >> >> >> -- >> Dushan Abeyruwan | Architect >> Technical Support,MV >> PMC Member Apache Synpase >> WSO2 Inc. http://wso2.com/ >> Blog:*http://www.dushantech.com/ <http://www.dushantech.com/>* >> LinkedIn:*https://www.linkedin.com/in/dushanabeyruwan >> <https://www.linkedin.com/in/dushanabeyruwan>* >> Mobile:(001)408-791-9312 <+1%20408-791-9312> >> >> > > > -- > Dimuthu Leelarathne > Director, Solutions Architecture > > WSO2, Inc. (http://wso2.com) > email: [email protected] > Mobile: +94773661935 <+94%2077%20366%201935> > Blog: http://muthulee.blogspot.com > > Lean . Enterprise . Middleware > -- Dushan Abeyruwan | Architect Technical Support,MV PMC Member Apache Synpase WSO2 Inc. http://wso2.com/ Blog:*http://www.dushantech.com/ <http://www.dushantech.com/>* LinkedIn:*https://www.linkedin.com/in/dushanabeyruwan <https://www.linkedin.com/in/dushanabeyruwan>* Mobile:(001)408-791-9312
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
