Hi Dushan, On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[email protected]> wrote:
> Hi Hasunie, > Current PTT design would build the message whenever if there is content > aware mediator available. However IIRC, I did this *message.builder.the > invoked* thing to cope with the WSO2 ELB we had (a few years ago). > No. I think it was *force.passthrough.builder *property which you introduced for ELB requirement. To be honest, that looks ugly isn it (in terms of overall picture). > Basically, what it does; even if there are content-aware mediators, the > engine would forcefully ignore that (it was ELB requirement :) ) but for > APIM I don't think that would be the same, cos we may have to deal with > many use cases sometimes of cause with content aware flows with API > compositions etc etc. > > So, let's think what we can do here; regex and XML threat protectors > equally important if security is priority thus, we would no longer able to > achieve the same core basic aspect (content none awareness) because, such > protections required you to walk through the nodes and verify some aspects > (basically, you need to expand the xml node tree to get result set) in that > way, it is required the message to be build. Anyway, what I would think the > best approach here is not to change complete synapse content awareness > logic rather I would think you may have mediator in place but only if such > protection engaged that may build the message to get XML inforset (rather > build through root, may be you can mark this meditor as content-aware > false, then build if message not already build prior to process) > > IMO, lets just not complicate the what we try to build around message > validation. I mean if we need such protection we may need to sacrify some > aspects am I? > > Cheers, > Dushan > > On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[email protected]> wrote: > >> Hi Hasunie, >> >> This is expected since the synapse engine now expecting an already built >> message. If I understood your requirement correctly, one option is to use a >> Builder Mediator before using any content aware mediator. Even though we do >> not recommend the Builder mediators now, still we can use it for your >> specific use case. Or you have to revert the *message.builder.invoked *pr >> operty to *false *again*.* >> >> Regards, >> Vinod >> >> On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[email protected]> wrote: >> >>> I'm trying to combine SQL injection(Regex) threat protector with the XML >>> threat protector. So I created a sequence[1] with XMLthreatprotector >>> mediator and regex mediator consecutively and uploaded it to be able to >>> validate the request message through both the xml validator and regex >>> validators. If I set the *message.builder.invoked *property to *TRUE *in >>> xml validator >>> mediator to avoid sending the content in pass-through pipe(request >>> message) as the response, Regex mediators is getting failed. The regex >>> mediator was designed a way that the incoming messages are built in synapse >>> level and eveluate the message content at the mediator level. It seems >>> like we can't continue any mediators which are required to get the message >>> content, after we manually set the aforementioned property to true in the >>> previous mediator. If I set it to true, RelayUtill will skip building the >>> message as in here [2]. Any thoughts regarding the issue. I'm currently >>> working on the issue to be able to combine regex and XML threat protectors. >>> [2] >>> >>> if (pipe != null >>> && !Boolean.TRUE.equals(messageContext >>> >>> .getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) && >>> forcePTBuild) { >>> InputStream in = pipe.getInputStream(); >>> >>> Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC); >>> if (http_sc != null && http_sc instanceof Integer && >>> http_sc.equals(202)) { >>> >>> messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED, >>> Boolean.TRUE); >>> return; >>> } >>> >>> builldMessage(messageContext, earlyBuild, in); >>> return; >>> } >>> >>> >>> >>> [1] >>> <sequence xmlns="http://ws.apache.org/ns/synapse"; name="validator"> >>> <property name="threatType" expression="get-property('threatType')" >>> value="SQL-Injection and XML validator"/> >>> <property> >>> <property> >>> -------- >>> <switch source="get-property('To')"> >>> -------------- >>> </switch> >>> *<class >>> name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>* >>> <property name="enabledCheckPathParams" >>> expression="get-property('enabledCheckPathParams')" value="true"/> >>> <property> >>> <property> >>> -- >>> *<class >>> name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>* >>> <filter source="get-property('threat')" regex=".*error.*"> >>> <then> >>> <sequence key="_threat_fault_"></sequence> >>> </then> >>> </filter> >>> </sequence> >>> >>> Regards >>> Hasunie >>> >>> On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[email protected]> >>> wrote: >>> >>>> Hi all, >>>> >>>> I'm working on threat protector feature in APIM. We're actually trying >>>> to achieve here is to protect both backend resources and gateway from >>>> the XML and JSON based attacks. The Balerina based APIM 3 gateway will be >>>> protected by threat handlers. But In here >>>> APIM 2.1.x we have implemented mediators to achieve it. >>>> >>>> If we allow building the request message at the synapse level, It will >>>> definitely affect the gateway, All the request messages which go through >>>> the mediators are built since the Abstarctemediator is designed a way that >>>> the isContentAware method always returns true. So we set it to false in >>>> both XML and JSON validator mediators and allow to parse the XML request >>>> via a third party StAX parser called woodstox hence it was the best option >>>> among other StAX parsers for threat protection features. It will keep >>>> counting the given limits and when the limit is exceeded, It will terminate >>>> the process and throw a meaningful exception. I have created a custom >>>> threat sequence(thrat_fault) and If a threat is detected by getting an >>>> exception I configured to direct the response through the custom error >>>> sequence. >>>> I reuse the same custom sequence which was implemented for the regex >>>> threat protector [1] >>>> >>>> Woodstox parser covers most of the vulnerabilities as in here >>>> >>>> *Vulnerablity:* >>>> >>>> *xml bomb* - DTD disabling >>>> >>>> *external entity attack* - disabling external entities. >>>> >>>> Note : >>>> Apart from the mediator level, The external entity >>>> reference property was disabled from the DOM parsers at the synapse level >>>> as well. >>>> >>>> import org.apache.xerces.impl.Constants; >>>> >>>> private static final int ENTITY_EXPANSION_LIMIT = 0; >>>> private static final DocumentBuilderFactory documentBuilderFactory = >>>> >>>> DocumentBuilderFactory.newInstance(); >>>> >>>> static { >>>> documentBuilderFactory.setNamespaceAware(true); >>>> documentBuilderFactory.setXIncludeAware(false); >>>> documentBuilderFactory.setExpandEntityReferences(false); >>>> >>>> try { >>>> documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX + >>>> Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false); >>>> } catch (ParserConfigurationException e) { >>>> >>>> *Buffer overflow attack* - by limiting the count of elements, children >>>> and length of attributes/keys/values. >>>> >>>> *woodstox properties:* >>>> dtdEnabled >>>> externalEntitiesEnabled >>>> maxDepth >>>> maxElementCount >>>> maxAttributeCount >>>> maxAttributeLength >>>> entityExpansionLimit >>>> maxChildrenPerElement >>>> >>>> >>>> For thwart cohesive attacks, we use both schema validator and depth >>>> limits. Ideally, only the woodstox validator should detect the cohesive >>>> attacks by exceeding the defined depth limit. But the schema validator will >>>> protect the schema poising attacks in the second step as well. >>>> >>>> I observed an issue when It comes to combining each other(woodstox+ >>>> schema validator). We have designed the feature in such a way that gets the >>>> inputstream from the message context and consumes it in the woodstox >>>> validator. but in here we have to consume the input stream again for the >>>> schema validation just after passing through the woodstox.That was the >>>> issue and I tried the following methodologies to resolved the issue >>>> >>>> 1. try to get the XML object from the woodstox parser to be able to >>>> avoid using the input stream again. >>>> 2. deep clone the inputstream and use cloned input stream for the >>>> schema validation. >>>> 3. reset, mark the buffered input stream(synapse engine also has done >>>> rest, mark) >>>> >>>> 1st one was taken time and much complex to get the XML object since >>>> Woodstock is based on the StAX parsers and also deep cloning was not >>>> working properly and experienced the same issue after cloning the >>>> inputstream. But the 3rd option makes life easy so I implemented a way that >>>> returning the buffered input stream, after doing the rest, mark, then It >>>> works properly. I went through the RelayUtil message builders [2] and It >>>> also uses the mark and reset methodology and return InputStream. >>>> >>>> >>>> I observed another issue once the validator throws an exception, the >>>> server hanged and didn't get any response and getting timeout issue. I was >>>> able to figure it out and Issue occurred while trying to build the request >>>> message in Relayutil.buildmessage().But Ideally, If we get an error we >>>> don't need the request message anymore. As I discussed offline with the >>>> APIM team, I used the *consumeAndDiscardMessage* method to discard the >>>> request message from the message context and set *message.builder.invoked >>>> *property to *TRUE. *It needs to be set to avoid sending the content >>>> in pass-through pipe (request message) as the response. >>>> >>>> >>>> [1] https://docs.wso2.com/display/AM210/Regular+Expression+T >>>> hreat+Protection+for+API+Gateway >>>> [2] https://github.com/wso2/wso2-synapse/blob/master/modules >>>> /transports/core/nhttp/src/main/java/org/apache/synapse/tran >>>> sport/passthru/util/RelayUtils.java#L121 >>>> >>>> >>>> Regards >>>> Hasunie >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> *Hasunie Adikari* >>>> Senior Software Engineer >>>> WSO2 Inc.; http://wso2.com >>>> lean.enterprise.middleware >>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/ >>>> Mobile:+94713095876 <+94%2071%20309%205876> >>>> >>>> >>> >>> >>> -- >>> *Hasunie Adikari* >>> Senior Software Engineer >>> WSO2 Inc.; http://wso2.com >>> lean.enterprise.middleware >>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/ >>> Mobile:+94713095876 <+94%2071%20309%205876> >>> >>> >> >> >> -- >> Vinod Kavinda >> Senior Software Engineer >> *WSO2 Inc. - lean . enterprise . middleware <http://www.wso2.com>.* >> Mobile : +94 (0) 712 415544 >> Blog : http://soatechflicks.blogspot.com/ >> [image: http://wso2.com/signature] >> <http://wso2.com/signature> >> >> > > > -- > Dushan Abeyruwan | Architect > Technical Support,MV > PMC Member Apache Synpase > WSO2 Inc. http://wso2.com/ > Blog:*http://www.dushantech.com/ <http://www.dushantech.com/>* > LinkedIn:*https://www.linkedin.com/in/dushanabeyruwan > <https://www.linkedin.com/in/dushanabeyruwan>* > Mobile:(001)408-791-9312 <+1%20408-791-9312> > > -- *Isuru Udana* Senior Technical Lead WSO2 Inc.; http://wso2.com email: [email protected] cell: +94 77 3791887 blog: http://mytecheye.blogspot.com/
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
