Hi Isuru,
As we discussed, I cloned the input stream by consuming the passthrough
pipe as in below.
if (pipe != null) {
bufferedInputStream = new
BufferedInputStream(pipe.getInputStream());
}
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int len;
while ((len = bufferedInputStream.read(buffer)) > -1 ) {
byteArrayOutputStream.write(buffer, 0, len);
}
byteArrayOutputStream.flush();
InputStream is1 = new
ByteArrayInputStream(byteArrayOutputStream.toByteArray());
InputStream is2 = new
ByteArrayInputStream(byteArrayOutputStream.toByteArray());
consume the clones for the validation and set another clone as a
buffereInputstream property in the axis2messagecontext.
BufferedInputStream bufferedInputStreamOriginal = new
BufferedInputStream(inputStreamOriginal);
axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM,
bufferedInputStreamOriginal);
I'm still getting the stream closed issue only for correct the messages
which have been passed through multiple validators. If the validators throw
an exception, the request is getting build and generate the custom response
as expected. It seems like we implemented a way that gets the inputstream
from the passthrough pipe for the content unaware flows. Unless it uses to
get the inputstream from the messagecontext. It was proven once I attached
an empty content aware mediator and test the same flow. I was able to
observe the expected behaviour for the same implementation with the content
aware mediator.
Do we have a way to define cloned input stream as an original
inputstream in passthrough pipe?
Regards
Hasunie
On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <isu...@wso2.com> wrote:
> Hi Dushan,
>
> On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <dus...@wso2.com> wrote:
>
>> Hi Hasunie,
>> Current PTT design would build the message whenever if there is content
>> aware mediator available. However IIRC, I did this *message.builder.the
>> invoked* thing to cope with the WSO2 ELB we had (a few years ago).
>>
> No. I think it was *force.passthrough.builder *property which you
> introduced for ELB requirement.
>
> To be honest, that looks ugly isn it (in terms of overall picture).
>> Basically, what it does; even if there are content-aware mediators, the
>> engine would forcefully ignore that (it was ELB requirement :) ) but for
>> APIM I don't think that would be the same, cos we may have to deal with
>> many use cases sometimes of cause with content aware flows with API
>> compositions etc etc.
>>
>> So, let's think what we can do here; regex and XML threat protectors
>> equally important if security is priority thus, we would no longer able to
>> achieve the same core basic aspect (content none awareness) because, such
>> protections required you to walk through the nodes and verify some aspects
>> (basically, you need to expand the xml node tree to get result set) in that
>> way, it is required the message to be build. Anyway, what I would think the
>> best approach here is not to change complete synapse content awareness
>> logic rather I would think you may have mediator in place but only if such
>> protection engaged that may build the message to get XML inforset (rather
>> build through root, may be you can mark this meditor as content-aware
>> false, then build if message not already build prior to process)
>>
>> IMO, lets just not complicate the what we try to build around message
>> validation. I mean if we need such protection we may need to sacrify some
>> aspects am I?
>>
>> Cheers,
>> Dushan
>>
>> On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <vi...@wso2.com> wrote:
>>
>>> Hi Hasunie,
>>>
>>> This is expected since the synapse engine now expecting an already built
>>> message. If I understood your requirement correctly, one option is to use a
>>> Builder Mediator before using any content aware mediator. Even though we do
>>> not recommend the Builder mediators now, still we can use it for your
>>> specific use case. Or you have to revert the *message.builder.invoked *
>>> property to *false *again*.*
>>>
>>> Regards,
>>> Vinod
>>>
>>> On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <hasu...@wso2.com>
>>> wrote:
>>>
>>>> I'm trying to combine SQL injection(Regex) threat protector with the
>>>> XML threat protector. So I created a sequence[1] with XMLthreatprotector
>>>> mediator and regex mediator consecutively and uploaded it to be able to
>>>> validate the request message through both the xml validator and regex
>>>> validators. If I set the *message.builder.invoked *property to *TRUE *in
>>>> xml validator
>>>> mediator to avoid sending the content in pass-through pipe(request
>>>> message) as the response, Regex mediators is getting failed. The regex
>>>> mediator was designed a way that the incoming messages are built in synapse
>>>> level and eveluate the message content at the mediator level. It seems
>>>> like we can't continue any mediators which are required to get the message
>>>> content, after we manually set the aforementioned property to true in the
>>>> previous mediator. If I set it to true, RelayUtill will skip building the
>>>> message as in here [2]. Any thoughts regarding the issue. I'm currently
>>>> working on the issue to be able to combine regex and XML threat protectors.
>>>> [2]
>>>>
>>>> if (pipe != null
>>>> && !Boolean.TRUE.equals(messageContext
>>>>
>>>> .getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) &&
>>>> forcePTBuild) {
>>>> InputStream in = pipe.getInputStream();
>>>>
>>>> Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
>>>> if (http_sc != null && http_sc instanceof Integer &&
>>>> http_sc.equals(202)) {
>>>>
>>>> messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
>>>> Boolean.TRUE);
>>>> return;
>>>> }
>>>>
>>>> builldMessage(messageContext, earlyBuild, in);
>>>> return;
>>>> }
>>>>
>>>>
>>>>
>>>> [1]
>>>> <sequence xmlns="http://ws.apache.org/ns/synapse"; name="validator">
>>>> <property name="threatType" expression="get-property('threatType')"
>>>> value="SQL-Injection and XML validator"/>
>>>> <property>
>>>> <property>
>>>> --------
>>>> <switch source="get-property('To')">
>>>> --------------
>>>> </switch>
>>>> *<class
>>>> name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>*
>>>> <property name="enabledCheckPathParams"
>>>> expression="get-property('enabledCheckPathParams')" value="true"/>
>>>> <property>
>>>> <property>
>>>> --
>>>> *<class
>>>> name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>*
>>>> <filter source="get-property('threat')" regex=".*error.*">
>>>> <then>
>>>> <sequence key="_threat_fault_"></sequence>
>>>> </then>
>>>> </filter>
>>>> </sequence>
>>>>
>>>> Regards
>>>> Hasunie
>>>>
>>>> On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <hasu...@wso2.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I'm working on threat protector feature in APIM. We're actually trying
>>>>> to achieve here is to protect both backend resources and gateway from
>>>>> the XML and JSON based attacks. The Balerina based APIM 3 gateway will be
>>>>> protected by threat handlers. But In here
>>>>> APIM 2.1.x we have implemented mediators to achieve it.
>>>>>
>>>>> If we allow building the request message at the synapse level, It will
>>>>> definitely affect the gateway, All the request messages which go through
>>>>> the mediators are built since the Abstarctemediator is designed a way that
>>>>> the isContentAware method always returns true. So we set it to false in
>>>>> both XML and JSON validator mediators and allow to parse the XML request
>>>>> via a third party StAX parser called woodstox hence it was the best option
>>>>> among other StAX parsers for threat protection features. It will keep
>>>>> counting the given limits and when the limit is exceeded, It will
>>>>> terminate
>>>>> the process and throw a meaningful exception. I have created a custom
>>>>> threat sequence(thrat_fault) and If a threat is detected by getting an
>>>>> exception I configured to direct the response through the custom error
>>>>> sequence.
>>>>> I reuse the same custom sequence which was implemented for the regex
>>>>> threat protector [1]
>>>>>
>>>>> Woodstox parser covers most of the vulnerabilities as in here
>>>>>
>>>>> *Vulnerablity:*
>>>>>
>>>>> *xml bomb* - DTD disabling
>>>>>
>>>>> *external entity attack* - disabling external entities.
>>>>>
>>>>> Note :
>>>>> Apart from the mediator level, The external entity
>>>>> reference property was disabled from the DOM parsers at the synapse level
>>>>> as well.
>>>>>
>>>>> import org.apache.xerces.impl.Constants;
>>>>>
>>>>> private static final int ENTITY_EXPANSION_LIMIT = 0;
>>>>> private static final DocumentBuilderFactory documentBuilderFactory =
>>>>>
>>>>> DocumentBuilderFactory.newInstance();
>>>>>
>>>>> static {
>>>>> documentBuilderFactory.setNamespaceAware(true);
>>>>> documentBuilderFactory.setXIncludeAware(false);
>>>>> documentBuilderFactory.setExpandEntityReferences(false);
>>>>>
>>>>> try {
>>>>> documentBuilderFactory.setFeature(Constants.SAX_FEATURE_PREFIX
>>>>> +
>>>>> Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
>>>>> } catch (ParserConfigurationException e) {
>>>>>
>>>>> *Buffer overflow attack* - by limiting the count of elements,
>>>>> children and length of attributes/keys/values.
>>>>>
>>>>> *woodstox properties:*
>>>>> dtdEnabled
>>>>> externalEntitiesEnabled
>>>>> maxDepth
>>>>> maxElementCount
>>>>> maxAttributeCount
>>>>> maxAttributeLength
>>>>> entityExpansionLimit
>>>>> maxChildrenPerElement
>>>>>
>>>>>
>>>>> For thwart cohesive attacks, we use both schema validator and depth
>>>>> limits. Ideally, only the woodstox validator should detect the cohesive
>>>>> attacks by exceeding the defined depth limit. But the schema validator
>>>>> will
>>>>> protect the schema poising attacks in the second step as well.
>>>>>
>>>>> I observed an issue when It comes to combining each other(woodstox+
>>>>> schema validator). We have designed the feature in such a way that gets
>>>>> the
>>>>> inputstream from the message context and consumes it in the woodstox
>>>>> validator. but in here we have to consume the input stream again for the
>>>>> schema validation just after passing through the woodstox.That was the
>>>>> issue and I tried the following methodologies to resolved the issue
>>>>>
>>>>> 1. try to get the XML object from the woodstox parser to be able to
>>>>> avoid using the input stream again.
>>>>> 2. deep clone the inputstream and use cloned input stream for the
>>>>> schema validation.
>>>>> 3. reset, mark the buffered input stream(synapse engine also has done
>>>>> rest, mark)
>>>>>
>>>>> 1st one was taken time and much complex to get the XML object since
>>>>> Woodstock is based on the StAX parsers and also deep cloning was not
>>>>> working properly and experienced the same issue after cloning the
>>>>> inputstream. But the 3rd option makes life easy so I implemented a way
>>>>> that
>>>>> returning the buffered input stream, after doing the rest, mark, then It
>>>>> works properly. I went through the RelayUtil message builders [2] and It
>>>>> also uses the mark and reset methodology and return InputStream.
>>>>>
>>>>>
>>>>> I observed another issue once the validator throws an exception, the
>>>>> server hanged and didn't get any response and getting timeout issue. I was
>>>>> able to figure it out and Issue occurred while trying to build the request
>>>>> message in Relayutil.buildmessage().But Ideally, If we get an error we
>>>>> don't need the request message anymore. As I discussed offline with the
>>>>> APIM team, I used the *consumeAndDiscardMessage* method to discard
>>>>> the request message from the message context and set
>>>>> *message.builder.invoked
>>>>> *property to *TRUE. *It needs to be set to avoid sending the content
>>>>> in pass-through pipe (request message) as the response.
>>>>>
>>>>>
>>>>> [1] https://docs.wso2.com/display/AM210/Regular+Expression+T
>>>>> hreat+Protection+for+API+Gateway
>>>>> [2] https://github.com/wso2/wso2-synapse/blob/master/modules
>>>>> /transports/core/nhttp/src/main/java/org/apache/synapse/tran
>>>>> sport/passthru/util/RelayUtils.java#L121
>>>>>
>>>>>
>>>>> Regards
>>>>> Hasunie
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Hasunie Adikari*
>>>>> Senior Software Engineer
>>>>> WSO2 Inc.; http://wso2.com
>>>>> lean.enterprise.middleware
>>>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/
>>>>> Mobile:+94713095876 <+94%2071%20309%205876>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Hasunie Adikari*
>>>> Senior Software Engineer
>>>> WSO2 Inc.; http://wso2.com
>>>> lean.enterprise.middleware
>>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/
>>>> Mobile:+94713095876 <+94%2071%20309%205876>
>>>>
>>>>
>>>
>>>
>>> --
>>> Vinod Kavinda
>>> Senior Software Engineer
>>> *WSO2 Inc. - lean . enterprise . middleware <http://www.wso2.com>.*
>>> Mobile : +94 (0) 712 415544
>>> Blog : http://soatechflicks.blogspot.com/
>>> [image: http://wso2.com/signature]
>>> <http://wso2.com/signature>
>>>
>>>
>>
>>
>> --
>> Dushan Abeyruwan | Architect
>> Technical Support,MV
>> PMC Member Apache Synpase
>> WSO2 Inc. http://wso2.com/
>> Blog:*http://www.dushantech.com/ <http://www.dushantech.com/>*
>> LinkedIn:*https://www.linkedin.com/in/dushanabeyruwan
>> <https://www.linkedin.com/in/dushanabeyruwan>*
>> Mobile:(001)408-791-9312 <+1%20408-791-9312>
>>
>>
>
>
> --
> *Isuru Udana*
> Senior Technical Lead
> WSO2 Inc.; http://wso2.com
> email: isu...@wso2.com cell: +94 77 3791887 <+94%2077%20379%201887>
> blog: http://mytecheye.blogspot.com/
>
--
*Hasunie Adikari*
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/
Mobile:+94713095876
_______________________________________________
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
