Hi all,
I encounter an issue while writing a unit test to clone the input stream. I
implemented the cloning method[1] a way that gets the input stream from the
passthrough pipe. As I discussed with the EI team, We can't mock the pipe
and also there were some obstacles for creating the pipe and I addressed
the issue in the same mail thread. So I am gonna create packeage private
clone method to continue the unit test flow.
[1] Pipe pipe = (Pipe)
axis2MC.getProperty(PassThroughConstants.PASS_THROUGH_PIPE);
BufferedInputStream bufferedInputStream = null;
Map<String, InputStream> inputStreams = null;
InputStream inputStreamSchema;
InputStream inputStreamXml;
---------
if (pipe != null) {
bufferedInputStream = new
BufferedInputStream(pipe.getInputStream());
}
if (bufferedInputStream != null) {
ByteArrayOutputStream byteArrayOutputStreamSchema = new
ByteArrayOutputStream();
byte[] buffer = new byte[1024];
int length;
while ((length = bufferedInputStream.read(buffer)) > -1) {
byteArrayOutputStreamSchema.write(buffer, 0, length);
}
byteArrayOutputStreamSchema.flush();
inputStreamMap = new HashMap<>();
inputStreamSchema = new
ByteArrayInputStream(byteArrayOutputStreamSchema.toByteArray());
inputStreamXml = new
ByteArrayInputStream(byteArrayOutputStreamSchema.toByteArray());
inputStreamSql = new
ByteArrayInputStream(byteArrayOutputStreamSchema.toByteArray());
------
inputStreams.put("Schema", inputStreamSchema);
inputStreams.put("XML", inputStreamXml);
---
}
Regards
Hasunie
On Tue, Jan 9, 2018 at 11:08 PM, Hasunie Adikari <[email protected]> wrote:
> Hi all,
>
> As I discussed with Isuru, There are some possible approaches to overcome
> the issue.
>
> 1. Create a new pass through pipe.
> - The data will be written to the pipe by a spawned thread and current
> thread will be consuming the data and continuing the message flow. We went
> through the pipe creation logic and seemed it tightly coupled with
> encoding, decoding methodologies so that it can't be
> implemented at APIM level.
>
> 2. Change the synapse level logic to get the ByteArrayInputstream and
> write it into the response.
> - It can be but have to thoroughly go through and do it carefully
> unless the default message flow would be affected.
>
> 3. Build the message by invoking RelayUtils.buildMessage() at the
> validator mediator after successfully parsing all the validators.
> - It will be slightly affected the performance but this is the
> straightforward solution at this moment.
>
> I have improved the code by applying the 3rd option as we discussed.
> Setting the PassThroughConstants.BUFFERED_INPUT_STREAM has an effect now
> onwards since we changed a way that building the message to achieve the
> content aware behavior which seeks the
> inputream from the axis2 message context instead of the original
> inputstream.
>
>
> Regards
> Hasunie
>
>
>
> On Tue, Jan 9, 2018 at 4:41 PM, Isuru Udana <[email protected]> wrote:
>
>> Hi Hasunie,
>>
>> As we discussed, setting the PassThroughConstants.BUFFERED_INPUT_STREAM
>> has no effect on the flow in this case and Passthough Sender still seek
>> content from the original input stream which got empty due to this cloning
>> logic. That's the reason for this behaviour.
>>
>> Thanks.
>>
>>
>>
>> On Tue, Jan 9, 2018 at 11:43 AM, Hasunie Adikari <[email protected]>
>> wrote:
>>
>>> Hi Isuru,
>>>
>>> As we discussed, I cloned the input stream by consuming the passthrough
>>> pipe as in below.
>>>
>>>
>>> if (pipe != null) {
>>> bufferedInputStream = new BufferedInputStream(pipe.getIn
>>> putStream());
>>>
>>> }
>>> ByteArrayOutputStream byteArrayOutputStream = new
>>> ByteArrayOutputStream();
>>> byte[] buffer = new byte[1024];
>>> int len;
>>> while ((len = bufferedInputStream.read(buffer)) > -1 ) {
>>> byteArrayOutputStream.write(buffer, 0, len);
>>> }
>>> byteArrayOutputStream.flush();
>>>
>>>
>>> InputStream is1 = new ByteArrayInputStream(byteArray
>>> OutputStream.toByteArray());
>>> InputStream is2 = new ByteArrayInputStream(byteArray
>>> OutputStream.toByteArray());
>>>
>>> consume the clones for the validation and set another clone as a
>>> buffereInputstream property in the axis2messagecontext.
>>>
>>> BufferedInputStream bufferedInputStreamOriginal = new
>>> BufferedInputStream(inputStreamOriginal);
>>> axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM,
>>> bufferedInputStreamOriginal);
>>>
>>> I'm still getting the stream closed issue only for correct the messages
>>> which have been passed through multiple validators. If the validators throw
>>> an exception, the request is getting build and generate the custom response
>>> as expected. It seems like we implemented a way that gets the
>>> inputstream from the passthrough pipe for the content unaware flows.
>>> Unless it uses to get the inputstream from the messagecontext. It was
>>> proven once I attached an empty content aware mediator and test the
>>> same flow. I was able to observe the expected behaviour for the same
>>> implementation with the content aware mediator.
>>>
>>> Do we have a way to define cloned input stream as an original
>>> inputstream in passthrough pipe?
>>>
>>>
>>> Regards
>>> Hasunie
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <[email protected]> wrote:
>>>
>>>> Hi Dushan,
>>>>
>>>> On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[email protected]>
>>>> wrote:
>>>>
>>>>> Hi Hasunie,
>>>>> Current PTT design would build the message whenever if there is
>>>>> content aware mediator available. However IIRC, I did this
>>>>> *message.builder.the
>>>>> invoked* thing to cope with the WSO2 ELB we had (a few years ago).
>>>>>
>>>> No. I think it was *force.passthrough.builder *property which you
>>>> introduced for ELB requirement.
>>>>
>>>> To be honest, that looks ugly isn it (in terms of overall picture).
>>>>> Basically, what it does; even if there are content-aware mediators, the
>>>>> engine would forcefully ignore that (it was ELB requirement :) ) but for
>>>>> APIM I don't think that would be the same, cos we may have to deal with
>>>>> many use cases sometimes of cause with content aware flows with API
>>>>> compositions etc etc.
>>>>>
>>>>> So, let's think what we can do here; regex and XML threat protectors
>>>>> equally important if security is priority thus, we would no longer able
>>>>> to
>>>>> achieve the same core basic aspect (content none awareness) because, such
>>>>> protections required you to walk through the nodes and verify some aspects
>>>>> (basically, you need to expand the xml node tree to get result set) in
>>>>> that
>>>>> way, it is required the message to be build. Anyway, what I would think
>>>>> the
>>>>> best approach here is not to change complete synapse content awareness
>>>>> logic rather I would think you may have mediator in place but only if such
>>>>> protection engaged that may build the message to get XML inforset (rather
>>>>> build through root, may be you can mark this meditor as content-aware
>>>>> false, then build if message not already build prior to process)
>>>>>
>>>>> IMO, lets just not complicate the what we try to build around message
>>>>> validation. I mean if we need such protection we may need to sacrify some
>>>>> aspects am I?
>>>>>
>>>>> Cheers,
>>>>> Dushan
>>>>>
>>>>> On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[email protected]> wrote:
>>>>>
>>>>>> Hi Hasunie,
>>>>>>
>>>>>> This is expected since the synapse engine now expecting an already
>>>>>> built message. If I understood your requirement correctly, one option is
>>>>>> to
>>>>>> use a Builder Mediator before using any content aware mediator. Even
>>>>>> though
>>>>>> we do not recommend the Builder mediators now, still we can use it for
>>>>>> your
>>>>>> specific use case. Or you have to revert the
>>>>>> *message.builder.invoked *property to *false *again*.*
>>>>>>
>>>>>> Regards,
>>>>>> Vinod
>>>>>>
>>>>>> On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> I'm trying to combine SQL injection(Regex) threat protector with the
>>>>>>> XML threat protector. So I created a sequence[1] with XMLthreatprotector
>>>>>>> mediator and regex mediator consecutively and uploaded it to be able to
>>>>>>> validate the request message through both the xml validator and regex
>>>>>>> validators. If I set the *message.builder.invoked *property to *TRUE
>>>>>>> *in xml validator mediator to avoid sending the content in
>>>>>>> pass-through pipe(request message) as the response, Regex mediators is
>>>>>>> getting failed. The regex mediator was designed a way that the incoming
>>>>>>> messages are built in synapse level and eveluate the message
>>>>>>> content at the mediator level. It seems like we can't continue any
>>>>>>> mediators which are required to get the message content, after we
>>>>>>> manually
>>>>>>> set the aforementioned property to true in the previous mediator. If I
>>>>>>> set
>>>>>>> it to true, RelayUtill will skip building the message as in here [2].
>>>>>>> Any
>>>>>>> thoughts regarding the issue. I'm currently working on the issue to be
>>>>>>> able
>>>>>>> to combine regex and XML threat protectors.
>>>>>>> [2]
>>>>>>>
>>>>>>> if (pipe != null
>>>>>>> && !Boolean.TRUE.equals(messageContext
>>>>>>>
>>>>>>> .getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) &&
>>>>>>> forcePTBuild) {
>>>>>>> InputStream in = pipe.getInputStream();
>>>>>>>
>>>>>>> Object http_sc = messageContext.getProperty(NhttpConstants.HTTP_SC);
>>>>>>> if (http_sc != null && http_sc instanceof Integer &&
>>>>>>> http_sc.equals(202)) {
>>>>>>>
>>>>>>> messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
>>>>>>> Boolean.TRUE);
>>>>>>> return;
>>>>>>> }
>>>>>>>
>>>>>>> builldMessage(messageContext, earlyBuild, in);
>>>>>>> return;
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> [1]
>>>>>>> <sequence xmlns="http://ws.apache.org/ns/synapse"; name="validator">
>>>>>>> <property name="threatType" expression="get-property('threatType')"
>>>>>>> value="SQL-Injection and XML validator"/>
>>>>>>> <property>
>>>>>>> <property>
>>>>>>> --------
>>>>>>> <switch source="get-property('To')">
>>>>>>> --------------
>>>>>>> </switch>
>>>>>>> *<class
>>>>>>> name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>*
>>>>>>> <property name="enabledCheckPathParams"
>>>>>>> expression="get-property('enabledCheckPathParams')" value="true"/>
>>>>>>> <property>
>>>>>>> <property>
>>>>>>> --
>>>>>>> *<class
>>>>>>> name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>*
>>>>>>> <filter source="get-property('threat')" regex=".*error.*">
>>>>>>> <then>
>>>>>>> <sequence key="_threat_fault_"></sequence>
>>>>>>> </then>
>>>>>>> </filter>
>>>>>>> </sequence>
>>>>>>>
>>>>>>> Regards
>>>>>>> Hasunie
>>>>>>>
>>>>>>> On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I'm working on threat protector feature in APIM. We're actually
>>>>>>>> trying to achieve here is to protect both backend resources and gateway
>>>>>>>> from the XML and JSON based attacks. The Balerina based APIM 3 gateway
>>>>>>>> will
>>>>>>>> be protected by threat handlers. But In here
>>>>>>>> APIM 2.1.x we have implemented mediators to achieve it.
>>>>>>>>
>>>>>>>> If we allow building the request message at the synapse level, It
>>>>>>>> will definitely affect the gateway, All the request messages which go
>>>>>>>> through the mediators are built since the Abstarctemediator is
>>>>>>>> designed a
>>>>>>>> way that the isContentAware method always returns true. So we set it to
>>>>>>>> false in both XML and JSON validator mediators and allow to parse the
>>>>>>>> XML request via a third party StAX parser called woodstox hence it was
>>>>>>>> the
>>>>>>>> best option among other StAX parsers for threat protection features. It
>>>>>>>> will keep counting the given limits and when the limit is exceeded, It
>>>>>>>> will
>>>>>>>> terminate the process and throw a meaningful exception. I have created
>>>>>>>> a
>>>>>>>> custom threat sequence(thrat_fault) and If a threat is detected by
>>>>>>>> getting
>>>>>>>> an exception I configured to direct the response through the custom
>>>>>>>> error
>>>>>>>> sequence.
>>>>>>>> I reuse the same custom sequence which was implemented for the
>>>>>>>> regex threat protector [1]
>>>>>>>>
>>>>>>>> Woodstox parser covers most of the vulnerabilities as in here
>>>>>>>>
>>>>>>>> *Vulnerablity:*
>>>>>>>>
>>>>>>>> *xml bomb* - DTD disabling
>>>>>>>>
>>>>>>>> *external entity attack* - disabling external entities.
>>>>>>>>
>>>>>>>> Note :
>>>>>>>> Apart from the mediator level, The external entity
>>>>>>>> reference property was disabled from the DOM parsers at the synapse
>>>>>>>> level
>>>>>>>> as well.
>>>>>>>>
>>>>>>>> import org.apache.xerces.impl.Constants;
>>>>>>>>
>>>>>>>> private static final int ENTITY_EXPANSION_LIMIT = 0;
>>>>>>>> private static final DocumentBuilderFactory documentBuilderFactory
>>>>>>>> =
>>>>>>>> DocumentBuilderFactory.newInstance();
>>>>>>>>
>>>>>>>> static {
>>>>>>>> documentBuilderFactory.setNamespaceAware(true);
>>>>>>>> documentBuilderFactory.setXIncludeAware(false);
>>>>>>>> documentBuilderFactory.setExpandEntityReferences(false);
>>>>>>>>
>>>>>>>> try {
>>>>>>>> documentBuilderFactory.setFeature(Constants.SAX_FEATU
>>>>>>>> RE_PREFIX +
>>>>>>>> Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
>>>>>>>> } catch (ParserConfigurationException e) {
>>>>>>>>
>>>>>>>> *Buffer overflow attack* - by limiting the count of elements,
>>>>>>>> children and length of attributes/keys/values.
>>>>>>>>
>>>>>>>> *woodstox properties:*
>>>>>>>> dtdEnabled
>>>>>>>> externalEntitiesEnabled
>>>>>>>> maxDepth
>>>>>>>> maxElementCount
>>>>>>>> maxAttributeCount
>>>>>>>> maxAttributeLength
>>>>>>>> entityExpansionLimit
>>>>>>>> maxChildrenPerElement
>>>>>>>>
>>>>>>>>
>>>>>>>> For thwart cohesive attacks, we use both schema validator and depth
>>>>>>>> limits. Ideally, only the woodstox validator should detect the cohesive
>>>>>>>> attacks by exceeding the defined depth limit. But the schema validator
>>>>>>>> will
>>>>>>>> protect the schema poising attacks in the second step as well.
>>>>>>>>
>>>>>>>> I observed an issue when It comes to combining each other(woodstox+
>>>>>>>> schema validator). We have designed the feature in such a way that
>>>>>>>> gets the
>>>>>>>> inputstream from the message context and consumes it in the woodstox
>>>>>>>> validator. but in here we have to consume the input stream again for
>>>>>>>> the
>>>>>>>> schema validation just after passing through the woodstox.That was the
>>>>>>>> issue and I tried the following methodologies to resolved the issue
>>>>>>>>
>>>>>>>> 1. try to get the XML object from the woodstox parser to be able to
>>>>>>>> avoid using the input stream again.
>>>>>>>> 2. deep clone the inputstream and use cloned input stream for the
>>>>>>>> schema validation.
>>>>>>>> 3. reset, mark the buffered input stream(synapse engine also has
>>>>>>>> done rest, mark)
>>>>>>>>
>>>>>>>> 1st one was taken time and much complex to get the XML object since
>>>>>>>> Woodstock is based on the StAX parsers and also deep cloning was not
>>>>>>>> working properly and experienced the same issue after cloning the
>>>>>>>> inputstream. But the 3rd option makes life easy so I implemented a way
>>>>>>>> that
>>>>>>>> returning the buffered input stream, after doing the rest, mark, then
>>>>>>>> It
>>>>>>>> works properly. I went through the RelayUtil message builders [2] and
>>>>>>>> It
>>>>>>>> also uses the mark and reset methodology and return InputStream.
>>>>>>>>
>>>>>>>>
>>>>>>>> I observed another issue once the validator throws an exception,
>>>>>>>> the server hanged and didn't get any response and getting timeout
>>>>>>>> issue. I
>>>>>>>> was able to figure it out and Issue occurred while trying to build the
>>>>>>>> request message in Relayutil.buildmessage().But Ideally, If we get an
>>>>>>>> error
>>>>>>>> we don't need the request message anymore. As I discussed offline with
>>>>>>>> the
>>>>>>>> APIM team, I used the *consumeAndDiscardMessage* method to discard
>>>>>>>> the request message from the message context and set
>>>>>>>> *message.builder.invoked
>>>>>>>> *property to *TRUE. *It needs to be set to avoid sending the
>>>>>>>> content in pass-through pipe (request message) as the response.
>>>>>>>>
>>>>>>>>
>>>>>>>> [1] https://docs.wso2.com/display/AM210/Regular+Expression+T
>>>>>>>> hreat+Protection+for+API+Gateway
>>>>>>>> [2] https://github.com/wso2/wso2-synapse/blob/master/modules
>>>>>>>> /transports/core/nhttp/src/main/java/org/apache/synapse/tran
>>>>>>>> sport/passthru/util/RelayUtils.java#L121
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Hasunie
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Hasunie Adikari*
>>>>>>>> Senior Software Engineer
>>>>>>>> WSO2 Inc.; http://wso2.com
>>>>>>>> lean.enterprise.middleware
>>>>>>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/
>>>>>>>> Mobile:+94713095876 <+94%2071%20309%205876>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Hasunie Adikari*
>>>>>>> Senior Software Engineer
>>>>>>> WSO2 Inc.; http://wso2.com
>>>>>>> lean.enterprise.middleware
>>>>>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/
>>>>>>> Mobile:+94713095876 <+94%2071%20309%205876>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Vinod Kavinda
>>>>>> Senior Software Engineer
>>>>>> *WSO2 Inc. - lean . enterprise . middleware <http://www.wso2.com>.*
>>>>>> Mobile : +94 (0) 712 415544
>>>>>> Blog : http://soatechflicks.blogspot.com/
>>>>>> [image: http://wso2.com/signature]
>>>>>> <http://wso2.com/signature>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dushan Abeyruwan | Architect
>>>>> Technical Support,MV
>>>>> PMC Member Apache Synpase
>>>>> WSO2 Inc. http://wso2.com/
>>>>> Blog:*http://www.dushantech.com/ <http://www.dushantech.com/>*
>>>>> LinkedIn:*https://www.linkedin.com/in/dushanabeyruwan
>>>>> <https://www.linkedin.com/in/dushanabeyruwan>*
>>>>> Mobile:(001)408-791-9312 <+1%20408-791-9312>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Isuru Udana*
>>>> Senior Technical Lead
>>>> WSO2 Inc.; http://wso2.com
>>>> email: [email protected] cell: +94 77 3791887 <+94%2077%20379%201887>
>>>> blog: http://mytecheye.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> *Hasunie Adikari*
>>> Senior Software Engineer
>>> WSO2 Inc.; http://wso2.com
>>> lean.enterprise.middleware
>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/
>>> Mobile:+94713095876 <071%20309%205876>
>>>
>>>
>>
>>
>> --
>> *Isuru Udana*
>> Senior Technical Lead
>> WSO2 Inc.; http://wso2.com
>> email: [email protected] cell: +94 77 3791887 <+94%2077%20379%201887>
>> blog: http://mytecheye.blogspot.com/
>>
>
>
>
> --
> *Hasunie Adikari*
> Senior Software Engineer
> WSO2 Inc.; http://wso2.com
> lean.enterprise.middleware
> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/
> Mobile:+94713095876 <+94%2071%20309%205876>
>
>
--
*Hasunie Adikari*
Senior Software Engineer
WSO2 Inc.; http://wso2.com
lean.enterprise.middleware
blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/
Mobile:+94713095876
_______________________________________________
Architecture mailing list
[email protected]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
