On Fri, Jan 12, 2018 at 3:21 PM, Hasunie Adikari <[email protected]> wrote:
> Hi all,
>
> I encounter an issue while writing a unit test to clone the input stream.
> I implemented the cloning method[1] a way that gets the input stream from
> the passthrough pipe. As I discussed with the EI team, We can't mock the
> pipe and also there were some obstacles for creating the pipe and I
> addressed the issue in the same mail thread. So I am gonna create packeage
> private clone method to continue the unit test flow.
>
+1. That is one reason for test classes and implementation to have the same
package.
thanks,
Dimuthu
> [1] Pipe pipe = (Pipe) axis2MC.getProperty(PassThroughConstants.PASS_
> THROUGH_PIPE);
> BufferedInputStream bufferedInputStream = null;
> Map<String, InputStream> inputStreams = null;
> InputStream inputStreamSchema;
> InputStream inputStreamXml;
> ---------
> if (pipe != null) {
> bufferedInputStream = new BufferedInputStream(pipe.
> getInputStream());
> }
> if (bufferedInputStream != null) {
> ByteArrayOutputStream byteArrayOutputStreamSchema = new
> ByteArrayOutputStream();
> byte[] buffer = new byte[1024];
> int length;
>
> while ((length = bufferedInputStream.read(buffer)) > -1) {
> byteArrayOutputStreamSchema.write(buffer, 0, length);
> }
> byteArrayOutputStreamSchema.flush();
> inputStreamMap = new HashMap<>();
> inputStreamSchema = new ByteArrayInputStream(
> byteArrayOutputStreamSchema.toByteArray());
> inputStreamXml = new ByteArrayInputStream(
> byteArrayOutputStreamSchema.toByteArray());
> inputStreamSql = new ByteArrayInputStream(
> byteArrayOutputStreamSchema.toByteArray());
> ------
> inputStreams.put("Schema", inputStreamSchema);
> inputStreams.put("XML", inputStreamXml);
> ---
> }
>
> Regards
> Hasunie
>
> On Tue, Jan 9, 2018 at 11:08 PM, Hasunie Adikari <[email protected]> wrote:
>
>> Hi all,
>>
>> As I discussed with Isuru, There are some possible approaches to overcome
>> the issue.
>>
>> 1. Create a new pass through pipe.
>> - The data will be written to the pipe by a spawned thread and current
>> thread will be consuming the data and continuing the message flow. We went
>> through the pipe creation logic and seemed it tightly coupled with
>> encoding, decoding methodologies so that it can't be
>> implemented at APIM level.
>>
>> 2. Change the synapse level logic to get the ByteArrayInputstream and
>> write it into the response.
>> - It can be but have to thoroughly go through and do it carefully
>> unless the default message flow would be affected.
>>
>> 3. Build the message by invoking RelayUtils.buildMessage() at the
>> validator mediator after successfully parsing all the validators.
>> - It will be slightly affected the performance but this is the
>> straightforward solution at this moment.
>>
>> I have improved the code by applying the 3rd option as we discussed.
>> Setting the PassThroughConstants.BUFFERED_INPUT_STREAM has an effect now
>> onwards since we changed a way that building the message to achieve the
>> content aware behavior which seeks the
>> inputream from the axis2 message context instead of the original
>> inputstream.
>>
>>
>> Regards
>> Hasunie
>>
>>
>>
>> On Tue, Jan 9, 2018 at 4:41 PM, Isuru Udana <[email protected]> wrote:
>>
>>> Hi Hasunie,
>>>
>>> As we discussed, setting the PassThroughConstants.BUFFERED_INPUT_STREAM
>>> has no effect on the flow in this case and Passthough Sender still seek
>>> content from the original input stream which got empty due to this cloning
>>> logic. That's the reason for this behaviour.
>>>
>>> Thanks.
>>>
>>>
>>>
>>> On Tue, Jan 9, 2018 at 11:43 AM, Hasunie Adikari <[email protected]>
>>> wrote:
>>>
>>>> Hi Isuru,
>>>>
>>>> As we discussed, I cloned the input stream by consuming the passthrough
>>>> pipe as in below.
>>>>
>>>>
>>>> if (pipe != null) {
>>>> bufferedInputStream = new BufferedInputStream(pipe.getIn
>>>> putStream());
>>>>
>>>> }
>>>> ByteArrayOutputStream byteArrayOutputStream = new
>>>> ByteArrayOutputStream();
>>>> byte[] buffer = new byte[1024];
>>>> int len;
>>>> while ((len = bufferedInputStream.read(buffer)) > -1 ) {
>>>> byteArrayOutputStream.write(buffer, 0, len);
>>>> }
>>>> byteArrayOutputStream.flush();
>>>>
>>>>
>>>> InputStream is1 = new ByteArrayInputStream(byteArray
>>>> OutputStream.toByteArray());
>>>> InputStream is2 = new ByteArrayInputStream(byteArray
>>>> OutputStream.toByteArray());
>>>>
>>>> consume the clones for the validation and set another clone as a
>>>> buffereInputstream property in the axis2messagecontext.
>>>>
>>>> BufferedInputStream bufferedInputStreamOriginal = new
>>>> BufferedInputStream(inputStreamOriginal);
>>>> axis2MC.setProperty(PassThroughConstants.BUFFERED_INPUT_STREAM,
>>>> bufferedInputStreamOriginal);
>>>>
>>>> I'm still getting the stream closed issue only for correct the messages
>>>> which have been passed through multiple validators. If the validators throw
>>>> an exception, the request is getting build and generate the custom response
>>>> as expected. It seems like we implemented a way that gets the
>>>> inputstream from the passthrough pipe for the content unaware flows.
>>>> Unless it uses to get the inputstream from the messagecontext. It was
>>>> proven once I attached an empty content aware mediator and test the
>>>> same flow. I was able to observe the expected behaviour for the same
>>>> implementation with the content aware mediator.
>>>>
>>>> Do we have a way to define cloned input stream as an original
>>>> inputstream in passthrough pipe?
>>>>
>>>>
>>>> Regards
>>>> Hasunie
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Jan 3, 2018 at 9:21 AM, Isuru Udana <[email protected]> wrote:
>>>>
>>>>> Hi Dushan,
>>>>>
>>>>> On Wed, Jan 3, 2018 at 9:06 AM, Dushan Abeyruwan <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Hi Hasunie,
>>>>>> Current PTT design would build the message whenever if there is
>>>>>> content aware mediator available. However IIRC, I did this
>>>>>> *message.builder.the
>>>>>> invoked* thing to cope with the WSO2 ELB we had (a few years ago).
>>>>>>
>>>>> No. I think it was *force.passthrough.builder *property which you
>>>>> introduced for ELB requirement.
>>>>>
>>>>> To be honest, that looks ugly isn it (in terms of overall picture).
>>>>>> Basically, what it does; even if there are content-aware mediators, the
>>>>>> engine would forcefully ignore that (it was ELB requirement :) ) but for
>>>>>> APIM I don't think that would be the same, cos we may have to deal with
>>>>>> many use cases sometimes of cause with content aware flows with API
>>>>>> compositions etc etc.
>>>>>>
>>>>>> So, let's think what we can do here; regex and XML threat protectors
>>>>>> equally important if security is priority thus, we would no longer able
>>>>>> to
>>>>>> achieve the same core basic aspect (content none awareness) because, such
>>>>>> protections required you to walk through the nodes and verify some
>>>>>> aspects
>>>>>> (basically, you need to expand the xml node tree to get result set) in
>>>>>> that
>>>>>> way, it is required the message to be build. Anyway, what I would think
>>>>>> the
>>>>>> best approach here is not to change complete synapse content awareness
>>>>>> logic rather I would think you may have mediator in place but only if
>>>>>> such
>>>>>> protection engaged that may build the message to get XML inforset (rather
>>>>>> build through root, may be you can mark this meditor as content-aware
>>>>>> false, then build if message not already build prior to process)
>>>>>>
>>>>>> IMO, lets just not complicate the what we try to build around message
>>>>>> validation. I mean if we need such protection we may need to sacrify some
>>>>>> aspects am I?
>>>>>>
>>>>>> Cheers,
>>>>>> Dushan
>>>>>>
>>>>>> On Tue, Jan 2, 2018 at 8:08 AM, Vinod Kavinda <[email protected]> wrote:
>>>>>>
>>>>>>> Hi Hasunie,
>>>>>>>
>>>>>>> This is expected since the synapse engine now expecting an already
>>>>>>> built message. If I understood your requirement correctly, one option
>>>>>>> is to
>>>>>>> use a Builder Mediator before using any content aware mediator. Even
>>>>>>> though
>>>>>>> we do not recommend the Builder mediators now, still we can use it for
>>>>>>> your
>>>>>>> specific use case. Or you have to revert the
>>>>>>> *message.builder.invoked *property to *false *again*.*
>>>>>>>
>>>>>>> Regards,
>>>>>>> Vinod
>>>>>>>
>>>>>>> On Tue, Jan 2, 2018 at 5:22 PM, Hasunie Adikari <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I'm trying to combine SQL injection(Regex) threat protector with
>>>>>>>> the XML threat protector. So I created a sequence[1] with
>>>>>>>> XMLthreatprotector mediator and regex mediator consecutively and
>>>>>>>> uploaded
>>>>>>>> it to be able to validate the request message through both the
>>>>>>>> xml validator and regex validators. If I set the
>>>>>>>> *message.builder.invoked *property to *TRUE *in xml validator
>>>>>>>> mediator to avoid sending the content in pass-through pipe(request
>>>>>>>> message) as the response, Regex mediators is getting failed. The regex
>>>>>>>> mediator was designed a way that the incoming messages are built in
>>>>>>>> synapse
>>>>>>>> level and eveluate the message content at the mediator level. It
>>>>>>>> seems like we can't continue any mediators which are required to get
>>>>>>>> the
>>>>>>>> message content, after we manually set the aforementioned property to
>>>>>>>> true
>>>>>>>> in the previous mediator. If I set it to true, RelayUtill will skip
>>>>>>>> building the message as in here [2]. Any thoughts regarding the issue.
>>>>>>>> I'm currently working on the issue to be able to combine regex and XML
>>>>>>>> threat protectors.
>>>>>>>> [2]
>>>>>>>>
>>>>>>>> if (pipe != null
>>>>>>>> && !Boolean.TRUE.equals(messageContext
>>>>>>>>
>>>>>>>> .getProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED)) &&
>>>>>>>> forcePTBuild) {
>>>>>>>> InputStream in = pipe.getInputStream();
>>>>>>>>
>>>>>>>> Object http_sc =
>>>>>>>> messageContext.getProperty(NhttpConstants.HTTP_SC);
>>>>>>>> if (http_sc != null && http_sc instanceof Integer &&
>>>>>>>> http_sc.equals(202)) {
>>>>>>>>
>>>>>>>> messageContext.setProperty(PassThroughConstants.MESSAGE_BUILDER_INVOKED,
>>>>>>>> Boolean.TRUE);
>>>>>>>> return;
>>>>>>>> }
>>>>>>>>
>>>>>>>> builldMessage(messageContext, earlyBuild, in);
>>>>>>>> return;
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> <sequence xmlns="http://ws.apache.org/ns/synapse"; name="validator">
>>>>>>>> <property name="threatType" expression="get-property('threatType')"
>>>>>>>> value="SQL-Injection and XML validator"/>
>>>>>>>> <property>
>>>>>>>> <property>
>>>>>>>> --------
>>>>>>>> <switch source="get-property('To')">
>>>>>>>> --------------
>>>>>>>> </switch>
>>>>>>>> *<class
>>>>>>>> name="org.wso2.carbon.apimgt.gateway.mediators.XMLSchemaValidator"/>*
>>>>>>>> <property name="enabledCheckPathParams"
>>>>>>>> expression="get-property('enabledCheckPathParams')" value="true"/>
>>>>>>>> <property>
>>>>>>>> <property>
>>>>>>>> --
>>>>>>>> *<class
>>>>>>>> name="org.wso2.carbon.apimgt.gateway.mediators.RegularExpressionProtector"/>*
>>>>>>>> <filter source="get-property('threat')" regex=".*error.*">
>>>>>>>> <then>
>>>>>>>> <sequence key="_threat_fault_"></sequence>
>>>>>>>> </then>
>>>>>>>> </filter>
>>>>>>>> </sequence>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Hasunie
>>>>>>>>
>>>>>>>> On Fri, Dec 22, 2017 at 5:14 PM, Hasunie Adikari <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> I'm working on threat protector feature in APIM. We're actually
>>>>>>>>> trying to achieve here is to protect both backend resources and
>>>>>>>>> gateway
>>>>>>>>> from the XML and JSON based attacks. The Balerina based APIM 3
>>>>>>>>> gateway will
>>>>>>>>> be protected by threat handlers. But In here
>>>>>>>>> APIM 2.1.x we have implemented mediators to achieve it.
>>>>>>>>>
>>>>>>>>> If we allow building the request message at the synapse level, It
>>>>>>>>> will definitely affect the gateway, All the request messages which go
>>>>>>>>> through the mediators are built since the Abstarctemediator is
>>>>>>>>> designed a
>>>>>>>>> way that the isContentAware method always returns true. So we set it
>>>>>>>>> to
>>>>>>>>> false in both XML and JSON validator mediators and allow to parse the
>>>>>>>>> XML request via a third party StAX parser called woodstox hence it
>>>>>>>>> was the
>>>>>>>>> best option among other StAX parsers for threat protection features.
>>>>>>>>> It
>>>>>>>>> will keep counting the given limits and when the limit is exceeded,
>>>>>>>>> It will
>>>>>>>>> terminate the process and throw a meaningful exception. I have
>>>>>>>>> created a
>>>>>>>>> custom threat sequence(thrat_fault) and If a threat is detected by
>>>>>>>>> getting
>>>>>>>>> an exception I configured to direct the response through the custom
>>>>>>>>> error
>>>>>>>>> sequence.
>>>>>>>>> I reuse the same custom sequence which was implemented for the
>>>>>>>>> regex threat protector [1]
>>>>>>>>>
>>>>>>>>> Woodstox parser covers most of the vulnerabilities as in here
>>>>>>>>>
>>>>>>>>> *Vulnerablity:*
>>>>>>>>>
>>>>>>>>> *xml bomb* - DTD disabling
>>>>>>>>>
>>>>>>>>> *external entity attack* - disabling external entities.
>>>>>>>>>
>>>>>>>>> Note :
>>>>>>>>> Apart from the mediator level, The external entity
>>>>>>>>> reference property was disabled from the DOM parsers at the synapse
>>>>>>>>> level
>>>>>>>>> as well.
>>>>>>>>>
>>>>>>>>> import org.apache.xerces.impl.Constants;
>>>>>>>>>
>>>>>>>>> private static final int ENTITY_EXPANSION_LIMIT = 0;
>>>>>>>>> private static final DocumentBuilderFactory
>>>>>>>>> documentBuilderFactory =
>>>>>>>>> DocumentBuilderFactory.newInstance();
>>>>>>>>>
>>>>>>>>> static {
>>>>>>>>> documentBuilderFactory.setNamespaceAware(true);
>>>>>>>>> documentBuilderFactory.setXIncludeAware(false);
>>>>>>>>> documentBuilderFactory.setExpandEntityReferences(false);
>>>>>>>>>
>>>>>>>>> try {
>>>>>>>>> documentBuilderFactory.setFeature(Constants.SAX_FEATU
>>>>>>>>> RE_PREFIX +
>>>>>>>>> Constants.EXTERNAL_GENERAL_ENTITIES_FEATURE, false);
>>>>>>>>> } catch (ParserConfigurationException e) {
>>>>>>>>>
>>>>>>>>> *Buffer overflow attack* - by limiting the count of elements,
>>>>>>>>> children and length of attributes/keys/values.
>>>>>>>>>
>>>>>>>>> *woodstox properties:*
>>>>>>>>> dtdEnabled
>>>>>>>>> externalEntitiesEnabled
>>>>>>>>> maxDepth
>>>>>>>>> maxElementCount
>>>>>>>>> maxAttributeCount
>>>>>>>>> maxAttributeLength
>>>>>>>>> entityExpansionLimit
>>>>>>>>> maxChildrenPerElement
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> For thwart cohesive attacks, we use both schema validator and
>>>>>>>>> depth limits. Ideally, only the woodstox validator should detect the
>>>>>>>>> cohesive attacks by exceeding the defined depth limit. But the schema
>>>>>>>>> validator will protect the schema poising attacks in the second step
>>>>>>>>> as
>>>>>>>>> well.
>>>>>>>>>
>>>>>>>>> I observed an issue when It comes to combining each
>>>>>>>>> other(woodstox+ schema validator). We have designed the feature in
>>>>>>>>> such a
>>>>>>>>> way that gets the inputstream from the message context and consumes
>>>>>>>>> it in
>>>>>>>>> the woodstox validator. but in here we have to consume the input
>>>>>>>>> stream
>>>>>>>>> again for the schema validation just after passing through the
>>>>>>>>> woodstox.That was the issue and I tried the following methodologies to
>>>>>>>>> resolved the issue
>>>>>>>>>
>>>>>>>>> 1. try to get the XML object from the woodstox parser to be able
>>>>>>>>> to avoid using the input stream again.
>>>>>>>>> 2. deep clone the inputstream and use cloned input stream for the
>>>>>>>>> schema validation.
>>>>>>>>> 3. reset, mark the buffered input stream(synapse engine also has
>>>>>>>>> done rest, mark)
>>>>>>>>>
>>>>>>>>> 1st one was taken time and much complex to get the XML object
>>>>>>>>> since Woodstock is based on the StAX parsers and also deep cloning
>>>>>>>>> was not
>>>>>>>>> working properly and experienced the same issue after cloning the
>>>>>>>>> inputstream. But the 3rd option makes life easy so I implemented a
>>>>>>>>> way that
>>>>>>>>> returning the buffered input stream, after doing the rest, mark,
>>>>>>>>> then It
>>>>>>>>> works properly. I went through the RelayUtil message builders [2] and
>>>>>>>>> It
>>>>>>>>> also uses the mark and reset methodology and return InputStream.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I observed another issue once the validator throws an exception,
>>>>>>>>> the server hanged and didn't get any response and getting timeout
>>>>>>>>> issue. I
>>>>>>>>> was able to figure it out and Issue occurred while trying to build the
>>>>>>>>> request message in Relayutil.buildmessage().But Ideally, If we get an
>>>>>>>>> error
>>>>>>>>> we don't need the request message anymore. As I discussed offline
>>>>>>>>> with the
>>>>>>>>> APIM team, I used the *consumeAndDiscardMessage* method to
>>>>>>>>> discard the request message from the message context and set
>>>>>>>>> *message.builder.invoked
>>>>>>>>> *property to *TRUE. *It needs to be set to avoid sending the
>>>>>>>>> content in pass-through pipe (request message) as the response.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [1] https://docs.wso2.com/display/AM210/Regular+Expression+T
>>>>>>>>> hreat+Protection+for+API+Gateway
>>>>>>>>> [2] https://github.com/wso2/wso2-synapse/blob/master/modules
>>>>>>>>> /transports/core/nhttp/src/main/java/org/apache/synapse/tran
>>>>>>>>> sport/passthru/util/RelayUtils.java#L121
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Hasunie
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Hasunie Adikari*
>>>>>>>>> Senior Software Engineer
>>>>>>>>> WSO2 Inc.; http://wso2.com
>>>>>>>>> lean.enterprise.middleware
>>>>>>>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/
>>>>>>>>> Mobile:+94713095876 <+94%2071%20309%205876>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Hasunie Adikari*
>>>>>>>> Senior Software Engineer
>>>>>>>> WSO2 Inc.; http://wso2.com
>>>>>>>> lean.enterprise.middleware
>>>>>>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/
>>>>>>>> Mobile:+94713095876 <+94%2071%20309%205876>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Vinod Kavinda
>>>>>>> Senior Software Engineer
>>>>>>> *WSO2 Inc. - lean . enterprise . middleware <http://www.wso2.com>.*
>>>>>>> Mobile : +94 (0) 712 415544
>>>>>>> Blog : http://soatechflicks.blogspot.com/
>>>>>>> [image: http://wso2.com/signature]
>>>>>>> <http://wso2.com/signature>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Dushan Abeyruwan | Architect
>>>>>> Technical Support,MV
>>>>>> PMC Member Apache Synpase
>>>>>> WSO2 Inc. http://wso2.com/
>>>>>> Blog:*http://www.dushantech.com/ <http://www.dushantech.com/>*
>>>>>> LinkedIn:*https://www.linkedin.com/in/dushanabeyruwan
>>>>>> <https://www.linkedin.com/in/dushanabeyruwan>*
>>>>>> Mobile:(001)408-791-9312 <+1%20408-791-9312>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Isuru Udana*
>>>>> Senior Technical Lead
>>>>> WSO2 Inc.; http://wso2.com
>>>>> email: [email protected] cell: +94 77 3791887 <+94%2077%20379%201887>
>>>>> blog: http://mytecheye.blogspot.com/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Hasunie Adikari*
>>>> Senior Software Engineer
>>>> WSO2 Inc.; http://wso2.com
>>>> lean.enterprise.middleware
>>>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/
>>>> Mobile:+94713095876 <071%20309%205876>
>>>>
>>>>
>>>
>>>
>>> --
>>> *Isuru Udana*
>>> Senior Technical Lead
>>> WSO2 Inc.; http://wso2.com
>>> email: [email protected] cell: +94 77 3791887 <+94%2077%20379%201887>
>>> blog: http://mytecheye.blogspot.com/
>>>
>>
>>
>>
>> --
>> *Hasunie Adikari*
>> Senior Software Engineer
>> WSO2 Inc.; http://wso2.com
>> lean.enterprise.middleware
>> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/
>> Mobile:+94713095876 <+94%2071%20309%205876>
>>
>>
>
>
> --
> *Hasunie Adikari*
> Senior Software Engineer
> WSO2 Inc.; http://wso2.com
> lean.enterprise.middleware
> blog http://hasuniea.blogspot.com | https://medium.com/@Hasunie/
> Mobile:+94713095876 <071%20309%205876>
>
>
--
Dimuthu Leelarathne
Director, Solutions Architecture
WSO2, Inc. (http://wso2.com)
email: [email protected]
Mobile: +94773661935
Blog: http://muthulee.blogspot.com
Lean . Enterprise . Middleware
_______________________________________________
Architecture mailing list
[email protected]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
