while(!carbonMsg.isEndOfMsgAdded) {
ByteBuffer chunk = carbonMsg.getMessageBody();
}
In the above segment, carbonMsg is the request CarbonMessage and the above
code segment doesn't work. Code inside the while loop never executes.
On Thu, Jun 2, 2016 at 9:47 PM, Isuru Ranawaka <[email protected]> wrote:
> Hi Azeez,
> yes that should work. Anyhow content needs to be added to carbonMsg from
> one thread and read to be from another thread.
>
> thanks
>
> On Thu, Jun 2, 2016 at 9:23 PM, Afkham Azeez <[email protected]> wrote:
>
>> Is the following code segment the way to read all the chunks;
>>
>> while(!carbonMsg.isEndOfMsgAdded) {
>> ByteBuffer chunk = carbonMsg.getMessageBody();
>>
>> }
>>
>> On Thu, Jun 2, 2016 at 9:21 PM, Afkham Azeez <[email protected]> wrote:
>>
>>> Looks like we require changes for input streaming as well.
>>>
>>> Does CarbonMessage.getMessageBody() block until the next chunk is
>>> received?
>>>
>>> On Thu, Jun 2, 2016 at 9:18 PM, Afkham Azeez <[email protected]> wrote:
>>>
>>>> Samiyuru/Isuru,
>>>> Does this mean that the input streaming in MSF4J is currently working
>>>> without any issue and only output streaming requires some work?
>>>>
>>>> On Tue, May 3, 2016 at 2:29 PM, Isuru Ranawaka <[email protected]> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> Following are the details on streaming support of carbon transport.
>>>>> Basically we can looking to streaming support in request path and response
>>>>> path separately.
>>>>>
>>>>> Request Path (Streaming is working)
>>>>>
>>>>> [image: requestpath.png]
>>>>>
>>>>> According to above diagram
>>>>>
>>>>> -
>>>>>
>>>>> We have blocking queue in the carbon message which keeps the
>>>>> content . when headers are received through Netty worker thread it will
>>>>> create CarbonMessage with a blocking queue and publish carbon message
>>>>> to
>>>>> engine level thread.
>>>>> -
>>>>>
>>>>> Reference for blocking queue is cached in the connection and when
>>>>> content is received it will be filled to that queue from Netty worker
>>>>> thread.
>>>>> -
>>>>>
>>>>> Meanwhile engine level threads can consume content through
>>>>> queue(While IO worker is filling ) and can directly send to file
>>>>> system or
>>>>> use sender for send messages to external service.
>>>>> -
>>>>>
>>>>> According to that streaming should work in Request path in
>>>>> Integration Server or MSF4J without any problem.
>>>>>
>>>>>
>>>>> Response Path (Streaming working scenario)
>>>>>
>>>>> [image: responsepathworking.png]
>>>>>
>>>>>
>>>>> In Response path basic difference we have is we are handling responses
>>>>> through callbacks.
>>>>>
>>>>>
>>>>> -
>>>>>
>>>>> Similar to Request path architecture Sender side IO thread
>>>>> creates a CM when response headers are received and publish to engine
>>>>> level thread .
>>>>> -
>>>>>
>>>>> Engine level thread calls carbonCallback.done() and waits on queue
>>>>> for content.
>>>>> -
>>>>>
>>>>> Meanwhile IO thread writes the content to the Queue .
>>>>> -
>>>>>
>>>>> So writing to Queue and reading from queue happens parallel and
>>>>> streaming should work properly.
>>>>> -
>>>>>
>>>>> So streaming is working fine with Integration Server for this kind
>>>>> of scenarios.
>>>>>
>>>>>
>>>>>
>>>>> Response Path (Streaming not working scenario)
>>>>>
>>>>> [image: responsepathnotworking.png]
>>>>>
>>>>> This scenario is basically, if we have a Echo mediator or assume MSF4J
>>>>> has a service which is reading a file as chunks and writes back to
>>>>> client .
>>>>>
>>>>>
>>>>> -
>>>>>
>>>>> Basic difference with the previous one is in this approach
>>>>> reading a file chunk or stream and writing that file chunk or stream
>>>>> to
>>>>> Queue is happened within the same engine level thread as well as with
>>>>> reading from queue and writing to Listener side Netty worker
>>>>> threads.(Same
>>>>> thread produce and consume causes for deadlock)
>>>>> -
>>>>>
>>>>> In the previous example reading from stream and filling the queue
>>>>> was happened through Sender side IO thread and consuming the queue and
>>>>> writing to Listener side IO thread was happened through engine level
>>>>> thread. (Two different threads produce and consume)
>>>>> -
>>>>>
>>>>> Streaming is not working in this kind of scenarios because we
>>>>> cannot write to queue and keep polling the queue within single thread.
>>>>>
>>>>>
>>>>> We can figure out following solutions and what will be the most
>>>>> suitable one.
>>>>>
>>>>>
>>>>>
>>>>> -
>>>>>
>>>>> Introduce another thread for run the callback logic instead of
>>>>> running through calling thread.
>>>>>
>>>>> This will solve the above streaming problem but when
>>>>> it comes to general message flow this will add another level of thread
>>>>> which will actually does not need.
>>>>>
>>>>> -
>>>>>
>>>>> Introduce another method in ResponseCallback which will support
>>>>> streaming which does not queuing contents. it will directly call IO
>>>>> threads and write contents to IO when chunks are received. This can be
>>>>> used
>>>>> only within single thread scenario(File reading and writing ).
>>>>> -
>>>>>
>>>>> Introduce another thread for read File stream or call Callback
>>>>> from another thread other than to reading thread from MSF4J
>>>>> level.Then it
>>>>> will be equivalent to how we used it in Integration Server with the
>>>>> use of
>>>>> Sender.
>>>>>
>>>>> ThankYou
>>>>>
>>>>> IsuruR
>>>>>
>>>>> On Tue, May 3, 2016 at 12:09 PM, Kasun Indrasiri <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Had a chat with Ranawaka on this.. seems like we do have couple of
>>>>>> ways to handle this. He will share the details.
>>>>>>
>>>>>> On Mon, May 2, 2016 at 6:13 AM, Afkham Azeez <[email protected]> wrote:
>>>>>>
>>>>>>> The problem is, we can't do the next MSF4J release because we can't
>>>>>>> lose a feature in a release. This is a blocker for us. We have plans to
>>>>>>> release in the next 2 weeks.
>>>>>>>
>>>>>>> On Mon, May 2, 2016 at 4:56 PM, Isuru Ranawaka <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Azeez,
>>>>>>>>
>>>>>>>> Currently streaming works if we used both sender side and listener
>>>>>>>> side only . But since MSF4J is using only listener side if we did not
>>>>>>>> spawn
>>>>>>>> separate thread from engine level for writing response it will not
>>>>>>>> work
>>>>>>>> because request reading and writing happens through same thread. But
>>>>>>>> with
>>>>>>>> the next release we will fix that. currently we are finalizing on
>>>>>>>> removing
>>>>>>>> engine level thread model from transport.
>>>>>>>>
>>>>>>>>
>>>>>>>> thanks
>>>>>>>> IsuruR
>>>>>>>>
>>>>>>>> On Mon, May 2, 2016 at 3:50 PM, Afkham Azeez <[email protected]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Is this working after moving to the new transport framework?
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Afkham Azeez*
>>>>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>>>>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>>>>>>> * <http://www.apache.org/>*
>>>>>>>>> *email: **[email protected]* <[email protected]>
>>>>>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>>>>>>>>> *http://blog.afkham.org* <http://blog.afkham.org>
>>>>>>>>> *twitter: **http://twitter.com/afkham_azeez*
>>>>>>>>> <http://twitter.com/afkham_azeez>
>>>>>>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>>>>>>>>> <http://lk.linkedin.com/in/afkhamazeez>*
>>>>>>>>>
>>>>>>>>> *Lean . Enterprise . Middleware*
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Dev mailing list
>>>>>>>>> [email protected]
>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards
>>>>>>>> Isuru Ranawaka
>>>>>>>> M: +94714629880
>>>>>>>> Blog : http://isurur.blogspot.com/
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Afkham Azeez*
>>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>>>>> * <http://www.apache.org/>*
>>>>>>> *email: **[email protected]* <[email protected]>
>>>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>>>>>>> *http://blog.afkham.org* <http://blog.afkham.org>
>>>>>>> *twitter: **http://twitter.com/afkham_azeez*
>>>>>>> <http://twitter.com/afkham_azeez>
>>>>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>>>>>>> <http://lk.linkedin.com/in/afkhamazeez>*
>>>>>>>
>>>>>>> *Lean . Enterprise . Middleware*
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Dev mailing list
>>>>>>> [email protected]
>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Kasun Indrasiri
>>>>>> Software Architect
>>>>>> WSO2, Inc.; http://wso2.com
>>>>>> lean.enterprise.middleware
>>>>>>
>>>>>> cell: +94 77 556 5206
>>>>>> Blog : http://kasunpanorama.blogspot.com/
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best Regards
>>>>> Isuru Ranawaka
>>>>> M: +94714629880
>>>>> Blog : http://isurur.blogspot.com/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Afkham Azeez*
>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>> * <http://www.apache.org/>*
>>>> *email: **[email protected]* <[email protected]>
>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>>>> *http://blog.afkham.org* <http://blog.afkham.org>
>>>> *twitter: **http://twitter.com/afkham_azeez*
>>>> <http://twitter.com/afkham_azeez>
>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>>>> <http://lk.linkedin.com/in/afkhamazeez>*
>>>>
>>>> *Lean . Enterprise . Middleware*
>>>>
>>>
>>>
>>>
>>> --
>>> *Afkham Azeez*
>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>> Member; Apache Software Foundation; http://www.apache.org/
>>> * <http://www.apache.org/>*
>>> *email: **[email protected]* <[email protected]>
>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>>> *http://blog.afkham.org* <http://blog.afkham.org>
>>> *twitter: **http://twitter.com/afkham_azeez*
>>> <http://twitter.com/afkham_azeez>
>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>>> <http://lk.linkedin.com/in/afkhamazeez>*
>>>
>>> *Lean . Enterprise . Middleware*
>>>
>>
>>
>>
>> --
>> *Afkham Azeez*
>> Director of Architecture; WSO2, Inc.; http://wso2.com
>> Member; Apache Software Foundation; http://www.apache.org/
>> * <http://www.apache.org/>*
>> *email: **[email protected]* <[email protected]>
>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>> *http://blog.afkham.org* <http://blog.afkham.org>
>> *twitter: **http://twitter.com/afkham_azeez*
>> <http://twitter.com/afkham_azeez>
>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>> <http://lk.linkedin.com/in/afkhamazeez>*
>>
>> *Lean . Enterprise . Middleware*
>>
>
>
>
> --
> Best Regards
> Isuru Ranawaka
> M: +94714629880
> Blog : http://isurur.blogspot.com/
>
--
*Afkham Azeez*
Director of Architecture; WSO2, Inc.; http://wso2.com
Member; Apache Software Foundation; http://www.apache.org/
* <http://www.apache.org/>*
*email: **[email protected]* <[email protected]>
* cell: +94 77 3320919blog: **http://blog.afkham.org*
<http://blog.afkham.org>
*twitter: **http://twitter.com/afkham_azeez*
<http://twitter.com/afkham_azeez>
*linked-in: **http://lk.linkedin.com/in/afkhamazeez
<http://lk.linkedin.com/in/afkhamazeez>*
*Lean . Enterprise . Middleware*
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
