Is the following code segment the way to read all the chunks;
while(!carbonMsg.isEndOfMsgAdded) {
ByteBuffer chunk = carbonMsg.getMessageBody();
}
On Thu, Jun 2, 2016 at 9:21 PM, Afkham Azeez <[email protected]> wrote:
> Looks like we require changes for input streaming as well.
>
> Does CarbonMessage.getMessageBody() block until the next chunk is
> received?
>
> On Thu, Jun 2, 2016 at 9:18 PM, Afkham Azeez <[email protected]> wrote:
>
>> Samiyuru/Isuru,
>> Does this mean that the input streaming in MSF4J is currently working
>> without any issue and only output streaming requires some work?
>>
>> On Tue, May 3, 2016 at 2:29 PM, Isuru Ranawaka <[email protected]> wrote:
>>
>>> Hi All,
>>>
>>> Following are the details on streaming support of carbon transport.
>>> Basically we can looking to streaming support in request path and response
>>> path separately.
>>>
>>> Request Path (Streaming is working)
>>>
>>> [image: requestpath.png]
>>>
>>> According to above diagram
>>>
>>> -
>>>
>>> We have blocking queue in the carbon message which keeps the content
>>> . when headers are received through Netty worker thread it will create
>>> CarbonMessage with a blocking queue and publish carbon message to engine
>>> level thread.
>>> -
>>>
>>> Reference for blocking queue is cached in the connection and when
>>> content is received it will be filled to that queue from Netty worker
>>> thread.
>>> -
>>>
>>> Meanwhile engine level threads can consume content through
>>> queue(While IO worker is filling ) and can directly send to file system
>>> or
>>> use sender for send messages to external service.
>>> -
>>>
>>> According to that streaming should work in Request path in
>>> Integration Server or MSF4J without any problem.
>>>
>>>
>>> Response Path (Streaming working scenario)
>>>
>>> [image: responsepathworking.png]
>>>
>>>
>>> In Response path basic difference we have is we are handling responses
>>> through callbacks.
>>>
>>>
>>> -
>>>
>>> Similar to Request path architecture Sender side IO thread
>>> creates a CM when response headers are received and publish to engine
>>> level thread .
>>> -
>>>
>>> Engine level thread calls carbonCallback.done() and waits on queue
>>> for content.
>>> -
>>>
>>> Meanwhile IO thread writes the content to the Queue .
>>> -
>>>
>>> So writing to Queue and reading from queue happens parallel and
>>> streaming should work properly.
>>> -
>>>
>>> So streaming is working fine with Integration Server for this kind
>>> of scenarios.
>>>
>>>
>>>
>>> Response Path (Streaming not working scenario)
>>>
>>> [image: responsepathnotworking.png]
>>>
>>> This scenario is basically, if we have a Echo mediator or assume MSF4J
>>> has a service which is reading a file as chunks and writes back to client .
>>>
>>>
>>> -
>>>
>>> Basic difference with the previous one is in this approach reading
>>> a file chunk or stream and writing that file chunk or stream to Queue is
>>> happened within the same engine level thread as well as with reading
>>> from
>>> queue and writing to Listener side Netty worker threads.(Same thread
>>> produce and consume causes for deadlock)
>>> -
>>>
>>> In the previous example reading from stream and filling the queue
>>> was happened through Sender side IO thread and consuming the queue and
>>> writing to Listener side IO thread was happened through engine level
>>> thread. (Two different threads produce and consume)
>>> -
>>>
>>> Streaming is not working in this kind of scenarios because we cannot
>>> write to queue and keep polling the queue within single thread.
>>>
>>>
>>> We can figure out following solutions and what will be the most suitable
>>> one.
>>>
>>>
>>>
>>> -
>>>
>>> Introduce another thread for run the callback logic instead of
>>> running through calling thread.
>>>
>>> This will solve the above streaming problem but when it
>>> comes to general message flow this will add another level of thread which
>>> will actually does not need.
>>>
>>> -
>>>
>>> Introduce another method in ResponseCallback which will support
>>> streaming which does not queuing contents. it will directly call IO
>>> threads and write contents to IO when chunks are received. This can be
>>> used
>>> only within single thread scenario(File reading and writing ).
>>> -
>>>
>>> Introduce another thread for read File stream or call Callback from
>>> another thread other than to reading thread from MSF4J level.Then it
>>> will
>>> be equivalent to how we used it in Integration Server with the use of
>>> Sender.
>>>
>>> ThankYou
>>>
>>> IsuruR
>>>
>>> On Tue, May 3, 2016 at 12:09 PM, Kasun Indrasiri <[email protected]> wrote:
>>>
>>>> Had a chat with Ranawaka on this.. seems like we do have couple of ways
>>>> to handle this. He will share the details.
>>>>
>>>> On Mon, May 2, 2016 at 6:13 AM, Afkham Azeez <[email protected]> wrote:
>>>>
>>>>> The problem is, we can't do the next MSF4J release because we can't
>>>>> lose a feature in a release. This is a blocker for us. We have plans to
>>>>> release in the next 2 weeks.
>>>>>
>>>>> On Mon, May 2, 2016 at 4:56 PM, Isuru Ranawaka <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Hi Azeez,
>>>>>>
>>>>>> Currently streaming works if we used both sender side and listener
>>>>>> side only . But since MSF4J is using only listener side if we did not
>>>>>> spawn
>>>>>> separate thread from engine level for writing response it will not work
>>>>>> because request reading and writing happens through same thread. But with
>>>>>> the next release we will fix that. currently we are finalizing on
>>>>>> removing
>>>>>> engine level thread model from transport.
>>>>>>
>>>>>>
>>>>>> thanks
>>>>>> IsuruR
>>>>>>
>>>>>> On Mon, May 2, 2016 at 3:50 PM, Afkham Azeez <[email protected]> wrote:
>>>>>>
>>>>>>> Is this working after moving to the new transport framework?
>>>>>>>
>>>>>>> --
>>>>>>> *Afkham Azeez*
>>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>>>>> * <http://www.apache.org/>*
>>>>>>> *email: **[email protected]* <[email protected]>
>>>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>>>>>>> *http://blog.afkham.org* <http://blog.afkham.org>
>>>>>>> *twitter: **http://twitter.com/afkham_azeez*
>>>>>>> <http://twitter.com/afkham_azeez>
>>>>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>>>>>>> <http://lk.linkedin.com/in/afkhamazeez>*
>>>>>>>
>>>>>>> *Lean . Enterprise . Middleware*
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Dev mailing list
>>>>>>> [email protected]
>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards
>>>>>> Isuru Ranawaka
>>>>>> M: +94714629880
>>>>>> Blog : http://isurur.blogspot.com/
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Afkham Azeez*
>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>>> * <http://www.apache.org/>*
>>>>> *email: **[email protected]* <[email protected]>
>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>>>>> *http://blog.afkham.org* <http://blog.afkham.org>
>>>>> *twitter: **http://twitter.com/afkham_azeez*
>>>>> <http://twitter.com/afkham_azeez>
>>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>>>>> <http://lk.linkedin.com/in/afkhamazeez>*
>>>>>
>>>>> *Lean . Enterprise . Middleware*
>>>>>
>>>>> _______________________________________________
>>>>> Dev mailing list
>>>>> [email protected]
>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Kasun Indrasiri
>>>> Software Architect
>>>> WSO2, Inc.; http://wso2.com
>>>> lean.enterprise.middleware
>>>>
>>>> cell: +94 77 556 5206
>>>> Blog : http://kasunpanorama.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards
>>> Isuru Ranawaka
>>> M: +94714629880
>>> Blog : http://isurur.blogspot.com/
>>>
>>
>>
>>
>> --
>> *Afkham Azeez*
>> Director of Architecture; WSO2, Inc.; http://wso2.com
>> Member; Apache Software Foundation; http://www.apache.org/
>> * <http://www.apache.org/>*
>> *email: **[email protected]* <[email protected]>
>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>> *http://blog.afkham.org* <http://blog.afkham.org>
>> *twitter: **http://twitter.com/afkham_azeez*
>> <http://twitter.com/afkham_azeez>
>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>> <http://lk.linkedin.com/in/afkhamazeez>*
>>
>> *Lean . Enterprise . Middleware*
>>
>
>
>
> --
> *Afkham Azeez*
> Director of Architecture; WSO2, Inc.; http://wso2.com
> Member; Apache Software Foundation; http://www.apache.org/
> * <http://www.apache.org/>*
> *email: **[email protected]* <[email protected]>
> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
> *http://blog.afkham.org* <http://blog.afkham.org>
> *twitter: **http://twitter.com/afkham_azeez*
> <http://twitter.com/afkham_azeez>
> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
> <http://lk.linkedin.com/in/afkhamazeez>*
>
> *Lean . Enterprise . Middleware*
>
--
*Afkham Azeez*
Director of Architecture; WSO2, Inc.; http://wso2.com
Member; Apache Software Foundation; http://www.apache.org/
* <http://www.apache.org/>*
*email: **[email protected]* <[email protected]>
* cell: +94 77 3320919blog: **http://blog.afkham.org*
<http://blog.afkham.org>
*twitter: **http://twitter.com/afkham_azeez*
<http://twitter.com/afkham_azeez>
*linked-in: **http://lk.linkedin.com/in/afkhamazeez
<http://lk.linkedin.com/in/afkhamazeez>*
*Lean . Enterprise . Middleware*
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
