Re: [jira] [Commented] (DIRMINA-1070) Avoid unbounded message queueing when sending large amounts of data to slow clients

[Jonathan Valliere]

I absolutely understand how you must write the entire message at once to
ensure that another message from another thread doesn't cause the stream to
be corrupted. That said, I also think this is an SSHD issue and not a Mina
one.

On Thu, Aug 31, 2017 at 2:50 AM Emmanuel Lecharny (JIRA) 
wrote:

>
> [
> https://issues.apache.org/jira/browse/DIRMINA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148533#comment-16148533
> ]
>
> Emmanuel Lecharny commented on DIRMINA-1070:
> 
>
> At this point, I think it would be a good idea to move this ticket to
> SSHD, for our fellow sshd dev to gie you a better answer!
>
> > Avoid unbounded message queueing when sending large amounts of data to
> slow clients
> >
> ---
> >
> > Key: DIRMINA-1070
> > URL: https://issues.apache.org/jira/browse/DIRMINA-1070
> > Project: MINA
> >  Issue Type: New Feature
> >  Components: Core
> >Reporter: jpalacios
> >  Labels: stability
> >
> > Our application runs an Apache MINA server to provide SSH support. We
> are seeing {{OutOfMemoryError}} s when certain clients establish a session
> with a large {{Window}} size. Particularly clients like TortoiseGit (which
> uses TortoisePlink which in turn seems to use Putty) use an initial window
> size of 2GB. From heap dumps we can see that the
> {{DefaultWriteRequestQueue}} is filling up with {{WriteRequest}} instances
> and taking up gigabytes of space until the heap blows.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.4.14#64029)
>

[jira] [Commented] (DIRMINA-1070) Avoid unbounded message queueing when sending large amounts of data to slow clients

[Emmanuel Lecharny (JIRA)]

[ 
https://issues.apache.org/jira/browse/DIRMINA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148533#comment-16148533
 ] 

Emmanuel Lecharny commented on DIRMINA-1070:


At this point, I think it would be a good idea to move this ticket to SSHD, for 
our fellow sshd dev to gie you a better answer! 

> Avoid unbounded message queueing when sending large amounts of data to slow 
> clients
> ---
>
> Key: DIRMINA-1070
> URL: https://issues.apache.org/jira/browse/DIRMINA-1070
> Project: MINA
>  Issue Type: New Feature
>  Components: Core
>Reporter: jpalacios
>  Labels: stability
>
> Our application runs an Apache MINA server to provide SSH support. We are 
> seeing {{OutOfMemoryError}} s when certain clients establish a session with a 
> large {{Window}} size. Particularly clients like TortoiseGit (which uses 
> TortoisePlink which in turn seems to use Putty) use an initial window size of 
> 2GB. From heap dumps we can see that the {{DefaultWriteRequestQueue}} is 
> filling up with {{WriteRequest}} instances and taking up gigabytes of space 
> until the heap blows. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DIRMINA-1070) Avoid unbounded message queueing when sending large amounts of data to slow clients

[Emmanuel Lecharny (JIRA)]

[ 
https://issues.apache.org/jira/browse/DIRMINA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148509#comment-16148509
 ] 

Emmanuel Lecharny commented on DIRMINA-1070:


I'm not very familiar with {{SSHd}}, so I don't know how it deals with big 
messages, but again, the logic is for your application to send smaller 
'messages' than one single big message. 

Can you paste the code where you write a message back to the client ?

> Avoid unbounded message queueing when sending large amounts of data to slow 
> clients
> ---
>
> Key: DIRMINA-1070
> URL: https://issues.apache.org/jira/browse/DIRMINA-1070
> Project: MINA
>  Issue Type: New Feature
>  Components: Core
>Reporter: jpalacios
>  Labels: stability
>
> Our application runs an Apache MINA server to provide SSH support. We are 
> seeing {{OutOfMemoryError}} s when certain clients establish a session with a 
> large {{Window}} size. Particularly clients like TortoiseGit (which uses 
> TortoisePlink which in turn seems to use Putty) use an initial window size of 
> 2GB. From heap dumps we can see that the {{DefaultWriteRequestQueue}} is 
> filling up with {{WriteRequest}} instances and taking up gigabytes of space 
> until the heap blows. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DIRMINA-1070) Avoid unbounded message queueing when sending large amounts of data to slow clients

[jpalacios (JIRA)]

[ 
https://issues.apache.org/jira/browse/DIRMINA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148493#comment-16148493
 ] 

jpalacios commented on DIRMINA-1070:


Actually we receive one message which is a request for data (which can be GBs 
in size), and we use a {{org.apache.sshd.server.Command}} implementation to 
write to the provided {{OutputStream}}. We don't load all the data in memory at 
any point (except when Apache MINA queues it in the unbounded 
{{DefaultWriteRequestQueue}}).

> Avoid unbounded message queueing when sending large amounts of data to slow 
> clients
> ---
>
> Key: DIRMINA-1070
> URL: https://issues.apache.org/jira/browse/DIRMINA-1070
> Project: MINA
>  Issue Type: New Feature
>  Components: Core
>Reporter: jpalacios
>  Labels: stability
>
> Our application runs an Apache MINA server to provide SSH support. We are 
> seeing {{OutOfMemoryError}} s when certain clients establish a session with a 
> large {{Window}} size. Particularly clients like TortoiseGit (which uses 
> TortoisePlink which in turn seems to use Putty) use an initial window size of 
> 2GB. From heap dumps we can see that the {{DefaultWriteRequestQueue}} is 
> filling up with {{WriteRequest}} instances and taking up gigabytes of space 
> until the heap blows. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DIRMINA-1070) Avoid unbounded message queueing when sending large amounts of data to slow clients

[Emmanuel Lecharny (JIRA)]

[ 
https://issues.apache.org/jira/browse/DIRMINA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148454#comment-16148454
 ] 

Emmanuel Lecharny commented on DIRMINA-1070:


If you are going to stream Gb of data to a client in one single message, I 
strongly suggest you split your message in smaller pieces in order to be able 
to throttle the writes. If you keep the message in one single block, you are 
going to be in trouble...



> Avoid unbounded message queueing when sending large amounts of data to slow 
> clients
> ---
>
> Key: DIRMINA-1070
> URL: https://issues.apache.org/jira/browse/DIRMINA-1070
> Project: MINA
>  Issue Type: New Feature
>  Components: Core
>Reporter: jpalacios
>  Labels: stability
>
> Our application runs an Apache MINA server to provide SSH support. We are 
> seeing {{OutOfMemoryError}} s when certain clients establish a session with a 
> large {{Window}} size. Particularly clients like TortoiseGit (which uses 
> TortoisePlink which in turn seems to use Putty) use an initial window size of 
> 2GB. From heap dumps we can see that the {{DefaultWriteRequestQueue}} is 
> filling up with {{WriteRequest}} instances and taking up gigabytes of space 
> until the heap blows. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [jira] [Commented] (DIRMINA-1070) Avoid unbounded message queueing when sending large amounts of data to slow clients

[Jonathan Valliere]

Are you saying that Mina has a bug or are you proposing some new feature to
handle your back pressure problem?

On Wed, Aug 30, 2017 at 7:52 PM jpalacios (JIRA)  wrote:

>
> [
> https://issues.apache.org/jira/browse/DIRMINA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148228#comment-16148228
> ]
>
> jpalacios commented on DIRMINA-1070:
> 
>
> Hi [~elecharny],
> Thank you for getting back to me. Let me provide a bit more context to my
> issue.
>
> Our application can stream gigabytes of data to clients based on a single
> message. We need to ensure that at any given time only a certain amount of
> bytes are inflight for any given session. The one message / one response
> approach I don't think will work for us. Also, in my testing (at least with
> my implementation of {{IoFilter}} I see that {{messageSent}} is not called
> only for write requests created by our application. There are other
> protocol messages which are handled by MINA while our application is
> streaming which produce multiple writes. For instance in my testing I've
> seen
> {{org.apache.sshd.common.session.helpers.AbstractSession#handleNewKeys}}
> putting messages into the {{DefaultWriteRequestQueue}}.
>
> Regards
> Juan
>
> > Avoid unbounded message queueing when sending large amounts of data to
> slow clients
> >
> ---
> >
> > Key: DIRMINA-1070
> > URL: https://issues.apache.org/jira/browse/DIRMINA-1070
> > Project: MINA
> >  Issue Type: New Feature
> >  Components: Core
> >Reporter: jpalacios
> >  Labels: stability
> >
> > Our application runs an Apache MINA server to provide SSH support. We
> are seeing {{OutOfMemoryError}} s when certain clients establish a session
> with a large {{Window}} size. Particularly clients like TortoiseGit (which
> uses TortoisePlink which in turn seems to use Putty) use an initial window
> size of 2GB. From heap dumps we can see that the
> {{DefaultWriteRequestQueue}} is filling up with {{WriteRequest}} instances
> and taking up gigabytes of space until the heap blows.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.4.14#64029)
>

[jira] [Commented] (DIRMINA-1070) Avoid unbounded message queueing when sending large amounts of data to slow clients

[jpalacios (JIRA)]

[ 
https://issues.apache.org/jira/browse/DIRMINA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148228#comment-16148228
 ] 

jpalacios commented on DIRMINA-1070:


Hi [~elecharny],
Thank you for getting back to me. Let me provide a bit more context to my issue.

Our application can stream gigabytes of data to clients based on a single 
message. We need to ensure that at any given time only a certain amount of 
bytes are inflight for any given session. The one message / one response 
approach I don't think will work for us. Also, in my testing (at least with my 
implementation of {{IoFilter}} I see that {{messageSent}} is not called only 
for write requests created by our application. There are other protocol 
messages which are handled by MINA while our application is streaming which 
produce multiple writes. For instance in my testing I've seen 
{{org.apache.sshd.common.session.helpers.AbstractSession#handleNewKeys}} 
putting messages into the {{DefaultWriteRequestQueue}}.

Regards
Juan

> Avoid unbounded message queueing when sending large amounts of data to slow 
> clients
> ---
>
> Key: DIRMINA-1070
> URL: https://issues.apache.org/jira/browse/DIRMINA-1070
> Project: MINA
>  Issue Type: New Feature
>  Components: Core
>Reporter: jpalacios
>  Labels: stability
>
> Our application runs an Apache MINA server to provide SSH support. We are 
> seeing {{OutOfMemoryError}} s when certain clients establish a session with a 
> large {{Window}} size. Particularly clients like TortoiseGit (which uses 
> TortoisePlink which in turn seems to use Putty) use an initial window size of 
> 2GB. From heap dumps we can see that the {{DefaultWriteRequestQueue}} is 
> filling up with {{WriteRequest}} instances and taking up gigabytes of space 
> until the heap blows. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DIRMINA-1070) Avoid unbounded message queueing when sending large amounts of data to slow clients

[Emmanuel Lecharny (JIRA)]

[ 
https://issues.apache.org/jira/browse/DIRMINA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146987#comment-16146987
 ] 

Emmanuel Lecharny commented on DIRMINA-1070:


This is just plain normal behaviour. You application must take care of a slow 
client.

Basically, you should not send a new response unless the previous one has been 
fully sent (ie, you've got a {{messageSent}} event).

Of course, it might be a bit limitating, so you can decide to wait a bit before 
stopping sending new data to a slow client, but again, this is an application 
decision, not a MINA decision.

All in all, here is what your application should do :

- when receiving a message, process it
- check that the previous message has been sent properly (or aborted, ie an 
exception has been received) checking a flag in the session attributes
- if it's ok, write the response
- otherwise, deal with the issue (ie discard the request, stack the response up 
to a limit your application would set, kill the session, etc)

Keep in mind that MINA is based on an asynchronous mechanism, so your session 
is actually never blocked on write like a synchronous system would, so every 
time you write something back to your client, it's either sent immediately to 
the socket, or if this socket is already full, it's stacked in a queue for a 
later processing (and this is the reason you see the memory being consumed fast 
with slow readers).

> Avoid unbounded message queueing when sending large amounts of data to slow 
> clients
> ---
>
> Key: DIRMINA-1070
> URL: https://issues.apache.org/jira/browse/DIRMINA-1070
> Project: MINA
>  Issue Type: New Feature
>  Components: Core
>Reporter: jpalacios
>  Labels: stability
>
> Our application runs an Apache MINA server to provide SSH support. We are 
> seeing {{OutOfMemoryError}}s when certain clients establish a session with a 
> large {{Window}} size. Particularly clients like TortoiseGit (which uses 
> TortoisePlink which in turn seems to use Putty) use an initial window size of 
> 2GB. From heap dumps we can see that the {{DefaultWriteRequestQueue}} is 
> filling up with {{WriteRequest}} instances and taking up gigabytes of space 
> until the heap blows. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)