[jira] [Comment Edited] (SSHD-952) What is the expected behavior of waitForSpace(long millis) method on default timeout of 30 seconds
[
https://issues.apache.org/jira/browse/SSHD-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969075#comment-16969075
]
Anudeep edited comment on SSHD-952 at 11/7/19 9:20 AM:
---
Hi [~lgoldstein],
We tried in a different manner this time ,instead of blocking the read
completely we added sleep while reading the data from channel on the client
side .
This time even after the timeout it was still continuing for some time and got
exited(channel got closed with an exception after some time). I suppose this
behavior might be due to sleep which we introduced at client side.
{code:java}
2019-11-07 05:42:24,630 INFO [com.ericsson.oss.itpf.EVENT_LOGGER] (EJB default
- 73) [NO USER DATA, Started BNSI NBI, COARSE, BNSI NBI Client, BNSI NBI
Server, ]
2019-11-07 05:42:26,611 DEBUG [org.apache.sshd.server.channel.ChannelSession]
(pool-7-thread-3) flush(ChannelOutputStream[ChannelSession[id=0,
recipient=43]-ServerSessi
onImpl[administrator@/192.168.0.42:49984]] SSH_MSG_CHANNEL_DATA) failed
(SocketTimeoutException) to wait for space of len=23892:
waitForCondition(Window[server/remote](
ChannelSession[id=0,
recipient=43]-ServerSessionImpl[administrator@/192.168.0.42:49984])) timeout
exceeded: 3
pStatus=0.%A
2019-11-07 05:42:33,149 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session]
(sshd-SshServer[57e76ebb](port=8345)-nio2-thread-2)
exceptionCaught(Nio2Session[local=/10.247
.246.52:8345, remote=/192.168.0.42:49984]) caught
InterruptedByTimeoutException[null] - calling handler
2019-11-07 05:42:33,149 WARN
[org.apache.sshd.server.session.ServerSessionImpl]
(sshd-SshServer[57e76ebb](port=8345)-nio2-thread-2)
exceptionCaught(ServerSessionImpl[a
dministrator@/192.168.0.42:49984])[state=Opened] InterruptedByTimeoutException:
null
2019-11-07 05:42:33,150 DEBUG
[org.apache.sshd.server.session.ServerSessionImpl]
(sshd-SshServer[57e76ebb](port=8345)-nio2-thread-2)
exceptionCaught(ServerSessionImpl[a
dministrator@/192.168.0.42:49984])[state=Opened] details:
java.nio.channels.InterruptedByTimeoutException
at
sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:456)
[rt.jar:1.8.0_221]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_221]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_221]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_221]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.8.0_221]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_221]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_221]
:
{code}
Thanks,
Anudeep
was (Author: anudeepgangasani):
Hi [~lgoldstein],
We tried in a different manner this time ,instead of blocking the read
completely we added sleep while reading the data from channel on the client
side .
This time even after the timeout it was still continuing for some time and got
exited(channel got closed with an exception after some time). I suppose this
behavior might be due to sleep which we introduced at client side.
{code:java}
pStatus=0.%A
2019-11-07 05:42:33,149 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session]
(sshd-SshServer[57e76ebb](port=8345)-nio2-thread-2)
exceptionCaught(Nio2Session[local=/10.247
.246.52:8345, remote=/192.168.0.42:49984]) caught
InterruptedByTimeoutException[null] - calling handler
2019-11-07 05:42:33,149 WARN
[org.apache.sshd.server.session.ServerSessionImpl]
(sshd-SshServer[57e76ebb](port=8345)-nio2-thread-2)
exceptionCaught(ServerSessionImpl[a
dministrator@/192.168.0.42:49984])[state=Opened] InterruptedByTimeoutException:
null
2019-11-07 05:42:33,150 DEBUG
[org.apache.sshd.server.session.ServerSessionImpl]
(sshd-SshServer[57e76ebb](port=8345)-nio2-thread-2)
exceptionCaught(ServerSessionImpl[a
dministrator@/192.168.0.42:49984])[state=Opened] details:
java.nio.channels.InterruptedByTimeoutException
at
sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:456)
[rt.jar:1.8.0_221]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_221]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_221]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_221]
at
[jira] [Commented] (SSHD-952) What is the expected behavior of waitForSpace(long millis) method on default timeout of 30 seconds
[
https://issues.apache.org/jira/browse/SSHD-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16969075#comment-16969075
]
Anudeep commented on SSHD-952:
--
Hi [~lgoldstein],
We tried in a different manner this time ,instead of blocking the read
completely we added sleep while reading the data from channel on the client
side .
This time even after the timeout it was still continuing for some time and got
exited(channel got closed with an exception after some time). I suppose this
behavior might be due to sleep which we introduced at client side.
{code:java}
pStatus=0.%A
2019-11-07 05:42:33,149 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session]
(sshd-SshServer[57e76ebb](port=8345)-nio2-thread-2)
exceptionCaught(Nio2Session[local=/10.247
.246.52:8345, remote=/192.168.0.42:49984]) caught
InterruptedByTimeoutException[null] - calling handler
2019-11-07 05:42:33,149 WARN
[org.apache.sshd.server.session.ServerSessionImpl]
(sshd-SshServer[57e76ebb](port=8345)-nio2-thread-2)
exceptionCaught(ServerSessionImpl[a
dministrator@/192.168.0.42:49984])[state=Opened] InterruptedByTimeoutException:
null
2019-11-07 05:42:33,150 DEBUG
[org.apache.sshd.server.session.ServerSessionImpl]
(sshd-SshServer[57e76ebb](port=8345)-nio2-thread-2)
exceptionCaught(ServerSessionImpl[a
dministrator@/192.168.0.42:49984])[state=Opened] details:
java.nio.channels.InterruptedByTimeoutException
at
sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:456)
[rt.jar:1.8.0_221]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_221]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[rt.jar:1.8.0_221]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_221]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.8.0_221]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_221]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_221]
:
{code}
Thanks,
Anudeep
> What is the expected behavior of waitForSpace(long millis) method on default
> timeout of 30 seconds
> --
>
> Key: SSHD-952
> URL: https://issues.apache.org/jira/browse/SSHD-952
> Project: MINA SSHD
> Issue Type: Question
>Affects Versions: 2.2.0
>Reporter: Anudeep
>Priority: Major
>
> Hi Team,
> Can you please let us know the expected behavior of waitForSpace(long millis)
> method of Window.java class . As per our test we see on timeout thread get
> released and it spawns a new thread on timeout. Is this an expected behavior ?
> 2019-10-24 12:33:26,123 DEBUG [org.apache.sshd.server.channel.ChannelSession]
> (Thread-44 (HornetQ-client-global-threads-407809182))
> flush(ChannelOutputStream[ChannelSession[id=0,
> recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]]
> SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of
> len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0,
> recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]))
> timeout exceeded: 2
> 2019-10-24 12:33:46,126 DEBUG [org.apache.sshd.server.channel.ChannelSession]
> (Thread-44 (HornetQ-client-global-threads-407809182))
> flush(ChannelOutputStream[ChannelSession[id=0,
> recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]]
> SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of
> len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0,
> recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]))
> timeout exceeded: 2
> 2019-10-24 12:34:06,129 DEBUG [org.apache.sshd.server.channel.ChannelSession]
> (Thread-44 (HornetQ-client-global-threads-407809182))
> flush(ChannelOutputStream[ChannelSession[id=0,
> recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]]
> SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of
> len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0,
> recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]))
> timeout exceeded: 2
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org
For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Comment Edited] (SSHD-952) What is the expected behavior of waitForSpace(long millis) method on default timeout of 30 seconds
[ https://issues.apache.org/jira/browse/SSHD-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967487#comment-16967487 ] Anudeep edited comment on SSHD-952 at 11/5/19 12:41 PM: In our test we blocked client to read data to make the window full and then we observed it was timing out and spawning a new thread. Client session was not closed. Regards, Anudeep was (Author: anudeepgangasani): In our test we blocked client to read data to make the window full and then we observed it was timing out and then spawning a new thread.Client session was not closed. Regards, Anudeep > What is the expected behavior of waitForSpace(long millis) method on default > timeout of 30 seconds > -- > > Key: SSHD-952 > URL: https://issues.apache.org/jira/browse/SSHD-952 > Project: MINA SSHD > Issue Type: Question >Affects Versions: 2.2.0 >Reporter: Anudeep >Priority: Major > > Hi Team, > Can you please let us know the expected behavior of waitForSpace(long millis) > method of Window.java class . As per our test we see on timeout thread get > released and it spawns a new thread on timeout. Is this an expected behavior ? > 2019-10-24 12:33:26,123 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 > 2019-10-24 12:33:46,126 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 > 2019-10-24 12:34:06,129 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Commented] (SSHD-952) What is the expected behavior of waitForSpace(long millis) method on default timeout of 30 seconds
[ https://issues.apache.org/jira/browse/SSHD-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967487#comment-16967487 ] Anudeep commented on SSHD-952: -- In our test we blocked client to read data to make the window full and then we observed it was timing out and then spawning a new thread.Client session was not closed. Regards, Anudeep > What is the expected behavior of waitForSpace(long millis) method on default > timeout of 30 seconds > -- > > Key: SSHD-952 > URL: https://issues.apache.org/jira/browse/SSHD-952 > Project: MINA SSHD > Issue Type: Question >Affects Versions: 2.2.0 >Reporter: Anudeep >Priority: Major > > Hi Team, > Can you please let us know the expected behavior of waitForSpace(long millis) > method of Window.java class . As per our test we see on timeout thread get > released and it spawns a new thread on timeout. Is this an expected behavior ? > 2019-10-24 12:33:26,123 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 > 2019-10-24 12:33:46,126 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 > 2019-10-24 12:34:06,129 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Comment Edited] (SSHD-952) What is the expected behavior of waitForSpace(long millis) method on default timeout of 30 seconds
[ https://issues.apache.org/jira/browse/SSHD-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962756#comment-16962756 ] Anudeep edited comment on SSHD-952 at 10/30/19 7:20 AM: Hi [~lgoldstein], I believe timeout was introduced to unblock the threads ,in previous versions if there is no space thread was waiiting and if there is closure due to some invalid chunk of data from client thread was blocked for ever and now with new implementation at-least thread gets released after some timeout. We have seen a case in previous versions if there are 3 blocked threads no new session was served as threads are exhausted. I want to understand if current logic will close the client session if the threads gets timed out or will it keep open until it free some space on the channel? Regards, Anudeep was (Author: anudeepgangasani): Hi [~lgoldstein], I believe timeout was introduced to unblock the threads ,in previous versions if there is no space thread was blocked for ever and now with new implementation at-least thread gets released after some timeout. We have seen a case in previous versions if there are 3 blocked threads no new session was served as threads are exhausted. I want to understand if current logic will close the client session if the threads gets timed out or will it keep open until it free some space on the channel? Regards, Anudeep > What is the expected behavior of waitForSpace(long millis) method on default > timeout of 30 seconds > -- > > Key: SSHD-952 > URL: https://issues.apache.org/jira/browse/SSHD-952 > Project: MINA SSHD > Issue Type: Question >Affects Versions: 2.2.0 >Reporter: Anudeep >Priority: Major > > Hi Team, > Can you please let us know the expected behavior of waitForSpace(long millis) > method of Window.java class . As per our test we see on timeout thread get > released and it spawns a new thread on timeout. Is this an expected behavior ? > 2019-10-24 12:33:26,123 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 > 2019-10-24 12:33:46,126 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 > 2019-10-24 12:34:06,129 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Comment Edited] (SSHD-952) What is the expected behavior of waitForSpace(long millis) method on default timeout of 30 seconds
[ https://issues.apache.org/jira/browse/SSHD-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962756#comment-16962756 ] Anudeep edited comment on SSHD-952 at 10/30/19 7:15 AM: Hi [~lgoldstein], I believe timeout was introduced to unblock the threads ,in previous versions if there is no space thread was blocked for ever and now with new implementation at-least thread gets released after some timeout. We have seen a case in previous versions if there are 3 blocked threads no new session was served as threads are exhausted. I want to understand if current logic will close the client session if the threads gets timed out or will it keep open until it free some space on the channel? Regards, Anudeep was (Author: anudeepgangasani): Hi [~lgoldstein], I believe timeout was introduced to unblock the threads ,in previous versions if there is no space thread was blocked for ever and now with new implementation at-least thread gets released after some timeout. We have seen a case in previous versions if there are 3 blocked threads no new session was served as threads are exhausted. I want to understand if current logic will close the client session if the threads gets timed out ? Regards, Anudeep > What is the expected behavior of waitForSpace(long millis) method on default > timeout of 30 seconds > -- > > Key: SSHD-952 > URL: https://issues.apache.org/jira/browse/SSHD-952 > Project: MINA SSHD > Issue Type: Question >Affects Versions: 2.2.0 >Reporter: Anudeep >Priority: Major > > Hi Team, > Can you please let us know the expected behavior of waitForSpace(long millis) > method of Window.java class . As per our test we see on timeout thread get > released and it spawns a new thread on timeout. Is this an expected behavior ? > 2019-10-24 12:33:26,123 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 > 2019-10-24 12:33:46,126 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 > 2019-10-24 12:34:06,129 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Commented] (SSHD-952) What is the expected behavior of waitForSpace(long millis) method on default timeout of 30 seconds
[ https://issues.apache.org/jira/browse/SSHD-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962756#comment-16962756 ] Anudeep commented on SSHD-952: -- Hi [~lgoldstein], I believe timeout was introduced to unblock the threads ,in previous versions if there is no space thread was blocked for ever and now with new implementation at-least thread gets released after some timeout. We have seen a case in previous versions if there are 3 blocked threads no new session was served as threads are exhausted. I want to understand if current logic will close the client session if the threads gets timed out ? Regards, Anudeep > What is the expected behavior of waitForSpace(long millis) method on default > timeout of 30 seconds > -- > > Key: SSHD-952 > URL: https://issues.apache.org/jira/browse/SSHD-952 > Project: MINA SSHD > Issue Type: Question >Affects Versions: 2.2.0 >Reporter: Anudeep >Priority: Major > > Hi Team, > Can you please let us know the expected behavior of waitForSpace(long millis) > method of Window.java class . As per our test we see on timeout thread get > released and it spawns a new thread on timeout. Is this an expected behavior ? > 2019-10-24 12:33:26,123 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 > 2019-10-24 12:33:46,126 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 > 2019-10-24 12:34:06,129 DEBUG [org.apache.sshd.server.channel.ChannelSession] > (Thread-44 (HornetQ-client-global-threads-407809182)) > flush(ChannelOutputStream[ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] > SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of > len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, > recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) > timeout exceeded: 2 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Created] (SSHD-952) What is the expected behavior of waitForSpace(long millis) method on default timeout of 30 seconds
Anudeep created SSHD-952: Summary: What is the expected behavior of waitForSpace(long millis) method on default timeout of 30 seconds Key: SSHD-952 URL: https://issues.apache.org/jira/browse/SSHD-952 Project: MINA SSHD Issue Type: Question Affects Versions: 2.2.0 Reporter: Anudeep Hi Team, Can you please let us know the expected behavior of waitForSpace(long millis) method of Window.java class . As per our test we see on timeout thread get released and it spawns a new thread on timeout. Is this an expected behavior ? 2019-10-24 12:33:26,123 DEBUG [org.apache.sshd.server.channel.ChannelSession] (Thread-44 (HornetQ-client-global-threads-407809182)) flush(ChannelOutputStream[ChannelSession[id=0, recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) timeout exceeded: 2 2019-10-24 12:33:46,126 DEBUG [org.apache.sshd.server.channel.ChannelSession] (Thread-44 (HornetQ-client-global-threads-407809182)) flush(ChannelOutputStream[ChannelSession[id=0, recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) timeout exceeded: 2 2019-10-24 12:34:06,129 DEBUG [org.apache.sshd.server.channel.ChannelSession] (Thread-44 (HornetQ-client-global-threads-407809182)) flush(ChannelOutputStream[ChannelSession[id=0, recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628]] SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of len=1190: waitForCondition(Window[server/remote](ChannelSession[id=0, recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58628])) timeout exceeded: 2 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Updated] (SSHD-940) Query on Thread pool and the ssh threads hanging in blocked state
[ https://issues.apache.org/jira/browse/SSHD-940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anudeep updated SSHD-940: - Description: Hi Team, We face a problem with sshd-core-0.13.0 (whenever a thread wait for some space on the window and the another thread initiate a close on the same connection threads go into blocked state and if the thread count of such threads(BLOCKED state) crosses three no new connections are opened). We have a simple question ,we know this is a bug in 0.13.0 and it was mentioned as fixed in 0.14.0 (https://issues.apache.org/jira/browse/SSHD-348) and now we need a clear picture if this issue is completely resolved in the later versions. If yes please let us know the version. We will migrate to later version based on your confirmation ,we cannot simply go and migrate to a later one and if the problem still exists it will not be a compromising one. Below are my previous requests raised for the same.please have a look on them and let us know if any further information is required from us. https://issues.apache.org/jira/browse/SSHD-938 https://issues.apache.org/jira/browse/SSHD-939 Thanks in Advance. was: Hi Team, We face a problem with sshd-core-0.13.0 (whenever a thread wait for some space on the window and the another thread initiate a close on the same connection threads go into blocked state and if the thread count of such threads(BLOCKED state) crosses three no new connections are opened). We have a simple question ,we know this is a bug in 0.13.0 and was fixed in 0.14.0 (https://issues.apache.org/jira/browse/SSHD-348) and now we need a clear picture if this issue is completely resolved in the later versions. If yes please let us know the version. We will migrate to later version based on your confirmation ,we cannot simply go and migrate to a later one and if the problem still exists it will not be a compromising one. Below are my previous requests raised for the same.please have a look on them and let us know if any further information is required from us. https://issues.apache.org/jira/browse/SSHD-938 https://issues.apache.org/jira/browse/SSHD-939 Thanks in Advance. > Query on Thread pool and the ssh threads hanging in blocked state > - > > Key: SSHD-940 > URL: https://issues.apache.org/jira/browse/SSHD-940 > Project: MINA SSHD > Issue Type: Question >Reporter: Anudeep >Priority: Blocker > > Hi Team, > We face a problem with sshd-core-0.13.0 (whenever a thread wait for some > space on the window and the another thread initiate a close on the same > connection threads go into blocked state and if the thread count of such > threads(BLOCKED state) crosses three no new connections are opened). > We have a simple question ,we know this is a bug in 0.13.0 and it was > mentioned as fixed in 0.14.0 (https://issues.apache.org/jira/browse/SSHD-348) > and now we need a clear picture if this issue is completely resolved in the > later versions. > If yes please let us know the version. > We will migrate to later version based on your confirmation ,we cannot simply > go and migrate to a later one and if the problem still exists it will not be > a compromising one. > Below are my previous requests raised for the same.please have a look on them > and let us know if any further information is required from us. > https://issues.apache.org/jira/browse/SSHD-938 > https://issues.apache.org/jira/browse/SSHD-939 > Thanks in Advance. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Created] (SSHD-940) Query on Thread pool and the ssh threads hanging in blocked state
Anudeep created SSHD-940: Summary: Query on Thread pool and the ssh threads hanging in blocked state Key: SSHD-940 URL: https://issues.apache.org/jira/browse/SSHD-940 Project: MINA SSHD Issue Type: Question Reporter: Anudeep Hi Team, We face a problem with sshd-core-0.13.0 (whenever a thread wait for some space on the window and the another thread initiate a close on the same connection threads go into blocked state and if the thread count of such threads(BLOCKED state) crosses three no new connections are opened). We have a simple question ,we know this is a bug in 0.13.0 and was fixed in 0.14.0 (https://issues.apache.org/jira/browse/SSHD-348) and now we need a clear picture if this issue is completely resolved in the later versions. If yes please let us know the version. We will migrate to later version based on your confirmation ,we cannot simply go and migrate to a later one and if the problem still exists it will not be a compromising one. Below are my previous requests raised for the same.please have a look on them and let us know if any further information is required from us. https://issues.apache.org/jira/browse/SSHD-938 https://issues.apache.org/jira/browse/SSHD-939 Thanks in Advance. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Commented] (SSHD-939) SSH threads get blocked in Object.wait() method forever resulting in CLOSE_WAIT connections
[
https://issues.apache.org/jira/browse/SSHD-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929037#comment-16929037
]
Anudeep commented on SSHD-939:
--
Hi [~lgoldstein],
This was my previous question is it posible to figure out the cipher from the
packet which we receive.
Regards,
Anudeep
> SSH threads get blocked in Object.wait() method forever resulting in
> CLOSE_WAIT connections
> ---
>
> Key: SSHD-939
> URL: https://issues.apache.org/jira/browse/SSHD-939
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Anudeep
>Assignee: Goldstein Lyor
>Priority: Major
> Attachments: svc-1-bnsiserv_06_09_2019-server.7z,
> svc-1-bnsiserv_thread.log
>
>
> Ssh threads get blocked in object.wait() for ever and if the count of such
> threads crosses 3 no more ssh requests are served.
> We had an analysis of the logs and from the thread model of apache we see the
> maximum number of I/O threads is number of cores +1 in our server the number
> of cores is 2.
> SO after three such blocked threads we are not able to open any new ssh
> connection.
> Can you please confirm if the above logic is correct .
>
> We too found a ticket raised in 0.14.0
> (https://issues.apache.org/jira/browse/SSHD-348)which says threads in blocked
> state are fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will
> fix the threads to be released and do not hung the process even if there are
> more than 3 of such a kind.
>
> Logs from thread dumps
> {code:java}
> From the thread dump it looks like we have a deadlock, e.g.
> In one thread we're trying to send out data on an SSH connection, but we're
> waiting for "space"
> "pool-7-thread-2213" #2677 prio=5 os_prio=0 tid=0x7f85554cc800 nid=0x61e3
> in Object.wait() [0x7f84ea29f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:148)
> - locked <0x000764942ad8> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116)
> - eliminated <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84)
> - locked <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
> at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
> at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
> at java.io.BufferedWriter.write(BufferedWriter.java:188)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.PrintWriter.write(PrintWriter.java:426)
> - locked <0x0007649434d8> (a java.io.BufferedWriter)
> at java.io.PrintWriter.write(PrintWriter.java:443)
> In another thread, we trying to close the same connection
> "sshd-SshServer[1ae27e4e]-nio2-thread-3" #356 prio=5 os_prio=0
> tid=0x015f6000 nid=0x30b8 waiting for monitor entry
> [0x7f851ab9c000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:146)
> - waiting to lock <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommandContext.shutdown(SshCommandContext.java:146)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand$ContextStarter.shutdown(SshCommand.java:104)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand.destroy(SshCommand.java:78)
> at
> org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:242)
> at
> org.apache.sshd.common.util.CloseableUtils$AbstractCloseable.close(CloseableUtils.java:276)
> at
> org.apache.sshd.common.util.CloseableUtils$ParallelCloseable.doClose(CloseableUtils.java:182)
> at
> org.apache.sshd.common.util.CloseableUtils$SimpleCloseable.close(CloseableUtils.java:151)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onFailed(Nio2Session.java:199)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$2.run(Nio2CompletionHandler.java:41)
> at java.security.AccessController.doPrivileged(Native Method)
> at
>
[jira] [Commented] (SSHD-939) SSH threads get blocked in Object.wait() method forever resulting in CLOSE_WAIT connections
[
https://issues.apache.org/jira/browse/SSHD-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929033#comment-16929033
]
Anudeep commented on SSHD-939:
--
Hi [~lgoldstein],
I am sorry but i need to ask for a clear picture. Cipher here you mean the
client one or the server one? What do you mean by decode bug ? The exceptions
invalid packet length is seen when we receive a packet from client.
Regards,
Anudeep
> SSH threads get blocked in Object.wait() method forever resulting in
> CLOSE_WAIT connections
> ---
>
> Key: SSHD-939
> URL: https://issues.apache.org/jira/browse/SSHD-939
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Anudeep
>Assignee: Goldstein Lyor
>Priority: Major
> Attachments: svc-1-bnsiserv_06_09_2019-server.7z,
> svc-1-bnsiserv_thread.log
>
>
> Ssh threads get blocked in object.wait() for ever and if the count of such
> threads crosses 3 no more ssh requests are served.
> We had an analysis of the logs and from the thread model of apache we see the
> maximum number of I/O threads is number of cores +1 in our server the number
> of cores is 2.
> SO after three such blocked threads we are not able to open any new ssh
> connection.
> Can you please confirm if the above logic is correct .
>
> We too found a ticket raised in 0.14.0
> (https://issues.apache.org/jira/browse/SSHD-348)which says threads in blocked
> state are fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will
> fix the threads to be released and do not hung the process even if there are
> more than 3 of such a kind.
>
> Logs from thread dumps
> {code:java}
> From the thread dump it looks like we have a deadlock, e.g.
> In one thread we're trying to send out data on an SSH connection, but we're
> waiting for "space"
> "pool-7-thread-2213" #2677 prio=5 os_prio=0 tid=0x7f85554cc800 nid=0x61e3
> in Object.wait() [0x7f84ea29f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:148)
> - locked <0x000764942ad8> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116)
> - eliminated <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84)
> - locked <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
> at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
> at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
> at java.io.BufferedWriter.write(BufferedWriter.java:188)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.PrintWriter.write(PrintWriter.java:426)
> - locked <0x0007649434d8> (a java.io.BufferedWriter)
> at java.io.PrintWriter.write(PrintWriter.java:443)
> In another thread, we trying to close the same connection
> "sshd-SshServer[1ae27e4e]-nio2-thread-3" #356 prio=5 os_prio=0
> tid=0x015f6000 nid=0x30b8 waiting for monitor entry
> [0x7f851ab9c000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:146)
> - waiting to lock <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommandContext.shutdown(SshCommandContext.java:146)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand$ContextStarter.shutdown(SshCommand.java:104)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand.destroy(SshCommand.java:78)
> at
> org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:242)
> at
> org.apache.sshd.common.util.CloseableUtils$AbstractCloseable.close(CloseableUtils.java:276)
> at
> org.apache.sshd.common.util.CloseableUtils$ParallelCloseable.doClose(CloseableUtils.java:182)
> at
> org.apache.sshd.common.util.CloseableUtils$SimpleCloseable.close(CloseableUtils.java:151)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onFailed(Nio2Session.java:199)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$2.run(Nio2CompletionHandler.java:41)
>
[jira] [Comment Edited] (SSHD-939) SSH threads get blocked in Object.wait() method forever resulting in CLOSE_WAIT connections
[
https://issues.apache.org/jira/browse/SSHD-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927711#comment-16927711
]
Anudeep edited comment on SSHD-939 at 9/11/19 4:00 PM:
---
Please answer above questions just for the understanding and then you can
proceed with the closure.
was (Author: anudeepgangasani):
Please answer below and then you can proceed with the closure.
> SSH threads get blocked in Object.wait() method forever resulting in
> CLOSE_WAIT connections
> ---
>
> Key: SSHD-939
> URL: https://issues.apache.org/jira/browse/SSHD-939
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Anudeep
>Assignee: Goldstein Lyor
>Priority: Major
> Attachments: svc-1-bnsiserv_06_09_2019-server.7z,
> svc-1-bnsiserv_thread.log
>
>
> Ssh threads get blocked in object.wait() for ever and if the count of such
> threads crosses 3 no more ssh requests are served.
> We had an analysis of the logs and from the thread model of apache we see the
> maximum number of I/O threads is number of cores +1 in our server the number
> of cores is 2.
> SO after three such blocked threads we are not able to open any new ssh
> connection.
> Can you please confirm if the above logic is correct .
>
> We too found a ticket raised in 0.14.0
> (https://issues.apache.org/jira/browse/SSHD-348)which says threads in blocked
> state are fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will
> fix the threads to be released and do not hung the process even if there are
> more than 3 of such a kind.
>
> Logs from thread dumps
> {code:java}
> From the thread dump it looks like we have a deadlock, e.g.
> In one thread we're trying to send out data on an SSH connection, but we're
> waiting for "space"
> "pool-7-thread-2213" #2677 prio=5 os_prio=0 tid=0x7f85554cc800 nid=0x61e3
> in Object.wait() [0x7f84ea29f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:148)
> - locked <0x000764942ad8> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116)
> - eliminated <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84)
> - locked <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
> at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
> at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
> at java.io.BufferedWriter.write(BufferedWriter.java:188)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.PrintWriter.write(PrintWriter.java:426)
> - locked <0x0007649434d8> (a java.io.BufferedWriter)
> at java.io.PrintWriter.write(PrintWriter.java:443)
> In another thread, we trying to close the same connection
> "sshd-SshServer[1ae27e4e]-nio2-thread-3" #356 prio=5 os_prio=0
> tid=0x015f6000 nid=0x30b8 waiting for monitor entry
> [0x7f851ab9c000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:146)
> - waiting to lock <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommandContext.shutdown(SshCommandContext.java:146)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand$ContextStarter.shutdown(SshCommand.java:104)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand.destroy(SshCommand.java:78)
> at
> org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:242)
> at
> org.apache.sshd.common.util.CloseableUtils$AbstractCloseable.close(CloseableUtils.java:276)
> at
> org.apache.sshd.common.util.CloseableUtils$ParallelCloseable.doClose(CloseableUtils.java:182)
> at
> org.apache.sshd.common.util.CloseableUtils$SimpleCloseable.close(CloseableUtils.java:151)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onFailed(Nio2Session.java:199)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$2.run(Nio2CompletionHandler.java:41)
> at
[jira] [Reopened] (SSHD-939) SSH threads get blocked in Object.wait() method forever resulting in CLOSE_WAIT connections
[
https://issues.apache.org/jira/browse/SSHD-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anudeep reopened SSHD-939:
--
Please answer below and then you can proceed with the closure.
> SSH threads get blocked in Object.wait() method forever resulting in
> CLOSE_WAIT connections
> ---
>
> Key: SSHD-939
> URL: https://issues.apache.org/jira/browse/SSHD-939
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Anudeep
>Assignee: Goldstein Lyor
>Priority: Major
> Attachments: svc-1-bnsiserv_06_09_2019-server.7z,
> svc-1-bnsiserv_thread.log
>
>
> Ssh threads get blocked in object.wait() for ever and if the count of such
> threads crosses 3 no more ssh requests are served.
> We had an analysis of the logs and from the thread model of apache we see the
> maximum number of I/O threads is number of cores +1 in our server the number
> of cores is 2.
> SO after three such blocked threads we are not able to open any new ssh
> connection.
> Can you please confirm if the above logic is correct .
>
> We too found a ticket raised in 0.14.0
> (https://issues.apache.org/jira/browse/SSHD-348)which says threads in blocked
> state are fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will
> fix the threads to be released and do not hung the process even if there are
> more than 3 of such a kind.
>
> Logs from thread dumps
> {code:java}
> From the thread dump it looks like we have a deadlock, e.g.
> In one thread we're trying to send out data on an SSH connection, but we're
> waiting for "space"
> "pool-7-thread-2213" #2677 prio=5 os_prio=0 tid=0x7f85554cc800 nid=0x61e3
> in Object.wait() [0x7f84ea29f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:148)
> - locked <0x000764942ad8> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116)
> - eliminated <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84)
> - locked <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
> at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
> at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
> at java.io.BufferedWriter.write(BufferedWriter.java:188)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.PrintWriter.write(PrintWriter.java:426)
> - locked <0x0007649434d8> (a java.io.BufferedWriter)
> at java.io.PrintWriter.write(PrintWriter.java:443)
> In another thread, we trying to close the same connection
> "sshd-SshServer[1ae27e4e]-nio2-thread-3" #356 prio=5 os_prio=0
> tid=0x015f6000 nid=0x30b8 waiting for monitor entry
> [0x7f851ab9c000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:146)
> - waiting to lock <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommandContext.shutdown(SshCommandContext.java:146)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand$ContextStarter.shutdown(SshCommand.java:104)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand.destroy(SshCommand.java:78)
> at
> org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:242)
> at
> org.apache.sshd.common.util.CloseableUtils$AbstractCloseable.close(CloseableUtils.java:276)
> at
> org.apache.sshd.common.util.CloseableUtils$ParallelCloseable.doClose(CloseableUtils.java:182)
> at
> org.apache.sshd.common.util.CloseableUtils$SimpleCloseable.close(CloseableUtils.java:151)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onFailed(Nio2Session.java:199)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$2.run(Nio2CompletionHandler.java:41)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:39)
> at
>
[jira] [Commented] (SSHD-939) SSH threads get blocked in Object.wait() method forever resulting in CLOSE_WAIT connections
[
https://issues.apache.org/jira/browse/SSHD-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927710#comment-16927710
]
Anudeep commented on SSHD-939:
--
Hi [~lgoldstein] ,
Its ok ,we suppose to get back to you for help if we face some hurdles in
uplifting.
One last question which we do not understand from logs is invalid length.
Is there a way to decode it .The log in Abstract session prints the encrypted
packet length /actual packet length ?
Below one
// Read packet length decoderLength = decoderBuffer.getInt();
{code:java}
for (;;) {
// Wait for beginning of packet
if (decoderState == 0) {
// The read position should always be 0 at this point because
we have compacted this buffer
assert decoderBuffer.rpos() == 0;
// If we have received enough bytes, start processing those
if (decoderBuffer.available() > inCipherSize) {
// Decrypt the first bytes
if (inCipher != null) {
inCipher.update(decoderBuffer.array(), 0, inCipherSize);
}
// Read packet length
decoderLength = decoderBuffer.getInt();
// Check packet length validity
if (decoderLength < 5 || decoderLength > (256 * 1024)) {
log.info("Error decoding packet (invalid length) {}",
decoderBuffer.printHex());
throw new
SshException(SshConstants.SSH2_DISCONNECT_PROTOCOL_ERROR,
"Invalid packet length: " +
decoderLength);
}
// Ok, that's good, we can go to the next step
decoderState = 1;
} else {
// need more data
break;
} {code}
Why we ask is the previous log before invalid length says it has read 184 bytes
and in the exception we see it as some long negative integer and even some
times a long positive integer.
{code:java}
2019-08-22 09:53:29,106 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session]
(sshd-SshServer[6162dce4]-nio2-th
read-2) Read 184 bytes
2019-08-22 09:53:29,106 INFO [org.apache.sshd.server.session.ServerSession]
(sshd-SshServer[6162dce4]-nio2-
thread-2) Error decoding packet (invalid length) a5 5f 74 5f a6 6f ea 41 02 c0
0a 54 83 22 e4 77 53 7d 0d b9
25 de 36 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8
ab f7 00 bf 4d 40 f5 56 b7 61
9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc aa 5b de 40 85 71 12
cc f6 c0 34 8d 0e 75 95 a5 5b
8d f6 b9 f6 94 04 d8 b5 25 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52
2d 93 4c 88 bc 17 53 ca eb 34
23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba 80
34 55 3f 38 49 0c a9 80 8c a3
fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 c4
2019-08-22 09:53:29,106 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session]
(sshd-SshServer[6162dce4]-nio2-th
read-2) Caught exception, now calling handler
2019-08-22 09:53:29,106 WARN [org.apache.sshd.server.session.ServerSession]
(sshd-SshServer[6162dce4]-nio2-
thread-2) Exception caught: org.apache.sshd.common.SshException: Invalid packet
length: -16976455
at
org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:677)
[sshd-core-0.13.0
.jar:0.13.0]
{code}
Please let us know if there is way to know what has caused this invalid length.
Regards,
Anudeep
> SSH threads get blocked in Object.wait() method forever resulting in
> CLOSE_WAIT connections
> ---
>
> Key: SSHD-939
> URL: https://issues.apache.org/jira/browse/SSHD-939
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Anudeep
>Assignee: Goldstein Lyor
>Priority: Major
> Attachments: svc-1-bnsiserv_06_09_2019-server.7z,
> svc-1-bnsiserv_thread.log
>
>
> Ssh threads get blocked in object.wait() for ever and if the count of such
> threads crosses 3 no more ssh requests are served.
> We had an analysis of the logs and from the thread model of apache we see the
> maximum number of I/O threads is number of cores +1 in our server the number
> of cores is 2.
> SO after three such blocked threads we are not able to open any new ssh
> connection.
> Can you please confirm if the above logic is correct .
>
> We too found a ticket raised in 0.14.0
> (https://issues.apache.org/jira/browse/SSHD-348)which says threads in blocked
> state are fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will
> fix the threads to be released and do not hung the process even if there are
> more than 3 of such a kind.
>
> Logs from thread dumps
>
[jira] [Updated] (SSHD-939) SSH threads get blocked in Object.wait() method forever resulting in CLOSE_WAIT connections
[
https://issues.apache.org/jira/browse/SSHD-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anudeep updated SSHD-939:
-
Attachment: svc-1-bnsiserv_06_09_2019-server.7z
> SSH threads get blocked in Object.wait() method forever resulting in
> CLOSE_WAIT connections
> ---
>
> Key: SSHD-939
> URL: https://issues.apache.org/jira/browse/SSHD-939
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Anudeep
>Priority: Major
> Attachments: svc-1-bnsiserv_06_09_2019-server.7z,
> svc-1-bnsiserv_thread.log
>
>
> Ssh threads get blocked in object.wait() for ever and if the count of such
> threads crosses 3 no more ssh requests are served.
> We had an analysis of the logs and from the thread model of apache we see the
> maximum number of I/O threads is number of cores +1 in our server the number
> of cores is 2.
> SO after three such blocked threads we are not able to open any new ssh
> connection.
> Can you please confirm if the above logic is correct .
>
> We too found a ticket raised in 0.14.0
> (https://issues.apache.org/jira/browse/SSHD-348)which says threads in blocked
> state are fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will
> fix the threads to be released and do not hung the process even if there are
> more than 3 of such a kind.
>
> Logs from thread dumps
> {code:java}
> From the thread dump it looks like we have a deadlock, e.g.
> In one thread we're trying to send out data on an SSH connection, but we're
> waiting for "space"
> "pool-7-thread-2213" #2677 prio=5 os_prio=0 tid=0x7f85554cc800 nid=0x61e3
> in Object.wait() [0x7f84ea29f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:148)
> - locked <0x000764942ad8> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116)
> - eliminated <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84)
> - locked <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
> at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
> at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
> at java.io.BufferedWriter.write(BufferedWriter.java:188)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.PrintWriter.write(PrintWriter.java:426)
> - locked <0x0007649434d8> (a java.io.BufferedWriter)
> at java.io.PrintWriter.write(PrintWriter.java:443)
> In another thread, we trying to close the same connection
> "sshd-SshServer[1ae27e4e]-nio2-thread-3" #356 prio=5 os_prio=0
> tid=0x015f6000 nid=0x30b8 waiting for monitor entry
> [0x7f851ab9c000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:146)
> - waiting to lock <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommandContext.shutdown(SshCommandContext.java:146)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand$ContextStarter.shutdown(SshCommand.java:104)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand.destroy(SshCommand.java:78)
> at
> org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:242)
> at
> org.apache.sshd.common.util.CloseableUtils$AbstractCloseable.close(CloseableUtils.java:276)
> at
> org.apache.sshd.common.util.CloseableUtils$ParallelCloseable.doClose(CloseableUtils.java:182)
> at
> org.apache.sshd.common.util.CloseableUtils$SimpleCloseable.close(CloseableUtils.java:151)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onFailed(Nio2Session.java:199)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$2.run(Nio2CompletionHandler.java:41)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:39)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:195)
> at
>
[jira] [Updated] (SSHD-939) SSH threads get blocked in Object.wait() method forever resulting in CLOSE_WAIT connections
[
https://issues.apache.org/jira/browse/SSHD-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anudeep updated SSHD-939:
-
Attachment: svc-1-bnsiserv_thread.log
> SSH threads get blocked in Object.wait() method forever resulting in
> CLOSE_WAIT connections
> ---
>
> Key: SSHD-939
> URL: https://issues.apache.org/jira/browse/SSHD-939
> Project: MINA SSHD
> Issue Type: Bug
>Affects Versions: 0.13.0
>Reporter: Anudeep
>Priority: Major
> Attachments: svc-1-bnsiserv_thread.log
>
>
> Ssh threads get blocked in object.wait() for ever and if the count of such
> threads crosses 3 no more ssh requests are served.
> We had an analysis of the logs and from the thread model of apache we see the
> maximum number of I/O threads is number of cores +1 in our server the number
> of cores is 2.
> SO after three such blocked threads we are not able to open any new ssh
> connection.
> Can you please confirm if the above logic is correct .
>
> We too found a ticket raised in 0.14.0
> (https://issues.apache.org/jira/browse/SSHD-348)which says threads in blocked
> state are fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will
> fix the threads to be released and do not hung the process even if there are
> more than 3 of such a kind.
>
> Logs from thread dumps
> {code:java}
> From the thread dump it looks like we have a deadlock, e.g.
> In one thread we're trying to send out data on an SSH connection, but we're
> waiting for "space"
> "pool-7-thread-2213" #2677 prio=5 os_prio=0 tid=0x7f85554cc800 nid=0x61e3
> in Object.wait() [0x7f84ea29f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:148)
> - locked <0x000764942ad8> (a
> org.apache.sshd.common.channel.Window)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116)
> - eliminated <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84)
> - locked <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
> at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
> at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
> at java.io.BufferedWriter.write(BufferedWriter.java:188)
> - locked <0x000764943500> (a java.io.OutputStreamWriter)
> at java.io.PrintWriter.write(PrintWriter.java:426)
> - locked <0x0007649434d8> (a java.io.BufferedWriter)
> at java.io.PrintWriter.write(PrintWriter.java:443)
> In another thread, we trying to close the same connection
> "sshd-SshServer[1ae27e4e]-nio2-thread-3" #356 prio=5 os_prio=0
> tid=0x015f6000 nid=0x30b8 waiting for monitor entry
> [0x7f851ab9c000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:146)
> - waiting to lock <0x000764942c20> (a
> org.apache.sshd.common.channel.ChannelOutputStream)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommandContext.shutdown(SshCommandContext.java:146)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand$ContextStarter.shutdown(SshCommand.java:104)
> at
> com.ericsson.oss.nbi.connection.ssh.SshCommand.destroy(SshCommand.java:78)
> at
> org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:242)
> at
> org.apache.sshd.common.util.CloseableUtils$AbstractCloseable.close(CloseableUtils.java:276)
> at
> org.apache.sshd.common.util.CloseableUtils$ParallelCloseable.doClose(CloseableUtils.java:182)
> at
> org.apache.sshd.common.util.CloseableUtils$SimpleCloseable.close(CloseableUtils.java:151)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onFailed(Nio2Session.java:199)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$2.run(Nio2CompletionHandler.java:41)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:39)
> at
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:195)
> at
>
[jira] [Created] (SSHD-939) SSH threads get blocked in Object.wait() method forever resulting in CLOSE_WAIT connections
Anudeep created SSHD-939:
Summary: SSH threads get blocked in Object.wait() method forever
resulting in CLOSE_WAIT connections
Key: SSHD-939
URL: https://issues.apache.org/jira/browse/SSHD-939
Project: MINA SSHD
Issue Type: Bug
Affects Versions: 0.13.0
Reporter: Anudeep
Ssh threads get blocked in object.wait() for ever and if the count of such
threads crosses 3 no more ssh requests are served.
We had an analysis of the logs and from the thread model of apache we see the
maximum number of I/O threads is number of cores +1 in our server the number of
cores is 2.
SO after three such blocked threads we are not able to open any new ssh
connection.
Can you please confirm if the above logic is correct .
We too found a ticket raised in 0.14.0 which says threads in blocked state are
fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will fix the
threads to release and do not hung the process even if there are more than 3 of
such a kind of threads in blocked state.
Logs from thread dumps
{code:java}
>From the thread dump it looks like we have a deadlock, e.g.
In one thread we're trying to send out data on an SSH connection, but we're
waiting for "space"
"pool-7-thread-2213" #2677 prio=5 os_prio=0 tid=0x7f85554cc800 nid=0x61e3
in Object.wait() [0x7f84ea29f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:148)
- locked <0x000764942ad8> (a org.apache.sshd.common.channel.Window)
at
org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116)
- eliminated <0x000764942c20> (a
org.apache.sshd.common.channel.ChannelOutputStream)
at
org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84)
- locked <0x000764942c20> (a
org.apache.sshd.common.channel.ChannelOutputStream)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
- locked <0x000764943500> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at java.io.BufferedWriter.write(BufferedWriter.java:188)
- locked <0x000764943500> (a java.io.OutputStreamWriter)
at java.io.PrintWriter.write(PrintWriter.java:426)
- locked <0x0007649434d8> (a java.io.BufferedWriter)
at java.io.PrintWriter.write(PrintWriter.java:443)
In another thread, we trying to close the same connection
"sshd-SshServer[1ae27e4e]-nio2-thread-3" #356 prio=5 os_prio=0
tid=0x015f6000 nid=0x30b8 waiting for monitor entry [0x7f851ab9c000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:146)
- waiting to lock <0x000764942c20> (a
org.apache.sshd.common.channel.ChannelOutputStream)
at
com.ericsson.oss.nbi.connection.ssh.SshCommandContext.shutdown(SshCommandContext.java:146)
at
com.ericsson.oss.nbi.connection.ssh.SshCommand$ContextStarter.shutdown(SshCommand.java:104)
at
com.ericsson.oss.nbi.connection.ssh.SshCommand.destroy(SshCommand.java:78)
at
org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:242)
at
org.apache.sshd.common.util.CloseableUtils$AbstractCloseable.close(CloseableUtils.java:276)
at
org.apache.sshd.common.util.CloseableUtils$ParallelCloseable.doClose(CloseableUtils.java:182)
at
org.apache.sshd.common.util.CloseableUtils$SimpleCloseable.close(CloseableUtils.java:151)
at
org.apache.sshd.common.io.nio2.Nio2Session$1.onFailed(Nio2Session.java:199)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler$2.run(Nio2CompletionHandler.java:41)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:39)
at
org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:195)
at
org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at sun.nio.ch.Invoker$2.run(Invoker.java:218)
at
[jira] [Updated] (SSHD-939) SSH threads get blocked in Object.wait() method forever resulting in CLOSE_WAIT connections
[
https://issues.apache.org/jira/browse/SSHD-939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anudeep updated SSHD-939:
-
Description:
Ssh threads get blocked in object.wait() for ever and if the count of such
threads crosses 3 no more ssh requests are served.
We had an analysis of the logs and from the thread model of apache we see the
maximum number of I/O threads is number of cores +1 in our server the number of
cores is 2.
SO after three such blocked threads we are not able to open any new ssh
connection.
Can you please confirm if the above logic is correct .
We too found a ticket raised in 0.14.0
(https://issues.apache.org/jira/browse/SSHD-348)which says threads in blocked
state are fixed with 0.14.0 so we need a confirmation from you if 0.14.0 will
fix the threads to be released and do not hung the process even if there are
more than 3 of such a kind.
Logs from thread dumps
{code:java}
>From the thread dump it looks like we have a deadlock, e.g.
In one thread we're trying to send out data on an SSH connection, but we're
waiting for "space"
"pool-7-thread-2213" #2677 prio=5 os_prio=0 tid=0x7f85554cc800 nid=0x61e3
in Object.wait() [0x7f84ea29f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:148)
- locked <0x000764942ad8> (a org.apache.sshd.common.channel.Window)
at
org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:116)
- eliminated <0x000764942c20> (a
org.apache.sshd.common.channel.ChannelOutputStream)
at
org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:84)
- locked <0x000764942c20> (a
org.apache.sshd.common.channel.ChannelOutputStream)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
- locked <0x000764943500> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at java.io.BufferedWriter.write(BufferedWriter.java:188)
- locked <0x000764943500> (a java.io.OutputStreamWriter)
at java.io.PrintWriter.write(PrintWriter.java:426)
- locked <0x0007649434d8> (a java.io.BufferedWriter)
at java.io.PrintWriter.write(PrintWriter.java:443)
In another thread, we trying to close the same connection
"sshd-SshServer[1ae27e4e]-nio2-thread-3" #356 prio=5 os_prio=0
tid=0x015f6000 nid=0x30b8 waiting for monitor entry [0x7f851ab9c000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:146)
- waiting to lock <0x000764942c20> (a
org.apache.sshd.common.channel.ChannelOutputStream)
at
com.ericsson.oss.nbi.connection.ssh.SshCommandContext.shutdown(SshCommandContext.java:146)
at
com.ericsson.oss.nbi.connection.ssh.SshCommand$ContextStarter.shutdown(SshCommand.java:104)
at
com.ericsson.oss.nbi.connection.ssh.SshCommand.destroy(SshCommand.java:78)
at
org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:242)
at
org.apache.sshd.common.util.CloseableUtils$AbstractCloseable.close(CloseableUtils.java:276)
at
org.apache.sshd.common.util.CloseableUtils$ParallelCloseable.doClose(CloseableUtils.java:182)
at
org.apache.sshd.common.util.CloseableUtils$SimpleCloseable.close(CloseableUtils.java:151)
at
org.apache.sshd.common.io.nio2.Nio2Session$1.onFailed(Nio2Session.java:199)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler$2.run(Nio2CompletionHandler.java:41)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:39)
at
org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:195)
at
org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32)
at java.security.AccessController.doPrivileged(Native Method)
at
org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30)
at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
at sun.nio.ch.Invoker$2.run(Invoker.java:218)
at
sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
[jira] [Commented] (SSHD-938) SSH sessions being closed with invalid packet length
[ https://issues.apache.org/jira/browse/SSHD-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924066#comment-16924066 ] Anudeep commented on SSHD-938: -- Hi [~lgoldstein], One quick question. Is there a way to decrypt the above packet to understand what actually it is. I mean this message Error decoding packet (invalid length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 c42019-08-22 09:53:29,106 Regards, Anudeep > SSH sessions being closed with invalid packet length > > > Key: SSHD-938 > URL: https://issues.apache.org/jira/browse/SSHD-938 > Project: MINA SSHD > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Anudeep >Assignee: Goldstein Lyor >Priority: Blocker > Attachments: server.7z > > > SSH connections are being closed with invalid packet length and TCP > connection goes into CLOSE_WAIT state and once CLOSE_WAIT state count crosses > 4 no more SSH connection is being allowed. > with SSHD-core-0.13.0 > > Below is the error > 2019-08-22 09:53:29,106 INFO [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid > length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 > 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 > bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc > aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 > 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb > 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba > 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 > c42019-08-22 09:53:29,106 INFO > [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid > length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 > 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 > bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc > aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 > 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb > 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba > 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 > c42019-08-22 09:53:29,106 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session] > (sshd-SshServer[6162dce4]-nio2-thread-2) Caught exception, now calling > handler2019-08-22 09:53:29,106 WARN > [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Exception caught: > org.apache.sshd.common.SshException: Invalid packet length: -16976455 at > org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:677) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:283) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:184) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32) > [sshd-core-0.13.0.jar:0.13.0] at > java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_172] > at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30) > [sshd-core-0.13.0.jar:0.13.0] at > sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [rt.jar:1.8.0_172] at > sun.nio.ch.Invoker$2.run(Invoker.java:218) [rt.jar:1.8.0_172] at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > [rt.jar:1.8.0_172] at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [rt.jar:1.8.0_172] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail:
[jira] [Reopened] (SSHD-938) SSH sessions being closed with invalid packet length
[ https://issues.apache.org/jira/browse/SSHD-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anudeep reopened SSHD-938: -- > SSH sessions being closed with invalid packet length > > > Key: SSHD-938 > URL: https://issues.apache.org/jira/browse/SSHD-938 > Project: MINA SSHD > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Anudeep >Assignee: Goldstein Lyor >Priority: Blocker > Attachments: server.7z > > > SSH connections are being closed with invalid packet length and TCP > connection goes into CLOSE_WAIT state and once CLOSE_WAIT state count crosses > 4 no more SSH connection is being allowed. > with SSHD-core-0.13.0 > > Below is the error > 2019-08-22 09:53:29,106 INFO [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid > length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 > 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 > bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc > aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 > 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb > 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba > 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 > c42019-08-22 09:53:29,106 INFO > [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid > length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 > 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 > bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc > aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 > 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb > 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba > 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 > c42019-08-22 09:53:29,106 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session] > (sshd-SshServer[6162dce4]-nio2-thread-2) Caught exception, now calling > handler2019-08-22 09:53:29,106 WARN > [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Exception caught: > org.apache.sshd.common.SshException: Invalid packet length: -16976455 at > org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:677) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:283) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:184) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32) > [sshd-core-0.13.0.jar:0.13.0] at > java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_172] > at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30) > [sshd-core-0.13.0.jar:0.13.0] at > sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [rt.jar:1.8.0_172] at > sun.nio.ch.Invoker$2.run(Invoker.java:218) [rt.jar:1.8.0_172] at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > [rt.jar:1.8.0_172] at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [rt.jar:1.8.0_172] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Commented] (SSHD-938) SSH sessions being closed with invalid packet length
[ https://issues.apache.org/jira/browse/SSHD-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915471#comment-16915471 ] Anudeep commented on SSHD-938: -- Hi [~lgoldstein], Are there any known issues around which we have raised. Regrads, Anudeep > SSH sessions being closed with invalid packet length > > > Key: SSHD-938 > URL: https://issues.apache.org/jira/browse/SSHD-938 > Project: MINA SSHD > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Anudeep >Assignee: Goldstein Lyor >Priority: Blocker > Attachments: server.7z > > > SSH connections are being closed with invalid packet length and TCP > connection goes into CLOSE_WAIT state and once CLOSE_WAIT state count crosses > 4 no more SSH connection is being allowed. > with SSHD-core-0.13.0 > > Below is the error > 2019-08-22 09:53:29,106 INFO [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid > length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 > 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 > bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc > aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 > 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb > 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba > 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 > c42019-08-22 09:53:29,106 INFO > [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid > length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 > 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 > bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc > aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 > 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb > 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba > 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 > c42019-08-22 09:53:29,106 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session] > (sshd-SshServer[6162dce4]-nio2-thread-2) Caught exception, now calling > handler2019-08-22 09:53:29,106 WARN > [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Exception caught: > org.apache.sshd.common.SshException: Invalid packet length: -16976455 at > org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:677) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:283) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:184) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32) > [sshd-core-0.13.0.jar:0.13.0] at > java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_172] > at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30) > [sshd-core-0.13.0.jar:0.13.0] at > sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [rt.jar:1.8.0_172] at > sun.nio.ch.Invoker$2.run(Invoker.java:218) [rt.jar:1.8.0_172] at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > [rt.jar:1.8.0_172] at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [rt.jar:1.8.0_172] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Commented] (SSHD-938) SSH sessions being closed with invalid packet length
[ https://issues.apache.org/jira/browse/SSHD-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915296#comment-16915296 ] Anudeep commented on SSHD-938: -- One last question why only 4 closewait connection block further requests.we expect some answer. > SSH sessions being closed with invalid packet length > > > Key: SSHD-938 > URL: https://issues.apache.org/jira/browse/SSHD-938 > Project: MINA SSHD > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Anudeep >Assignee: Goldstein Lyor >Priority: Blocker > Attachments: server.7z > > > SSH connections are being closed with invalid packet length and TCP > connection goes into CLOSE_WAIT state and once CLOSE_WAIT state count crosses > 4 no more SSH connection is being allowed. > with SSHD-core-0.13.0 > > Below is the error > 2019-08-22 09:53:29,106 INFO [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid > length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 > 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 > bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc > aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 > 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb > 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba > 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 > c42019-08-22 09:53:29,106 INFO > [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid > length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 > 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 > bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc > aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 > 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb > 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba > 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 > c42019-08-22 09:53:29,106 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session] > (sshd-SshServer[6162dce4]-nio2-thread-2) Caught exception, now calling > handler2019-08-22 09:53:29,106 WARN > [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Exception caught: > org.apache.sshd.common.SshException: Invalid packet length: -16976455 at > org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:677) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:283) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:184) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32) > [sshd-core-0.13.0.jar:0.13.0] at > java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_172] > at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30) > [sshd-core-0.13.0.jar:0.13.0] at > sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [rt.jar:1.8.0_172] at > sun.nio.ch.Invoker$2.run(Invoker.java:218) [rt.jar:1.8.0_172] at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > [rt.jar:1.8.0_172] at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [rt.jar:1.8.0_172] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Reopened] (SSHD-938) SSH sessions being closed with invalid packet length
[ https://issues.apache.org/jira/browse/SSHD-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anudeep reopened SSHD-938: -- Please do not close we need some support as we cannot migrate to latest version that easy. We expect some clues so that we can go for some workaround > SSH sessions being closed with invalid packet length > > > Key: SSHD-938 > URL: https://issues.apache.org/jira/browse/SSHD-938 > Project: MINA SSHD > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Anudeep >Assignee: Goldstein Lyor >Priority: Blocker > Attachments: server.7z > > > SSH connections are being closed with invalid packet length and TCP > connection goes into CLOSE_WAIT state and once CLOSE_WAIT state count crosses > 4 no more SSH connection is being allowed. > with SSHD-core-0.13.0 > > Below is the error > 2019-08-22 09:53:29,106 INFO [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid > length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 > 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 > bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc > aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 > 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb > 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba > 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 > c42019-08-22 09:53:29,106 INFO > [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid > length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 > 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 > bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc > aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 > 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb > 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba > 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 > c42019-08-22 09:53:29,106 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session] > (sshd-SshServer[6162dce4]-nio2-thread-2) Caught exception, now calling > handler2019-08-22 09:53:29,106 WARN > [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Exception caught: > org.apache.sshd.common.SshException: Invalid packet length: -16976455 at > org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:677) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:283) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:184) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32) > [sshd-core-0.13.0.jar:0.13.0] at > java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_172] > at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30) > [sshd-core-0.13.0.jar:0.13.0] at > sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [rt.jar:1.8.0_172] at > sun.nio.ch.Invoker$2.run(Invoker.java:218) [rt.jar:1.8.0_172] at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > [rt.jar:1.8.0_172] at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [rt.jar:1.8.0_172] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Commented] (SSHD-938) SSH sessions being closed with invalid packet length
[ https://issues.apache.org/jira/browse/SSHD-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914463#comment-16914463 ] Anudeep commented on SSHD-938: -- Hi @Goldstein Lyor I understand but can we have some clue what happens if there are few connections in closewait. Why does it blocks completely. And what is invalid here the client message ? > SSH sessions being closed with invalid packet length > > > Key: SSHD-938 > URL: https://issues.apache.org/jira/browse/SSHD-938 > Project: MINA SSHD > Issue Type: Bug >Affects Versions: 0.13.0 >Reporter: Anudeep >Assignee: Goldstein Lyor >Priority: Blocker > Attachments: server.7z > > > SSH connections are being closed with invalid packet length and TCP > connection goes into CLOSE_WAIT state and once CLOSE_WAIT state count crosses > 4 no more SSH connection is being allowed. > with SSHD-core-0.13.0 > > Below is the error > 2019-08-22 09:53:29,106 INFO [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid > length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 > 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 > bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc > aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 > 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb > 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba > 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 > c42019-08-22 09:53:29,106 INFO > [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid > length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 > 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 > bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc > aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 > 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb > 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba > 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 > c42019-08-22 09:53:29,106 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session] > (sshd-SshServer[6162dce4]-nio2-thread-2) Caught exception, now calling > handler2019-08-22 09:53:29,106 WARN > [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Exception caught: > org.apache.sshd.common.SshException: Invalid packet length: -16976455 at > org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:677) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:283) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:184) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170) > [sshd-core-0.13.0.jar:0.13.0] at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32) > [sshd-core-0.13.0.jar:0.13.0] at > java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_172] > at > org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30) > [sshd-core-0.13.0.jar:0.13.0] at > sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [rt.jar:1.8.0_172] at > sun.nio.ch.Invoker$2.run(Invoker.java:218) [rt.jar:1.8.0_172] at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > [rt.jar:1.8.0_172] at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [rt.jar:1.8.0_172] -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Updated] (SSHD-938) SSH sessions being closed with invalid packet length
[ https://issues.apache.org/jira/browse/SSHD-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anudeep updated SSHD-938: - Description: SSH connections are being closed with invalid packet length and TCP connection goes into CLOSE_WAIT state and once CLOSE_WAIT state count crosses 4 no more SSH connection is being allowed. with SSHD-core-0.13.0 Below is the error 2019-08-22 09:53:29,106 INFO [org.apache.sshd.server.session.ServerSession] (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 c42019-08-22 09:53:29,106 INFO [org.apache.sshd.server.session.ServerSession] (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 c42019-08-22 09:53:29,106 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session] (sshd-SshServer[6162dce4]-nio2-thread-2) Caught exception, now calling handler2019-08-22 09:53:29,106 WARN [org.apache.sshd.server.session.ServerSession] (sshd-SshServer[6162dce4]-nio2-thread-2) Exception caught: org.apache.sshd.common.SshException: Invalid packet length: -16976455 at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:677) [sshd-core-0.13.0.jar:0.13.0] at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:283) [sshd-core-0.13.0.jar:0.13.0] at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54) [sshd-core-0.13.0.jar:0.13.0] at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:184) [sshd-core-0.13.0.jar:0.13.0] at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170) [sshd-core-0.13.0.jar:0.13.0] at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32) [sshd-core-0.13.0.jar:0.13.0] at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_172] at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30) [sshd-core-0.13.0.jar:0.13.0] at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [rt.jar:1.8.0_172] at sun.nio.ch.Invoker$2.run(Invoker.java:218) [rt.jar:1.8.0_172] at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) [rt.jar:1.8.0_172] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_172] was: SSH connections are being closed with invalid packet length and TCP connection goes into CLOSE_WAIT state and once CLOSE_WAIT state count crosses 4 no more SSH connection is being allowed. with SSHD-core-1.13.0 Below is the error 2019-08-22 09:53:29,106 INFO [org.apache.sshd.server.session.ServerSession] (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 c42019-08-22 09:53:29,106 INFO [org.apache.sshd.server.session.ServerSession] (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb 34
[jira] [Updated] (SSHD-938) SSH sessions being closed with invalid packet length
[ https://issues.apache.org/jira/browse/SSHD-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anudeep updated SSHD-938: - Description: SSH connections are being closed with invalid packet length and TCP connection goes into CLOSE_WAIT state and once CLOSE_WAIT state count crosses 4 no more SSH connection is being allowed. with SSHD-core-1.13.0 Below is the error 2019-08-22 09:53:29,106 INFO [org.apache.sshd.server.session.ServerSession] (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 c42019-08-22 09:53:29,106 INFO [org.apache.sshd.server.session.ServerSession] (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba 80 34 55 3f 38 49 0c a9 80 8c a3 fe 6f b6 b6 5d 45 41 e9 e0 e6 d5 15 d6 44 f8 c42019-08-22 09:53:29,106 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session] (sshd-SshServer[6162dce4]-nio2-thread-2) Caught exception, now calling handler2019-08-22 09:53:29,106 WARN [org.apache.sshd.server.session.ServerSession] (sshd-SshServer[6162dce4]-nio2-thread-2) Exception caught: org.apache.sshd.common.SshException: Invalid packet length: -16976455 at org.apache.sshd.common.session.AbstractSession.decode(AbstractSession.java:677) [sshd-core-0.13.0.jar:0.13.0] at org.apache.sshd.common.session.AbstractSession.messageReceived(AbstractSession.java:283) [sshd-core-0.13.0.jar:0.13.0] at org.apache.sshd.common.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:54) [sshd-core-0.13.0.jar:0.13.0] at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:184) [sshd-core-0.13.0.jar:0.13.0] at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:170) [sshd-core-0.13.0.jar:0.13.0] at org.apache.sshd.common.io.nio2.Nio2CompletionHandler$1.run(Nio2CompletionHandler.java:32) [sshd-core-0.13.0.jar:0.13.0] at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_172] at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:30) [sshd-core-0.13.0.jar:0.13.0] at sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126) [rt.jar:1.8.0_172] at sun.nio.ch.Invoker$2.run(Invoker.java:218) [rt.jar:1.8.0_172] at sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) [rt.jar:1.8.0_172] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_172] was: SSH connections are being closed with invalid packet length and TCP connection goes into CLOSE_WAIT state and once CLOSE_WAIT state count crosses 4 no more SSH connection is being allowed. with SSHD-core-1.13.0 > SSH sessions being closed with invalid packet length > > > Key: SSHD-938 > URL: https://issues.apache.org/jira/browse/SSHD-938 > Project: MINA SSHD > Issue Type: Bug >Reporter: Anudeep >Priority: Blocker > Attachments: server.7z > > > SSH connections are being closed with invalid packet length and TCP > connection goes into CLOSE_WAIT state and once CLOSE_WAIT state count crosses > 4 no more SSH connection is being allowed. > with SSHD-core-1.13.0 > > Below is the error > 2019-08-22 09:53:29,106 INFO [org.apache.sshd.server.session.ServerSession] > (sshd-SshServer[6162dce4]-nio2-thread-2) Error decoding packet (invalid > length) a5 5f 74 5f a6 6f ea 41 02 c0 0a 54 83 22 e4 77 53 7d 0d b9 25 de 36 > 1d 07 cb cd 43 66 95 51 b2 72 54 d5 4e 2d 0a d5 17 f4 5e 81 4d 6a b8 ab f7 00 > bf 4d 40 f5 56 b7 61 9b 8a be fe 54 d4 89 a2 c5 07 3c de 25 95 44 18 4f 90 dc > aa 5b de 40 85 71 12 cc f6 c0 34 8d 0e 75 95 a5 5b 8d f6 b9 f6 94 04 d8 b5 25 > 5c d5 58 06 d1 82 68 13 2f 4c a0 ee 78 6b 20 97 52 2d 93 4c 88 bc 17 53 ca eb > 34 23 c1 c6 1e 5a f9 31 e7 52 f9 ef 88 03 de e8 af ba 11 5e 97 50 1e c2 26 ba > 80 34 55 3f
[jira] [Updated] (SSHD-938) SSH sessions being closed with invalid packet length
[ https://issues.apache.org/jira/browse/SSHD-938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anudeep updated SSHD-938: - Attachment: server.7z > SSH sessions being closed with invalid packet length > > > Key: SSHD-938 > URL: https://issues.apache.org/jira/browse/SSHD-938 > Project: MINA SSHD > Issue Type: Bug >Reporter: Anudeep >Priority: Blocker > Attachments: server.7z > > > SSH connections are being closed with invalid packet length and TCP > connection goes into CLOSE_WAIT state and once CLOSE_WAIT state count crosses > 4 no more SSH connection is being allowed. > with SSHD-core-1.13.0 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Created] (SSHD-938) SSH sessions being closed with invalid packet length
Anudeep created SSHD-938: Summary: SSH sessions being closed with invalid packet length Key: SSHD-938 URL: https://issues.apache.org/jira/browse/SSHD-938 Project: MINA SSHD Issue Type: Bug Reporter: Anudeep SSH connections are being closed with invalid packet length and TCP connection goes into CLOSE_WAIT state and once CLOSE_WAIT state count crosses 4 no more SSH connection is being allowed. with SSHD-core-1.13.0 -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@mina.apache.org For additional commands, e-mail: dev-h...@mina.apache.org
[jira] [Commented] (SSHD-891) SSh connections gets closed for every one hour with 0.13.0 version
[
https://issues.apache.org/jira/browse/SSHD-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770968#comment-16770968
]
Anudeep commented on SSHD-891:
--
Hi [~lgoldstein],
We see issue to be with ssh client which uses lib-ssh(0.4.5) .
In apache sshd-core(0.13.0) there is a logic to reexchange the keys for every
one hour ,server is able to send the key to client and client is not able to
send it back to server and session gets timeout and closed.
Please find below logs
{code:java}
02019-02-16 11:12:52,438 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session]
(EJB default - 11) Writing 68 bytes
2019-02-16 11:12:52,439 INFO [org.apache.sshd.server.session.ServerSession]
(EJB default - 11) Initiating key re-exchange
2019-02-16 11:12:52,439 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session]
(sshd-SshServer[78670623]-nio2-thread-3) Finished writing
2019-02-16 11:12:52,439 DEBUG [org.apache.sshd.server.session.ServerSession]
(EJB default - 11) Send SSH_MSG_KEXINIT
2019-02-16 11:12:52,439 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session] (EJB
default - 11) Writing 644 bytes
2019-02-16 11:12:52,444 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session]
(sshd-SshServer[78670623]-nio2-thread-1) Finished writing
2019-02-16 11:13:02,438 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 15) Consume server remote window by 4 down to 191444
2019-02-16 11:13:02,439 DEBUG [org.apache.sshd.server.channel.ChannelSession]
(EJB default - 15) Send SSH_MSG_CHANNEL_DATA on channel 0
2019-02-16 11:13:02,440 INFO [org.apache.sshd.server.session.ServerSession]
(EJB default - 15) Start flagging packets as pending until key exchange is done
2019-02-16 11:13:12,439 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 19) Consume server remote window by 4 down to 191440
2019-02-16 11:13:12,439 DEBUG [org.apache.sshd.server.channel.ChannelSession]
(EJB default - 19) Send SSH_MSG_CHANNEL_DATA on channel 0
2019-02-16 11:13:22,439 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 23) Consume server remote window by 4 down to 191436
2019-02-16 11:13:22,439 DEBUG [org.apache.sshd.server.channel.ChannelSession]
(EJB default - 23) Send SSH_MSG_CHANNEL_DATA on channel 0
2019-02-16 11:13:32,438 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 27) Consume server remote window by 4 down to 191432
2019-02-16 11:13:32,438 DEBUG [org.apache.sshd.server.channel.ChannelSession]
(EJB default - 27) Send SSH_MSG_CHANNEL_DATA on channel 0
2019-02-16 11:13:42,439 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 31) Consume server remote window by 4 down to 1:
{code}
{code}
public class ServerSession extends AbstractSession {
protected static final long MAX_PACKETS = (1l << 31);
private long authTimeoutTimestamp;
private long idleTimeoutTimestamp = 0L;
private int authTimeoutMs = 2 * 60 * 1000;// 2 minutes in milliseconds
private int idleTimeoutMs = 10 * 60 * 1000; // 10 minutes in milliseconds
private long maxBytes = 1024 * 1024 * 1024; // 1 GB
private long maxKeyInterval = 60*60*1000 ; // 1 hour
public ServerSession(ServerFactoryManager server, IoSession ioSession)
throws Exception {
super(true, server, ioSession);
authTimeoutMs = getIntProperty(ServerFactoryManager.AUTH_TIMEOUT,
authTimeoutMs);
authTimeoutTimestamp = System.currentTimeMillis() + authTimeoutMs;
idleTimeoutMs = getIntProperty(ServerFactoryManager.IDLE_TIMEOUT,
idleTimeoutMs);
maxBytes = Math.max(32,
getLongProperty(ServerFactoryManager.REKEY_BYTES_LIMIT, maxBytes));
maxKeyInterval = getLongProperty(ServerFactoryManager.REKEY_TIME_LIMIT,
maxKeyInterval);
log.info("Server session created from {}",
ioSession.getRemoteAddress());
sendServerIdentification();
kexState.set(KEX_STATE_INIT);
sendKexInit();
}
{code}
Can you please let us know if there is a way to change the configuration of
maxKeyInterval ,i mean can we do it via any configuration file.
Regards,
Anudeep
> SSh connections gets closed for every one hour with 0.13.0 version
> --
>
> Key: SSHD-891
> URL: https://issues.apache.org/jira/browse/SSHD-891
> Project: MINA SSHD
> Issue Type: Bug
>Reporter: Anudeep
>Assignee: Goldstein Lyor
>Priority: Blocker
>
> SSH connections are getting closed for every one hours after 10 mins of idle
> time even though there are incoming pacets data is not being wriiten to
> client for every one hour and is being idle(10 mins which is default timeout
> of ssh session).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SSHD-891) SSh connections gets closed for every one hour with 0.13.0 version
[
https://issues.apache.org/jira/browse/SSHD-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16770055#comment-16770055
]
Anudeep commented on SSHD-891:
--
Hi [~lgoldstein],
We have uplifted sshd-core version from 0.13.0 to 2.1.0.
But with 2.1.0 we see many impacts
one such is client session is getting closed abruptly after few minutes
Please find below logs
{code:java}
[id=0, recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58516])
by 4 down to 191244
2019-02-16 06:34:10,697 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session] (EJB
default - 44) writePacket(Nio2Session[local=/10.247.
246.100:8345, remote=/141.137.237.205:58516]) Writing 68 bytes
2019-02-16 06:34:10,697 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session]
(sshd-SshServer[5c704bd8]-nio2-thread-3)
handleCompletedWriteCycle(Nio2Session[local=/10.247.246.100:8345,
remote=/141.137.237.205:58516]) finished writing len=68
2019-02-16 06:34:17,050 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session]
(sshd-SshServer[5c704bd8]-nio2-thread-1)
exceptionCaught(Nio2Session[local=/10.247.246.100:8345,
remote=/141.137.237.205:58516]) caught InterruptedByTimeoutException[null] -
calling handler
2019-02-16 06:34:17,050 WARN [org.apache.sshd.server.session.ServerSessionImpl]
(sshd-SshServer[5c704bd8]-nio2-thread-1)
exceptionCaught(ServerSessionImpl[administrator@/141.137.237.205:58516])[state=Opened]
InterruptedByTimeoutException: null
2019-02-16 06:34:17,050 DEBUG
[org.apache.sshd.server.session.ServerSessionImpl]
(sshd-SshServer[5c704bd8]-nio2-thread-1)
exceptionCaught(ServerSessionImpl[administrator@/141.137.237.205:58516])[state=Opened]
details: java.nio.channels.InterruptedByTimeoutException
at
sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:456)
[rt.jar:1.8.0_172]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[rt.jar:1.8.0_172]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_172]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[rt.jar:1.8.0_172]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[rt.jar:1.8.0_172]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[rt.jar:1.8.0_172]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[rt.jar:1.8.0_172]
at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_172]
2019-02-16 06:34:17,053 DEBUG
[org.apache.sshd.common.session.helpers.SessionTimeoutListener]
(sshd-SshServer[5c704bd8]-nio2-thread-1)
sessionException(ServerSessionImpl[administrator@/141.137.237.205:58516])
InterruptedByTimeoutException: null
2019-02-16 06:34:17,053 DEBUG
[org.apache.sshd.common.session.helpers.SessionTimeoutListener]
(sshd-SshServer[5c704bd8]-nio2-thread-1)
sessionClosed(ServerSessionImpl[administrator@/141.137.237.205:58516])
un-tracked
2019-02-16 06:34:17,053 DEBUG
[org.apache.sshd.server.session.ServerSessionImpl]
(sshd-SshServer[5c704bd8]-nio2-thread-1)
close(ServerSessionImpl[administrator@/141.137.237.205:58516]) Closing
immediately
2019-02-16 06:34:17,053 DEBUG
[org.apache.sshd.server.session.ServerConnectionService]
(sshd-SshServer[5c704bd8]-nio2-thread-1)
close(ServerConnectionService[ServerSessionImpl[administrator@/141.137.237.205:58516]])
Closing immediately
2019-02-16 06:34:17,054 DEBUG [org.apache.sshd.server.channel.ChannelSession]
(sshd-SshServer[5c704bd8]-nio2-thread-1) close(ChannelSession[id=0,
recipient=43]-ServerSessionImpl[administrator@/141.137.237.205:58516]) Closing
immediately
2019-02-16 06:34:17,054 DEBUG [org.apache.sshd.server.channel.ChannelSession]
(sshd-SshServer[5c704bd8]-nio2-thread-1) close(Channel:
{code}
And with respect to setting the state of sshd we are not able to access the
state variable from ssh-server package as state varaible is made protected in
ssh-common package
{code}
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.sshd.common.util.closeable;
import java.util.concurrent.atomic.AtomicReference;
import
[jira] [Commented] (SSHD-891) SSh connections gets closed for every one hour with 0.13.0 version
[
https://issues.apache.org/jira/browse/SSHD-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764711#comment-16764711
]
Anudeep commented on SSHD-891:
--
Hi [~lgoldstein],
Regarding the message sent that would be a message from server(there is a timer
at server side which sends "HB" message from server for every 30 seconds if
there are no incoming packets from server to client.
Please find below logs
Lsof output :svc-2-sg
Wed Feb 6 05:30:17 CET 2019
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 4409 root 502u IPv6 1015404 0t0 TCP svc-2-sg:8345->192.168.42.24:33589
(ESTABLISHED)
java 4409 root 525u IPv6 19463 0t0 TCP *:8345 (LISTEN)
502 is the FD here for the VM
VM Server.log :
{code:java}
Please find Apache being idle in the 10 mins as mentioned (ie 5:29 to 5:39) in
the server.log of VM.
2019-02-06 05:28:27,759 INFO [com.ericsson.oss.itpf.COMMAND_LOGGER]
(pool-7-thread-1888) [omega, Acknowledge, STARTED, BNSI NBI Client, BNSI NBI
Server, NotificationIdentifier = 140154550]
2019-02-06 05:28:57,723 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 8) Consume server remote window by 4 down to 79982
2019-02-06 05:28:57,724 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session] (EJB
default - 8) Writing 68 bytes
2019-02-06 05:28:57,726 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session]
(sshd-SshServer[4166a223]-nio2-thread-3) Finished writing
2019-02-06 05:29:27,721 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 19) Consume server remote window by 4 down to 79978
2019-02-06 05:29:27,722 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session] (EJB
default - 19) Writing 68 bytes
2019-02-06 05:29:27,723 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session]
(sshd-SshServer[4166a223]-nio2-thread-1) Finished writing
2019-02-06 05:29:27,724 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session] (EJB
default - 19) Writing 532 bytes
2019-02-06 05:29:27,725 DEBUG [org.apache.sshd.common.io.nio2.Nio2Session]
(sshd-SshServer[4166a223]-nio2-thread-3) Finished writing
2019-02-06 05:29:57,722 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 32) Consume server remote window by 4 down to 79974
2019-02-06 05:30:27,721 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 42) Consume server remote window by 4 down to 79970
2019-02-06 05:30:57,722 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 53) Consume server remote window by 4 down to 79966
2019-02-06 05:31:27,722 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 65) Consume server remote window by 4 down to 79962
2019-02-06 05:31:57,721 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 75) Consume server remote window by 4 down to 79958
2019-02-06 05:32:10,641 TRACE [org.apache.sshd.common.channel.Window]
(Thread-1259 (HornetQ-client-global-threads-22246058)) Consume server remote
window by 1532 down to 78426
2019-02-06 05:32:40,645 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 91) Consume server remote window by 4 down to 78422
2019-02-06 05:33:10,652 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 101) Consume server remote window by 4 down to 78418
2019-02-06 05:33:30,662 TRACE [org.apache.sshd.common.channel.Window]
(Thread-1260 (HornetQ-client-global-threads-22246058)) Consume server remote
window by 1532 down to 76886
2019-02-06 05:34:00,665 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 121) Consume server remote window by 4 down to 76882
2019-02-06 05:34:30,668 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 3) Consume server remote window by 4 down to 76878
2019-02-06 05:35:00,665 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 15) Consume server remote window by 4 down to 76874
2019-02-06 05:35:30,666 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 27) Consume server remote window by 4 down to 76870
2019-02-06 05:36:00,665 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 37) Consume server remote window by 4 down to 76866
2019-02-06 05:36:30,665 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 51) Consume server remote window by 4 down to 76862
2019-02-06 05:37:00,665 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 59) Consume server remote window by 4 down to 76858
2019-02-06 05:37:30,666 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 70) Consume server remote window by 4 down to 76854
2019-02-06 05:38:00,665 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 84) Consume server remote window by 4 down to 76850
2019-02-06 05:38:30,666 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 90) Consume server remote window by 4 down to 76846
2019-02-06 05:39:00,666 TRACE [org.apache.sshd.common.channel.Window] (EJB
default - 103) Consume server remote window by 4 down to 76842
2019-02-06 05:39:28,487 DEBUG
[jira] [Commented] (SSHD-891) SSh connections gets closed for every one hour with 0.13.0 version
[ https://issues.apache.org/jira/browse/SSHD-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764396#comment-16764396 ] Anudeep commented on SSHD-891: -- Hi [~lgoldstein], I would like to provide a brief note on the issue. There is a client with ssh v2 version (omega) which is connecting to a server .In order to keep the session alive there is a mechanism from sever to send a message to client for every 30 seconds so that if there is not packet which needs to be send from server to client then a message would be sent to client for every 30 seconds. The above scenario is able to withstand for one hour and immediately after one hour even though server sends a message to client ,it is not able to write and session is becoming idle for 10 mins . Since ssh default idle timeout is 10 mins ssh server is killing the session. So we need to understand why exactly after every one hour the message from server is not written. It is not that easy for us now to migrate to latest version ,if we can find the root cause we will try to migrate. Side effect of this issue is for every two days we are not able to open a new ssh connection. connections seems to be in CLOSE_WAIT state and no new connection is established. Regards, Anudeep > SSh connections gets closed for every one hour with 0.13.0 version > -- > > Key: SSHD-891 > URL: https://issues.apache.org/jira/browse/SSHD-891 > Project: MINA SSHD > Issue Type: Bug >Reporter: Anudeep >Assignee: Goldstein Lyor >Priority: Blocker > > SSH connections are getting closed for every one hours after 10 mins of idle > time even though there are incoming pacets data is not being wriiten to > client for every one hour and is being idle(10 mins which is default timeout > of ssh session). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (SSHD-891) SSh connections gets closed for every one hour with 0.13.0 version
[ https://issues.apache.org/jira/browse/SSHD-891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764397#comment-16764397 ] Anudeep edited comment on SSHD-891 at 2/10/19 12:52 PM: Can you please let us know the root cause ,so that we can decide whether to migrate or not. was (Author: anudeep): Please can you please let us know the root cause ,so that we can decide whether to migrate or not. > SSh connections gets closed for every one hour with 0.13.0 version > -- > > Key: SSHD-891 > URL: https://issues.apache.org/jira/browse/SSHD-891 > Project: MINA SSHD > Issue Type: Bug >Reporter: Anudeep >Assignee: Goldstein Lyor >Priority: Blocker > > SSH connections are getting closed for every one hours after 10 mins of idle > time even though there are incoming pacets data is not being wriiten to > client for every one hour and is being idle(10 mins which is default timeout > of ssh session). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (SSHD-891) SSh connections gets closed for every one hour with 0.13.0 version
[ https://issues.apache.org/jira/browse/SSHD-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anudeep reopened SSHD-891: -- Please can you please let us know the root cause ,so that we can decide whether to migrate or not. > SSh connections gets closed for every one hour with 0.13.0 version > -- > > Key: SSHD-891 > URL: https://issues.apache.org/jira/browse/SSHD-891 > Project: MINA SSHD > Issue Type: Bug >Reporter: Anudeep >Assignee: Goldstein Lyor >Priority: Blocker > > SSH connections are getting closed for every one hours after 10 mins of idle > time even though there are incoming pacets data is not being wriiten to > client for every one hour and is being idle(10 mins which is default timeout > of ssh session). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SSHD-891) SSh connections gets closed for every one hour with 0.13.0 version
[ https://issues.apache.org/jira/browse/SSHD-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anudeep updated SSHD-891: - Priority: Blocker (was: Major) > SSh connections gets closed for every one hour with 0.13.0 version > -- > > Key: SSHD-891 > URL: https://issues.apache.org/jira/browse/SSHD-891 > Project: MINA SSHD > Issue Type: Bug >Reporter: Anudeep >Priority: Blocker > > SSH connections are getting closed for every one hours after 10 mins of idle > time even though there are incoming pacets data is not being wriiten to > client for every one hour and is being idle(10 mins which is default timeout > of ssh session). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SSHD-891) SSh connections gets closed for every one hour with 0.13.0 version
Anudeep created SSHD-891: Summary: SSh connections gets closed for every one hour with 0.13.0 version Key: SSHD-891 URL: https://issues.apache.org/jira/browse/SSHD-891 Project: MINA SSHD Issue Type: Bug Reporter: Anudeep SSH connections are getting closed for every one hours after 10 mins of idle time even though there are incoming pacets data is not being wriiten to client for every one hour and is being idle(10 mins which is default timeout of ssh session). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
