[
https://issues.apache.org/jira/browse/DIRMINA-1021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15152639#comment-15152639
]
Doug Kelly commented on DIRMINA-1021:
-------------------------------------
I've built the latest version of mina-sshd against mina-core 2.0.13, and I have
managed to reproduce this issue at least once:
{code}
$ ssh localhost -p 29418 gerrit show-connections
Session Start Idle User Remote Host
--------------------------------------------------------------
b6219fea 11:08:25 00:00:09 dougk ?
f6d95718 11:08:34 00:00:00 dougk localhost
--
SSHD Backend: mina
{code}
The "?" is indicative of a session that's been closed, has no more remote host
information, but hasn't been removed from the session list.
In addition to seeing "already closed" exceptions in my log:
{code}
[2016-02-18 11:08:31,221] [SSH git-upload-pack '/gerrit' (dougk)] ERROR
com.google.gerrit.sshd.BaseCommand : Internal server error (user dougk acc
ount 1000000) during git-upload-pack '/gerrit'
org.apache.sshd.common.channel.WindowClosedException: Already closed:
Window[server/remote](ChannelSession[id=0, recipient=0]-GerritServerSession[
dougk@/127.0.0.1:49347])
at
org.apache.sshd.common.channel.Window.waitForCondition(Window.java:311)
at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:264)
at
org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:165)
at
org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:119)
at
org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:171)
at
org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:151)
at
org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:126)
at
org.eclipse.jgit.internal.storage.file.ByteArrayWindow.write(ByteArrayWindow.java:89)
at
org.eclipse.jgit.internal.storage.file.WindowCursor.copyPackAsIs(WindowCursor.java:248)
at
org.eclipse.jgit.internal.storage.file.PackFile.copyPackAsIs(PackFile.java:355)
at
org.eclipse.jgit.internal.storage.file.LocalCachedPack.copyAsIs(LocalCachedPack.java:85)
at
org.eclipse.jgit.internal.storage.file.WindowCursor.copyPackAsIs(WindowCursor.java:236)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1010)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1538)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1391)
at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:772)
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:666)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:85)
at
com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
at
com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
at
com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)
at
com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:437)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:377)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
I also had one instance of a "timeout exceeded" error:
{code}
[2016-02-18 11:09:55,729] [SSH git-upload-pack '/gerrit' (dougk)] ERROR
com.google.gerrit.sshd.BaseCommand : Internal server error (user dougk account
1000000) during git-upload-pack '/gerrit'
java.net.SocketTimeoutException:
waitForCondition(Window[server/remote](ChannelSession[id=0,
recipient=0]-GerritServerSession[dougk@null])) timeout exceeded: 30000
at
org.apache.sshd.common.channel.Window.waitForCondition(Window.java:314)
at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:264)
at
org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:165)
at
org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:119)
at
org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:171)
at
org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:151)
at
org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:126)
at
org.eclipse.jgit.internal.storage.file.ByteArrayWindow.write(ByteArrayWindow.java:89)
at
org.eclipse.jgit.internal.storage.file.WindowCursor.copyPackAsIs(WindowCursor.java:248)
at
org.eclipse.jgit.internal.storage.file.PackFile.copyPackAsIs(PackFile.java:355)
at
org.eclipse.jgit.internal.storage.file.LocalCachedPack.copyAsIs(LocalCachedPack.java:85)
at
org.eclipse.jgit.internal.storage.file.WindowCursor.copyPackAsIs(WindowCursor.java:236)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1010)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1538)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1391)
at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:772)
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:666)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:85)
at
com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:101)
at
com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
at
com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:70)
at
com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:437)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:377)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
The hostname ("dougk@null") looks oddly suspicious there...
But, the revert you made seems to have at least fixed the
IllegalThreadStateException.
> MINA-CORE does not remove sessions if exceptions occur while closing
> --------------------------------------------------------------------
>
> Key: DIRMINA-1021
> URL: https://issues.apache.org/jira/browse/DIRMINA-1021
> Project: MINA
> Issue Type: Bug
> Affects Versions: 2.0.8
> Environment: mina-ssh 0.14.0
> mina-core 2.0.8
> Multiple OSes / Java configurations:
> * Mac OS X El Capitan on Java 8 (1.8.0_60)
> * CentOS 6.4 on Java 8 (1.8.0_60)
> * CentOS 6.5 on Java 8 (1.8.0_20-b26).
> Reporter: Doug Kelly
> Attachments: attempt-removing-sessions-closing.patch
>
>
> MINA SSHD isn't removing sessions when using the MINA/NIO backend if an
> exception as received as the session is closing (such as a connection reset
> is received with data still in the write buffer). When this case happens, it
> seems that {{NioProcessor.getState}} returns the state as {{CLOSING}} (I'm
> assuming since the underlying channel is now closed), which means that the
> {{AbstractPollingIoProcessor.removeSessions()}} won't ever prune the session,
> since a {{CLOSING}} state is simply ignored. The result is a resource leak
> over time, since these sessions are never pruned (it's a slow leak, since
> entering this condition is racy – on my workstation, I can produce it through
> randomly interrupting connections anywhere from 1/6 to 1/10th of the time).
> (This may either be major or critical; reprioritize as necessary.)
> I specifically see this error with Gerrit 2.10.4 and Gerrit 2.11.5 (using
> mina-sshd 0.14.0 / mina-core 2.0.8), and it looks like the code path is
> unchanged in mina-sshd 1.0.0 / mina-core 2.0.9. I was unsure if this is
> specifically a bug in mina-core or, if it's something unique to mina-sshd. My
> local development system runs Mac OS X El Capitan on Java 8 (1.8.0_60), but
> I've also seen this on Linux (CentOS 6.4, again Java 1.8.0_60 and CentOS 6.5
> on Java 1.8.0_20-b26).
> The fix may be as simple as attempting to remove the session if {{OPENED}} or
> {{CLOSING}}, but I'm unsure what side-effects this may have with other
> backends. I'll be happy to test it locally, but I'm fairly ignorant when it
> comes to MINA's code.
> The attached patch (to mina-core) seems to resolve the issue by following the
> reproduction case I have on the [Gerrit issue
> tracker|https://code.google.com/p/gerrit/issues/detail?id=3685].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
