[
https://issues.apache.org/jira/browse/SSHD-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110526#comment-17110526
]
Francois Ferrand commented on SSHD-966:
---------------------------------------
There is definitely an issue. When downloading a single (large), after having
reduced the rekey limits (100M / 1min), I get the following:
{code:java}
$ git clone ssh://gerrit/aosp/external/linux-kselftest
Cloning into 'linux-kselftest'...
remote: Counting objects: 6370616, done
remote: Finding sources: 100% (6370616/6370616)
fatal: internal server error086/6370616), 199.75 MiB | 4.24 MiB/s
fatal: the remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
{code}
Actually, it fails after having retrieved ~200MiB, after getting stuck there
for some time.
Note that another time it failed at ~100MiB, so this is not just about the
second rekey:
{code:java}
$ git clone ssh://gerrit/aosp/external/linux-kselftest
Cloning into 'linux-kselftest'...
remote: Counting objects: 6370616, done
remote: Finding sources: 100% (6370616/6370616)
fatal: internal server error479/6370616), 99.85 MiB | 3.06 MiB/s
fatal: the remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
{code}
Here is the logs (for both attempts):
{code:java}
[2020-05-18 18:38:36,913] [SSH git-upload-pack aosp/external/linux-kselftest
(francois)] ERROR org.apache.sshd.server.channel.ChannelSession :
lockedFlush(ChannelOutputStream[ChannelSession[id=0,
recipient=0]-ServerSessionImpl[francois@/172.16.0.10:51178]]
SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of
len=20480: waitForCondition(Window[server/remote](ChannelSession[id=0,
recipient=0]-ServerSessionImpl[francois@/172.16.0.10:51178])) timeout exceeded:
30000
[2020-05-18 18:38:36,913] [SSH git-upload-pack aosp/external/linux-kselftest
(francois)] ERROR org.apache.sshd.server.channel.ChannelSession :
write(ChannelOutputStream[ChannelSession[id=0,
recipient=0]-ServerSessionImpl[francois@/172.16.0.10:51178]]
SSH_MSG_CHANNEL_DATA) len=65520 failed to write
java.net.SocketTimeoutException:
waitForCondition(Window[server/remote](ChannelSession[id=0,
recipient=0]-ServerSessionImpl[francois@/172.16.0.10:51178])) timeout exceeded:
30000
at
org.apache.sshd.common.channel.Window.waitForCondition(Window.java:307)
at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:255)
at
org.apache.sshd.common.channel.ChannelOutputStream.lockedFlush(ChannelOutputStream.java:275)
at
org.apache.sshd.common.channel.ChannelOutputStream.lockedWrite(ChannelOutputStream.java:175)
at
org.apache.sshd.common.channel.ChannelOutputStream.lambda$write$1(ChannelOutputStream.java:138)
at
org.apache.sshd.common.session.helpers.AbstractSession.executeUnderPendingPacketsLock(AbstractSession.java:691)
at
org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:136)
at
org.eclipse.jgit.transport.UploadPack$ResponseBufferedOutputStream.write(UploadPack.java:2387)
at
org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:141)
at
org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:120)
at
org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:99)
at
org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:564)
at
org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:399)
at
org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:188)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1703)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1680)
at
org.eclipse.jgit.internal.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:138)
at
org.eclipse.jgit.internal.storage.file.WindowCursor.writeObjects(WindowCursor.java:196)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1668)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1653)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1214)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2328)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2164)
at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:1078)
at
org.eclipse.jgit.transport.UploadPack.uploadWithExceptionPropagation(UploadPack.java:835)
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:749)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:78)
at
com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:99)
at
com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
at
com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:64)
at
com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:469)
at
com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:83)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:646)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2020-05-18 18:38:36,929] [SSH git-upload-pack aosp/external/linux-kselftest
(francois)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error
(user francois account 1000002) during git-upload-pack
'aosp/external/linux-kselftest'
org.eclipse.jgit.transport.UploadPackInternalServerErrorException
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:772)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:78)
at
com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:99)
at
com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
at
com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:64)
at
com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:469)
at
com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:83)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:646)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException:
waitForCondition(Window[server/remote](ChannelSession[id=0,
recipient=0]-ServerSessionImpl[francois@/172.16.0.10:51178])) timeout exceeded:
30000
at
org.apache.sshd.common.channel.Window.waitForCondition(Window.java:307)
at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:255)
at
org.apache.sshd.common.channel.ChannelOutputStream.lockedFlush(ChannelOutputStream.java:275)
at
org.apache.sshd.common.channel.ChannelOutputStream.lockedWrite(ChannelOutputStream.java:175)
at
org.apache.sshd.common.channel.ChannelOutputStream.lambda$write$1(ChannelOutputStream.java:138)
at
org.apache.sshd.common.session.helpers.AbstractSession.executeUnderPendingPacketsLock(AbstractSession.java:691)
at
org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:136)
at
org.eclipse.jgit.transport.UploadPack$ResponseBufferedOutputStream.write(UploadPack.java:2387)
at
org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:141)
at
org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:120)
at
org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:99)
at
org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:564)
at
org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:399)
at
org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:188)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1703)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1680)
at
org.eclipse.jgit.internal.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:138)
at
org.eclipse.jgit.internal.storage.file.WindowCursor.writeObjects(WindowCursor.java:196)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1668)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1653)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1214)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2328)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2164)
at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:1078)
at
org.eclipse.jgit.transport.UploadPack.uploadWithExceptionPropagation(UploadPack.java:835)
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:749)
... 14 more
[2020-05-18 18:43:06,336] [SSH git-upload-pack aosp/external/linux-kselftest
(francois)] ERROR org.apache.sshd.server.channel.ChannelSession :
lockedFlush(ChannelOutputStream[ChannelSession[id=0,
recipient=0]-ServerSessionImpl[francois@/172.16.0.10:52292]]
SSH_MSG_CHANNEL_DATA) failed (SocketTimeoutException) to wait for space of
len=28672: waitForCondition(Window[server/remote](ChannelSession[id=0,
recipient=0]-ServerSessionImpl[francois@/172.16.0.10:52292])) timeout exceeded:
30000
[2020-05-18 18:43:06,337] [SSH git-upload-pack aosp/external/linux-kselftest
(francois)] ERROR org.apache.sshd.server.channel.ChannelSession :
write(ChannelOutputStream[ChannelSession[id=0,
recipient=0]-ServerSessionImpl[francois@/172.16.0.10:52292]]
SSH_MSG_CHANNEL_DATA) len=65520 failed to write
java.net.SocketTimeoutException:
waitForCondition(Window[server/remote](ChannelSession[id=0,
recipient=0]-ServerSessionImpl[francois@/172.16.0.10:52292])) timeout exceeded:
30000
at
org.apache.sshd.common.channel.Window.waitForCondition(Window.java:307)
at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:255)
at
org.apache.sshd.common.channel.ChannelOutputStream.lockedFlush(ChannelOutputStream.java:275)
at
org.apache.sshd.common.channel.ChannelOutputStream.lockedWrite(ChannelOutputStream.java:175)
at
org.apache.sshd.common.channel.ChannelOutputStream.lambda$write$1(ChannelOutputStream.java:138)
at
org.apache.sshd.common.session.helpers.AbstractSession.executeUnderPendingPacketsLock(AbstractSession.java:691)
at
org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:136)
at
org.eclipse.jgit.transport.UploadPack$ResponseBufferedOutputStream.write(UploadPack.java:2387)
at
org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:141)
at
org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:120)
at
org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:99)
at
org.eclipse.jgit.internal.storage.file.ByteArrayWindow.write(ByteArrayWindow.java:58)
at
org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:547)
at
org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:399)
at
org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:188)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1703)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1680)
at
org.eclipse.jgit.internal.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:138)
at
org.eclipse.jgit.internal.storage.file.WindowCursor.writeObjects(WindowCursor.java:196)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1668)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1653)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1214)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2328)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2164)
at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:1078)
at
org.eclipse.jgit.transport.UploadPack.uploadWithExceptionPropagation(UploadPack.java:835)
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:749)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:78)
at
com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:99)
at
com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
at
com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:64)
at
com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:469)
at
com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:83)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:646)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2020-05-18 18:43:06,348] [SSH git-upload-pack aosp/external/linux-kselftest
(francois)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error
(user francois account 1000002) during git-upload-pack
'aosp/external/linux-kselftest'
org.eclipse.jgit.transport.UploadPackInternalServerErrorException
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:772)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:78)
at
com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:99)
at
com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
at
com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:64)
at
com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:469)
at
com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:83)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:646)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException:
waitForCondition(Window[server/remote](ChannelSession[id=0,
recipient=0]-ServerSessionImpl[francois@/172.16.0.10:52292])) timeout exceeded:
30000
at
org.apache.sshd.common.channel.Window.waitForCondition(Window.java:307)
at org.apache.sshd.common.channel.Window.waitForSpace(Window.java:255)
at
org.apache.sshd.common.channel.ChannelOutputStream.lockedFlush(ChannelOutputStream.java:275)
at
org.apache.sshd.common.channel.ChannelOutputStream.lockedWrite(ChannelOutputStream.java:175)
at
org.apache.sshd.common.channel.ChannelOutputStream.lambda$write$1(ChannelOutputStream.java:138)
at
org.apache.sshd.common.session.helpers.AbstractSession.executeUnderPendingPacketsLock(AbstractSession.java:691)
at
org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:136)
at
org.eclipse.jgit.transport.UploadPack$ResponseBufferedOutputStream.write(UploadPack.java:2387)
at
org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:141)
at
org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:120)
at
org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:99)
at
org.eclipse.jgit.internal.storage.file.ByteArrayWindow.write(ByteArrayWindow.java:58)
at
org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:547)
at
org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:399)
at
org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:188)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1703)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1680)
at
org.eclipse.jgit.internal.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:138)
at
org.eclipse.jgit.internal.storage.file.WindowCursor.writeObjects(WindowCursor.java:196)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1668)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1653)
at
org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1214)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2328)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2164)
at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:1078)
at
org.eclipse.jgit.transport.UploadPack.uploadWithExceptionPropagation(UploadPack.java:835)
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:749)
... 14 more
{code}
> Deadlock on disconnection at the end of key-exchange
> ----------------------------------------------------
>
> Key: SSHD-966
> URL: https://issues.apache.org/jira/browse/SSHD-966
> Project: MINA SSHD
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: Francois Ferrand
> Assignee: Lyor Goldstein
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> We are using git-repo to download projects from Gerrit server, using SSH.
> Gerrit is in version 2.16.16. which uses SSHD 2.0.0 and Mina 2.0.17 with NIO2
> backend.
> One particularity of this setup is that git-repo creates a single control
> master channel, and then downloads *lots* of Git repositories (500
> repositories, some of them relatively large), with some degree of
> parallelism. This takes a long time, lots of data, and the multiplexed
> connections are handled by gerrit in multiple threads.
> In some cases, we experience a deadlock when an error happens at the end of
> the key exchange, while sending pending packets:
> {noformat}
> Warning, the following threads are deadlocked : SSH git-upload-pack /project1
> (myuser), sshd-SshServer[df5f657]-nio2-thread-3
> "SSH git-upload-pack /project1 (myuser)" prio=1 BLOCKED
>
> org.apache.sshd.common.session.helpers.AbstractSession.writePacket(AbstractSession.java:1107)
>
> org.apache.sshd.common.channel.AbstractChannel.writePacket(AbstractChannel.java:798)
>
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:227)
>
> org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:127)
>
> org.eclipse.jgit.transport.UploadPack$ResponseBufferedOutputStream.write(UploadPack.java:2183)
>
> org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:174)
>
> org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:153)
>
> org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:132)
>
> org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:614)
>
> org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:433)
>
> org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:221)
>
> org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1644)
>
> org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1621)
>
> org.eclipse.jgit.internal.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:171)
>
> org.eclipse.jgit.internal.storage.file.WindowCursor.writeObjects(WindowCursor.java:229)
>
> org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1609)
>
> org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1597)
>
> org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1154)
> org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2133)
> org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1947)
> org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:971)
> org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:776)
> com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:77)
>
> com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:98)
>
> com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:31)
>
> com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:63)
> com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:467)
>
> com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:83)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:646)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> "sshd-SshServer[df5f657]-nio2-thread-3" daemon prio=5 BLOCKED
>
> org.apache.sshd.common.channel.ChannelOutputStream.close(ChannelOutputStream.java:249)
> org.apache.sshd.common.util.io.IoUtils.closeQuietly(IoUtils.java:151)
>
> org.apache.sshd.server.channel.ChannelSession.doCloseImmediately(ChannelSession.java:205)
>
> org.apache.sshd.common.util.closeable.AbstractCloseable.close(AbstractCloseable.java:83)
>
> org.apache.sshd.common.channel.AbstractChannel.close(AbstractChannel.java:576)
>
> org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(ParallelCloseable.java:65)
>
> org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleCloseable.java:63)
>
> org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46)
>
> org.apache.sshd.common.util.closeable.AbstractCloseable.close(AbstractCloseable.java:83)
>
> org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(ParallelCloseable.java:65)
>
> org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleCloseable.java:63)
>
> org.apache.sshd.common.util.closeable.SequentialCloseable$1.operationComplete(SequentialCloseable.java:56)
>
> org.apache.sshd.common.util.closeable.SequentialCloseable$1.operationComplete(SequentialCloseable.java:45)
>
> org.apache.sshd.common.util.closeable.SequentialCloseable.doClose(SequentialCloseable.java:69)
>
> org.apache.sshd.common.util.closeable.SimpleCloseable.close(SimpleCloseable.java:63)
>
> org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doCloseImmediately(AbstractInnerCloseable.java:46)
>
> org.apache.sshd.common.util.closeable.AbstractCloseable.close(AbstractCloseable.java:83)
>
> org.apache.sshd.common.session.helpers.AbstractSession.exceptionCaught(AbstractSession.java:973)
>
> org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.exceptionCaught(AbstractSessionIoHandler.java:53)
>
> org.apache.sshd.common.io.nio2.Nio2Session.exceptionCaught(Nio2Session.java:186)
>
> org.apache.sshd.common.io.nio2.Nio2Session.handleWriteCycleFailure(Nio2Session.java:460)
>
> org.apache.sshd.common.io.nio2.Nio2Session$2.onFailed(Nio2Session.java:415)
>
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$failed$1(Nio2CompletionHandler.java:46)
>
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$627/1874891543.run(Unknown
> Source)
> java.security.AccessController.doPrivileged(Native Method)
>
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.failed(Nio2CompletionHandler.java:45)
> sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:128)
> sun.nio.ch.Invoker.invokeDirect(Invoker.java:157)
>
> sun.nio.ch.UnixAsynchronousSocketChannelImpl.implWrite(UnixAsynchronousSocketChannelImpl.java:736)
>
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:382)
>
> sun.nio.ch.AsynchronousSocketChannelImpl.write(AsynchronousSocketChannelImpl.java:399)
>
> org.apache.sshd.common.io.nio2.Nio2Session.doWriteCycle(Nio2Session.java:401)
>
> org.apache.sshd.common.io.nio2.Nio2Session.startWriting(Nio2Session.java:386)
>
> org.apache.sshd.common.io.nio2.Nio2Session.writePacket(Nio2Session.java:169)
>
> org.apache.sshd.common.session.helpers.AbstractSession.doWritePacket(AbstractSession.java:1181)
>
> org.apache.sshd.common.session.helpers.AbstractSession.sendPendingPackets(AbstractSession.java:910)
>
> org.apache.sshd.common.session.helpers.AbstractSession.handleNewKeys(AbstractSession.java:875)
>
> org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(AbstractSession.java:606)
>
> org.apache.sshd.common.session.helpers.AbstractSession.handleMessage(AbstractSession.java:555)
>
> org.apache.sshd.common.session.helpers.AbstractSession.decode(AbstractSession.java:1527)
>
> org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:516)
>
> org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:63)
>
> org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:339)
>
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:318)
>
> org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:315)
>
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
>
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler$$Lambda$349/1690110070.run(Unknown
> Source)
> java.security.AccessController.doPrivileged(Native Method)
>
> org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
> sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:126)
> sun.nio.ch.Invoker$2.run(Invoker.java:218)
>
> sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> "SSH git-upload-pack /project2 (myuser)" prio=1 BLOCKED
>
> org.apache.sshd.common.session.helpers.AbstractSession.writePacket(AbstractSession.java:1107)
>
> org.apache.sshd.common.channel.AbstractChannel.writePacket(AbstractChannel.java:798)
>
> org.apache.sshd.common.channel.ChannelOutputStream.flush(ChannelOutputStream.java:227)
>
> org.apache.sshd.common.channel.ChannelOutputStream.write(ChannelOutputStream.java:127)
>
> org.eclipse.jgit.transport.UploadPack$ResponseBufferedOutputStream.write(UploadPack.java:2183)
>
> org.eclipse.jgit.transport.SideBandOutputStream.writeBuffer(SideBandOutputStream.java:174)
>
> org.eclipse.jgit.transport.SideBandOutputStream.write(SideBandOutputStream.java:153)
>
> org.eclipse.jgit.internal.storage.pack.PackOutputStream.write(PackOutputStream.java:132)
>
> org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:614)
>
> org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:433)
>
> org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:221)
>
> org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1644)
>
> org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1621)
>
> org.eclipse.jgit.internal.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:171)
>
> org.eclipse.jgit.internal.storage.file.WindowCursor.writeObjects(WindowCursor.java:229)
>
> org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1609)
>
> org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1597)
>
> org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1154)
> org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2133)
> org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:1947)
> org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:971)
> org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:776)
> com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:77)
>
> com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:98)
>
> com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:31)
>
> com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:63)
> com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:467)
>
> com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:83)
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:646)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> {noformat}
> In AbstractSession.handleNewKeys(), there is a lock on {{pendingEvents}},
> while trying to send these packets; when sending the packets fails, an
> exception is generated, which causes the channel to get closed by calling
> ChannelOutputStream.close(), which is a synchronized method.
> At the same time, some other threads are trying to write data to the session:
> it calls ChannelOutputStream.write(), which is also a synchronized method,
> which calls AbstractSession.writePacket() which attempts to lock
> {{pendingPackets}} to queue the packets.
> *NOTE*: in our setup, we can reproduce this quite easily by increasing the
> parallelism in git-repo, reducing the values of waitTimeout
> (WAIT_FOR_SPACE_TIMEOUT), rekeyBytesLimit (REKEY_TIME_LIMIT) and
> rekeyTimeLimit (REKEY_TIME_LIMIT) configuration options.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
