[
https://issues.apache.org/jira/browse/TINKERPOP-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17316153#comment-17316153
]
Florian Hockmann commented on TINKERPOP-2341:
---------------------------------------------
[~kirk-marple] Sorry for the late answer but I see that you have already
created a new issue for the problem you're describing: TINKERPOP-2541. It also
doesn't sound that related to the problem in this issue.
{quote}Are there still any known issues around this in 3.4.10?
{quote}
I'm at least not aware of any issues that are really related to the problem
here about the driver hanging when it tries to dispose the connection.
Given my earlier comment from November, I'm closing this issue as it seems to
be resolved already as part of our work on TINKERPOP-2288 which completely
changed how the driver handles closed connections.
> GremlinClientExtensions.SubmitAsync hangs as it tries to dispose connection
> ---------------------------------------------------------------------------
>
> Key: TINKERPOP-2341
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2341
> Project: TinkerPop
> Issue Type: Bug
> Components: dotnet
> Affects Versions: 3.4.3
> Reporter: Wilson Li
> Priority: Major
>
> We have been seeing occasional hangs when logging to Cosmos DB graph using
> the Gremlin.Net NuGet (version 3.4.3).
>
> Here's our code snippet:
> var results = await
> GremlinClientExtensions.SubmitAsync<dynamic>(this.Client,
> gremlinExpression).ConfigureAwait(false);
>
> foreach (var result in results)
> {
> Tracer.TraceMessage()(
> traceId,
> "Successfully logged activity result (ElapsedMilliseconds=\{0}):
> '\{1}'. Request charge is '\{2}'.",
> sw.ElapsedMilliseconds,
> JsonConvert.SerializeObject(result),
> this.GetRequestCharge(results.StatusAttributes));
> }
>
> From the dumps, it seems the connection to the DB may have hit some
> {color:#de350b}failure{color}:
> 0:000> !PrintException /d 000001c61a0fd2c8
> Exception object: 000001c61a0fd2c8
> Exception type: System.Net.WebSockets.WebSocketException
> Message: An internal WebSocket error occurred. Please see the innerException,
> if present, for more details.
> InnerException: System.IO.IOException, Use !PrintException 000001c61a0fa760
> to see more.
> StackTrace (generated):
> SP IP Function
> 000000D5B446CA20 00007FFD9263D383
> System_ni!System.Net.WebSockets.WebSocketBase.ThrowIfConvertibleException(System.String,
> System.Exception, System.Threading.CancellationToken, Boolean)+0x483
> 000000D5B446CAC0 00007FFD9203372B
> System_ni!System.Net.WebSockets.WebSocketBase+<ReceiveAsyncCore>d__45.MoveNext()+0x56b
> 000000D5B446A360 00007FFD93F7C3DF
> mscorlib_ni!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()+0x1f
> 000000D5B446A390 00007FFD935EDF2C
> mscorlib_ni!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)+0x3c
> 000000D5B446A3C0 00007FFD35D7B46C
> Gremlin_Net!Gremlin.Net.Driver.WebSocketConnection+<ReceiveMessageAsync>d__9.MoveNext()+0x20c
> 000000D5B4467CA0 00007FFD93F7C3DF
> mscorlib_ni!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()+0x1f
> 000000D5B4467CD0 00007FFD935EDF2C
> mscorlib_ni!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)+0x3c
> 000000D5B4467D00 00007FFD35D7A96F
> Gremlin_Net!Gremlin.Net.Driver.Connection+<ReceiveMessagesAsync>d__20.MoveNext()+0x16f
> 000000D3EF8FE770 00007FFD93F7C3DF
> mscorlib_ni!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()+0x1f
> 000000D3EF8FE7A0 00007FFD935EDF2C
> mscorlib_ni!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)+0x3c
> 000000D3EF8FE7D0 00007FFD35D8B1BC
> Gremlin_Net!Gremlin.Net.Driver.ProxyConnection+<SubmitAsync>d__3`1[[System.__Canon,
> mscorlib]].MoveNext()+0x15c
> 000000D3EF8FC0D0 00007FFD93F7C3DF
> mscorlib_ni!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()+0x1f
> 000000D3EF8FC100 00007FFD935EDF2C
> mscorlib_ni!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)+0x3c
> 000000D3EF8FC130 00007FFD35D8887C
> Gremlin_Net!Gremlin.Net.Driver.GremlinClient+<SubmitAsync>d__6`1[[System.__Canon,
> mscorlib]].MoveNext()+0x2ac
>
> StackTraceString: <none>
> HResult: 80004005
> 0:000> !PrintException /d 000001c61a0fa760
> Exception object: 000001c61a0fa760
> Exception type: System.IO.IOException
> Message: Unable to read data from the transport connection: An existing
> connection was forcibly closed by the remote host.
> InnerException: System.Net.Sockets.SocketException, Use !PrintException
> 000001c61a0fa570 to see more.
> StackTrace (generated):
> SP IP Function
> 000000D5B447A610 00007FFD928993C3
> System_ni!System.Net.Security._SslStream.EndRead(System.IAsyncResult)+0x7f3873
> 000000D5B447A660 00007FFD920A5B4C
> System_ni!System.Net.TlsStream.EndRead(System.IAsyncResult)+0x10c
> 000000D5B447A6B0 00007FFD94043F13
> mscorlib_ni!System.Threading.Tasks.TaskFactory`1+FromAsyncTrimPromise`1[[System.Int32,
> mscorlib],[System.__Canon, mscorlib]].Complete(System.__Canon,
> System.Func`3<System.__Canon,System.IAsyncResult,Int32>, System.IAsyncResult,
> Boolean)+0x23
> 000000D5B4475DD0 00007FFD93F7C3DF
> mscorlib_ni!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()+0x1f
> 000000D5B4475E00 00007FFD935EDF2C
> mscorlib_ni!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)+0x3c
> 000000D5B4475E30 00007FFD92035D25
> System_ni!System.Net.WebSockets.WebSocketConnectionStream+<ReadAsync>d__21.MoveNext()+0x385
> 000000D5B4471430 00007FFD93F7C3DF
> mscorlib_ni!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()+0x1f
> 000000D5B4471460 00007FFD935EDF2C
> mscorlib_ni!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)+0x3c
> 000000D5B4471490 00007FFD92034541
> System_ni!System.Net.WebSockets.WebSocketBase+WebSocketOperation+<Process>d__19.MoveNext()+0x8e1
> 000000D5B446ED40 00007FFD93F7C3DF
> mscorlib_ni!System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()+0x1f
> 000000D5B446ED70 00007FFD935EDF2C
> mscorlib_ni!System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)+0x3c
> 000000D5B446EDA0 00007FFD9203339E
> System_ni!System.Net.WebSockets.WebSocketBase+<ReceiveAsyncCore>d__45.MoveNext()+0x1de
>
> StackTraceString: <none>
> HResult: 80131620
> 0:000> !PrintException /d 000001c61a0fa570
> Exception object: 000001c61a0fa570
> {color:#de350b}Exception type: System.Net.Sockets.SocketException{color}
> {color:#de350b}Message: An existing connection was forcibly closed by the
> remote host{color}
> InnerException: <none>
> StackTrace (generated):
> SP IP Function
> 000000D5B447ED00 00007FFD928975F7
> System_ni!System.Net.Sockets.Socket.EndReceive(System.IAsyncResult)+0x7fe207
> 000000D5B447ED40 00007FFD9209924D
> System_ni!System.Net.Sockets.NetworkStream.EndRead(System.IAsyncResult)+0x2d
>
>
> But then the Gremlin.Net 's exception handling to {color:#ff8b00}dispose and
> close{color} connections seemed to hit some deadlock:
> 0:000> ~26e !clrstack
> OS Thread Id: 0x31b8 (26)
> Child SP IP Call Site
> 000000d3ef8f8798 +00007ffda80b6594+ [GCFrame: 000000d3ef8f8798]
> 000000d3ef8f8868 +00007ffda80b6594+ [HelperMethodFrame_1OBJ:
> 000000d3ef8f8868] System.Threading.Monitor.ObjWait(Boolean, Int32,
> System.Object)
> 000000d3ef8f8980 +00007ffd935f6a0f+
> System.Threading.ManualResetEventSlim.Wait(Int32,
> System.Threading.CancellationToken)
> [f:\dd\ndp\clr\src\BCL\system\threading\ManualResetEventSlim.cs @ 669]
> 000000d3ef8f8a10 +00007ffd935ef815+
> System.Threading.Tasks.Task.SpinThenBlockingWait(Int32,
> System.Threading.CancellationToken)
> [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Task.cs @ 3320]
> 000000d3ef8f8a80 +00007ffd93f60a88+
> System.Threading.Tasks.Task.InternalWait(Int32,
> System.Threading.CancellationToken)
> [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Task.cs @ 3259]
> 000000d3ef8f8b50 +00007ffd935eef08+ System.Threading.Tasks.Task.Wait(Int32,
> System.Threading.CancellationToken)
> [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Task.cs @ 3167]
> {color:#ff8b00}000000d3ef8f8b80 +00007ffd35d53f7f+
> Gremlin.Net.Process.Utils.WaitUnwrap(System.Threading.Tasks.Task){color}
> {color:#ff8b00}000000d3ef8f8bc0 +00007ffd35db01dc+
> Gremlin.Net.Driver.ConnectionPool.ReturnConnectionIfOpen(Gremlin.Net.Driver.Connection){color}
> 000000d3ef8f8bf0 +00007ffd35d88937+
> Gremlin.Net.Driver.GremlinClient+d__6`1[[System.__Canon, mscorlib]].MoveNext()
> 000000d3ef8fbfe8 +00007ffd94756f21+ [HelperMethodFrame: 000000d3ef8fbfe8]
> 000000d3ef8fc0d0 +00007ffd93f7c3df+
> System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
> [f:\dd\ndp\clr\src\BCL\system\runtime\exceptionservices\exceptionservicescommon.cs
> @ 133]
> 000000d3ef8fc100 +00007ffd935edf2c+
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
> [f:\dd\ndp\clr\src\BCL\system\runtime\compilerservices\TaskAwaiter.cs @ 156]
> 000000d3ef8fc130 +00007ffd35d8887c+
> Gremlin.Net.Driver.GremlinClient+d__6`1[[System.__Canon, mscorlib]].MoveNext()
> 000000d3ef8fc1e0 +00007ffd935a5ab3+
> System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext,
> System.Threading.ContextCallback, System.Object, Boolean)
> [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 954]
> 000000d3ef8fc2b0 +00007ffd935a5944+
> System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
> System.Threading.ContextCallback, System.Object, Boolean)
> [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 902]
> 000000d3ef8fc2e0 +00007ffd935f12bc+
> System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
> [f:\dd\ndp\clr\src\BCL\system\runtime\compilerservices\AsyncMethodBuilder.cs
> @ 1070]
> 000000d3ef8fc330 +00007ffd935f80b2+
> System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action,
> Boolean, System.Threading.Tasks.Task ByRef)
> [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\TaskContinuation.cs @ 810]
> 000000d3ef8fc380 +00007ffd93573dc7+
> System.Threading.Tasks.Task.FinishContinuations()
> [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Task.cs @ 3617]
> 000000d3ef8fc410 +00007ffd935ee890+
> System.Threading.Tasks.Task.Finish(Boolean)
> [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Task.cs @ 2263]
> 000000d3ef8fc470 +00007ffd935f0101+
> System.Threading.Tasks.Task`1[[System.__Canon,
> mscorlib]].TrySetException(System.Object)
> [f:\dd\ndp\clr\src\BCL\system\threading\Tasks\Future.cs @ 606]
> 000000d3ef8fc4c0 +00007ffd935f06a8+
> System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon,
> mscorlib]].SetException(System.Exception)
> 000000d3ef8fc500 +00007ffd35d8b27d+
> Gremlin.Net.Driver.ProxyConnection+d__3`1[[System.__Canon,
> mscorlib]].MoveNext()
> 000000d3ef8fe688 +00007ffd94756f21+ [HelperMethodFrame: 000000d3ef8fe688]
> 000000d3ef8fe770 +00007ffd93f7c3df+
> System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
> [f:\dd\ndp\clr\src\BCL\system\runtime\exceptionservices\exceptionservicescommon.cs
> @ 133]
> 000000d3ef8fe7a0 +00007ffd935edf2c+
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
> [f:\dd\ndp\clr\src\BCL\system\runtime\compilerservices\TaskAwaiter.cs @ 156]
> 000000d3ef8fe7d0 +00007ffd35d8b1bc+
> Gremlin.Net.Driver.ProxyConnection+d__3`1[[System.__Canon,
> mscorlib]].MoveNext()
> 000000d3ef8fe860 +00007ffd935a5ab3+
> System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext,
> System.Threading.ContextCallback, System.Object, Boolean)
> [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 954]
> 000000d3ef8fe930 +00007ffd935a5944+
> System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
> System.Threading.ContextCallback, System.Object, Boolean)
> [f:\dd\ndp\clr\src\BCL\system\threading\executioncontext.cs @ 902]
> 000000d3ef8fe960 +00007ffd935f12bc+
> System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
> [f:\dd\ndp\clr\src\BCL\system\runtime\compilerservices\AsyncMethodBuilder.cs
> @ 1070]
> 000000d3ef8fe9b0 +00007ffd935e4e22+
> System.Threading.ThreadPoolWorkQueue.Dispatch()
> [f:\dd\ndp\clr\src\BCL\system\threading\threadpool.cs @ 820]
> 000000d3ef8fee48 +00007ffd94646a93+ [DebuggerU2MCatchHandlerFrame:
> 000000d3ef8fee48]
>
> Looks like Gremlin.Net handles a dead connection by attempting to forcing a
> closure of the connection pool. This path calls Task.Wait() instead of using
> await Task, so it is very possible that this is the cause of the hang.
>
> Might be related to this issue:
> [https://issues.apache.org/jira/projects/TINKERPOP/issues/TINKERPOP-2268?filter=allopenissues&orderby=affectedVersion+DESC%2C+priority+DESC%2C+updated+DESC]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
