[
https://issues.apache.org/jira/browse/TINKERPOP-2288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957660#comment-16957660
]
Some One edited comment on TINKERPOP-2288 at 10/23/19 8:51 AM:
---------------------------------------------------------------
For everyone who is waiting for a fix, the following work around with polly
does the trick for us, although it is not ideal. We already handled the '429
Too Many Requests' exceptions before, but now also added the handling of
ConnectionPoolBusyExceptions and ServerUnavailableExceptions where we
immediately create a new client (by calling the SetupGremlinClient method
below).
{code:java|title=gremlinClient.SubmitAsync|borderStyle=solid}
ResultSet<dynamic> response = await Polly.Policy
.Handle<ResponseException>(exception => exception.CosmosDbStatusCode() ==
RetryAfterStatusCode)
.Or<ConnectionPoolBusyException>()
.Or<ServerUnavailableException>()
.WaitAndRetryAsync(
retryCount: 10,
sleepDurationProvider: (attempt, exception, _) =>
{
if (exception is ResponseException responseException)
return responseException.CosmosDbRetryAfter();
// In case this is a connection issue, we try again immediately
return TimeSpan.Zero;
},
onRetryAsync: (exception, waitTime, attempt, _) =>
{
// In case this is a connection issue we reset the client
if (exception is ConnectionPoolBusyException || exception is
ServerUnavailableException)
{
_log.LogWarning($"Resetting the GremlinClient after
experiencing an '{exception.GetType()}'");
SetupGremlinClient();
}
return Task.CompletedTask;
})
.ExecuteAsync(() => _gremlinClient.SubmitAsync<dynamic>(query));
{code}
{code:java|title=SetupGremlinClient()|borderStyle=solid}
private void SetupGremlinClient()
{
var stopWatch = new Stopwatch();
stopWatch.Start();
var settings = new ConnectionPoolSettings{MaxInProcessPerConnection = 64,
PoolSize = 8};// Default MacInProcessPerConnection = 32, PoolSize = 4
_gremlinClient = new GremlinClient(_gremlinServer, new GraphSON2Reader(),
new GraphSON2Writer(),
GremlinClient.GraphSON2MimeType,settings
);
stopWatch.Stop();
_log.LogInformation($"Setting up GremlinClient with PoolSize of
{settings.PoolSize} and MaxInProcessPerConnection of
{settings.MaxInProcessPerConnection} took {stopWatch.ElapsedMilliseconds}ms");
}
{code}
{code:java|title=ResponseExceptionExtensions.cs|borderStyle=solid}
using System;
using Gremlin.Net.Driver.Exceptions;
// copied from
https://spzsource.github.io/programming/2018/11/01/gremlin-net-retry-policy.html
public static class ResponseExceptionExtensions
{
public static int CosmosDbStatusCode(this ResponseException source)
{
if (!source.StatusAttributes.TryGetValue("x-ms-status-code", out var
code))
{
throw new InvalidOperationException("Header 'x-ms-status-code' is
not presented.");
}
return Int32.Parse(code.ToString());
}
public static TimeSpan CosmosDbRetryAfter(this ResponseException source)
{
if (!source.StatusAttributes.TryGetValue("x-ms-retry-after-ms", out var
time))
{
throw new InvalidOperationException("Header 'x-ms-retry-after-ms'
is not presented.");
}
return TimeSpan.Parse(time.ToString());
}
}
{code}
was (Author: someoneelse):
For everyone who is waiting for a fix, the following work around with polly
does the trick for us, although it is not ideal. We already handled the '429
Too Many Requests' exceptions before, but now also added the handling of
ConnectionPoolBusyExceptions and ServerUnavailableExceptions where we
immediately create a new client (by calling the SetupGremlinClient method
below).
{code:java|title=gremlinClient.SubmitAsync|borderStyle=solid}
ResultSet<dynamic> response = await Polly.Policy
.Handle<ResponseException>(exception => exception.CosmosDbStatusCode() ==
RetryAfterStatusCode)
.Or<ConnectionPoolBusyException>()
.Or<ServerUnavailableException>()
.WaitAndRetryAsync(
retryCount: 10,
sleepDurationProvider: (attempt, exception, _) =>
{
if (exception is ResponseException responseException)
return responseException.CosmosDbRetryAfter();
// In case this is a connection issue, we try again immediately
return TimeSpan.Zero;
},
onRetryAsync: (exception, waitTime, attempt, _) =>
{
// In case this is a connection issue we reset the client
if (exception is ConnectionPoolBusyException || exception is
ServerUnavailableException)
{
_log.LogWarning($"Resetting the GremlinClient after
experiencing an '{exception.GetType()}'");
SetupGremlinClient();
}
return Task.CompletedTask;
})
.ExecuteAsync(() => _gremlinClient.SubmitAsync<dynamic>(query));
{code}
{code:java|title=SetupGremlinClient()|borderStyle=solid}
private void SetupGremlinClient()
{
var stopWatch = new Stopwatch();
stopWatch.Start();
var settings = new ConnectionPoolSettings{MaxInProcessPerConnection = 64,
PoolSize = 8};// Default MacInProcessPerConnection = 32, PoolSize = 4
_gremlinClient = new GremlinClient(_gremlinServer, new GraphSON2Reader(),
new GraphSON2Writer(),
GremlinClient.GraphSON2MimeType,settings
);
stopWatch.Stop();
_log.LogInformation($"Setting up GremlinClient with PoolSize of
{settings.PoolSize} and MaxInProcessPerConnection of
{settings.MaxInProcessPerConnection} took {stopWatch.ElapsedMilliseconds}ms");
}
{code}
> Get ConnectionPoolBusyException and then ServerUnavailableExceptions
> --------------------------------------------------------------------
>
> Key: TINKERPOP-2288
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2288
> Project: TinkerPop
> Issue Type: Bug
> Components: dotnet
> Affects Versions: 3.4.1
> Environment: Gremlin.Net 3.4.1
> Microsoft.NetCore.App 2.2
> Azure Cosmos DB
> Reporter: patrice huot
> Priority: Critical
>
> I am using .Net core Gremlin API query Cosmos DB.
> From time to time we are getting an error saying that no connection is
> available and then the server become unavailable. When this is occurring we
> need to restart the server. It looks like the connections are not released
> properly and become unavailable forever.
> We have configured the pool size to 50 and the MaxInProcessPerConnection to
> 32 (Which I guess should be sufficient).
> To diagnose the issue, Is there a way to access diagnostic information on the
> connection pool in order to know how many connections are open and how many
> processes are running in each connection?
> I would like to be able to monitor the connections usage to see if they are
> about to be exhausted and to see if the number of used connections is always
> increasing or of the connection lease is release when the queries completes?
> As a work around, Is there a way we can access this information from the code
> so that I can catch those scenario and create logic that re-initiate the
> connection pool?
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
