[
https://issues.apache.org/jira/browse/JCLOUDS-1294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Svetoslav Neykov updated JCLOUDS-1294:
--------------------------------------
Description:
The retry logic in {{AzureRateLimitRetryHandler}} doesn't catch all requests
that can be retried.
{noformat}
>> PUT https://management.azure.com/subscriptions/<subscription
>> id>/resourcegroups/jclouds-westeurope/providers/Microsoft.Network/virtualNetworks/network-security-vnet-q8nvtq9kkr/subnets/web-tier?api-version=2017-03-01
>> HTTP/1.1
>> Accept: application/json
>> Content-Type: application/json
>> Content-Length: 48
...
2017-05-12 13:40:31,962 DEBUG 106
o.j.i.ContentMetadataCodec$DefaultContentMetadataCodec [ager-N7FStlcv-20]
Invalid Expires header (-1); should be in RFC-1123 format; treating as already
expired: Error parsing data at 0
2017-05-12 13:40:31,962 DEBUG 106 o.j.h.o.OkHttpCommandExecutorService
[ager-N7FStlcv-20] Receiving response 290079285: HTTP/1.1 429
<< HTTP/1.1 429
<< Connection: close
<< Date: Fri, 12 May 2017 10:40:30 GMT
<< OkHttp-Received-Millis: 1494585631962
<< OkHttp-Selected-Protocol: http/1.1
<< OkHttp-Sent-Millis: 1494585586349
<< Pragma: no-cache
<< Server: Microsoft-HTTPAPI/2.0
<< Strict-Transport-Security: max-age=31536000; includeSubDomains
<< x-ms-correlation-request-id: 4b7a44ea-1976-4dc2-95ef-1a7c80f82d3c
<< x-ms-ratelimit-remaining-subscription-writes: 1195
<< x-ms-request-id: 4b7a44ea-1976-4dc2-95ef-1a7c80f82d3c
<< x-ms-routing-request-id: CANADAEAST:20170512T104031Z:4b7a44ea-
<< Cache-Control: no-cache
<< Content-Type: application/json; charset=utf-8
<< Content-Length: 2545
<< Expires: Thu Jan 01 02:00:00 EET 1970
2017-05-12 13:40:31,963 ERROR 106 o.j.a.a.h.AzureRateLimitRetryHandler
[ager-N7FStlcv-20] Cannot retry after rate limit error, no retry information
provided in the response
...
2017-05-12 13:40:31,967 DEBUG 126 o.a.b.l.j.JcloudsLocation [ager-N7FStlcv-20]
org.jclouds.azurecompute.arm.exceptions.AzureComputeRateLimitExceededException:
HTTP/1.1 429
{x-ms-ratelimit-remaining-subscription-writes=[1195]}
...
Caused by: org.jclouds.http.HttpResponseException: command: PUT
https://management.azure.com/subscriptions/<subscription
id>/resourcegroups/jclouds-westeurope/providers/Microsoft.Network/virtualNetworks/network-security-vnet-q8nvtq9kkr/subnets/web-tier
HTTP/1.1 failed with response: HTTP/1.1 429 ; content:
[{"Error":{"Details":[{"Code":"GatewayError","Message":"Error occured in
resource provider infrastructure
services.","Target":null}],"InnerError":"Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException:
A retryable error occured. --->
Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException: Error
occured in resource provider infrastructure services. --->
System.Threading.Tasks.TaskCanceledException: A task was canceled.\r\n at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)\r\n at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n
at
Microsoft.WindowsAzure.Networking.Nrp.Common.ReliableHttpClient.HttpClientWrapper.<SendAsync>d__4.MoveNext()
in
E:\\bt\\804732\\repo\\src\\sources\\Common\\ReliableHttpClient\\HttpClientWrapper.cs:line
80\r\n--- End of stack trace from previous location where exception was thrown
---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task)\r\n at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)\r\n at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n
at
Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<CallNrpAndGetResponseWithRetries>d__23.MoveNext()
in
E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line
414\r\n--- End of stack trace from previous location where exception was
thrown ---\r\n at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)\r\n at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n
at
Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<ForwardMessageToNrpAndProcessResponseAsync>d__20.MoveNext()
in
E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line
160\r\n --- End of inner exception stack trace ---\r\n --- End of inner
exception stack trace ---","Code":"RetryableError","Message":"A retryable error
occured.","Target":null}}]
at
org.jclouds.azurecompute.arm.handlers.AzureComputeErrorHandler.handleError(AzureComputeErrorHandler.java:45)
... 33 more
{noformat}
The response body expanded here for readability:
{noformat}
PUT
https://management.azure.com/subscriptions/341751b0-f348-45ce-9498-41cc68b4b45f/resourcegroups/jclouds-westeurope/providers/Microsoft.Network/virtualNetworks/network-security-vnet-q8nvtq9kkr/subnets/web-tier
HTTP/1.1
failed with response: HTTP/1.1 429 ; content:
[{"Error":{"Details":[{"Code":"GatewayError","Message":"Error occured in
resource provider infrastructure services.","Target":null}],
"InnerError":"Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException:
A retryable error occured.
---> Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException: Error
occured in resource provider infrastructure services.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at
Microsoft.WindowsAzure.Networking.Nrp.Common.ReliableHttpClient.HttpClientWrapper.<SendAsync>d__4.MoveNext()
in
E:\\bt\\804732\\repo\\src\\sources\\Common\\ReliableHttpClient\\HttpClientWrapper.cs:line
80
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at
Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<CallNrpAndGetResponseWithRetries>d__23.MoveNext()
in
E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line
414
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at
Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<ForwardMessageToNrpAndProcessResponseAsync>d__20.MoveNext()
in
E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line
160
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
","Code":"RetryableError","Message":"A retryable error
occured.","Target":null}}]
{noformat}
---
Not all retriable requests get the {{Retry-After}} header.
was:
The retry logic in {{AzureRateLimitRetryHandler}} doesn't catch all requests
that can be retried.
Example 1:
{noformat}
>> PUT https://management.azure.com/subscriptions/<subscription
>> id>/resourcegroups/jclouds-westeurope/providers/Microsoft.Network/virtualNetworks/network-security-vnet-q8nvtq9kkr/subnets/web-tier?api-version=2017-03-01
>> HTTP/1.1
>> Accept: application/json
>> Content-Type: application/json
>> Content-Length: 48
...
2017-05-12 13:40:31,962 DEBUG 106
o.j.i.ContentMetadataCodec$DefaultContentMetadataCodec [ager-N7FStlcv-20]
Invalid Expires header (-1); should be in RFC-1123 format; treating as already
expired: Error parsing data at 0
2017-05-12 13:40:31,962 DEBUG 106 o.j.h.o.OkHttpCommandExecutorService
[ager-N7FStlcv-20] Receiving response 290079285: HTTP/1.1 429
<< HTTP/1.1 429
<< Connection: close
<< Date: Fri, 12 May 2017 10:40:30 GMT
<< OkHttp-Received-Millis: 1494585631962
<< OkHttp-Selected-Protocol: http/1.1
<< OkHttp-Sent-Millis: 1494585586349
<< Pragma: no-cache
<< Server: Microsoft-HTTPAPI/2.0
<< Strict-Transport-Security: max-age=31536000; includeSubDomains
<< x-ms-correlation-request-id: 4b7a44ea-1976-4dc2-95ef-1a7c80f82d3c
<< x-ms-ratelimit-remaining-subscription-writes: 1195
<< x-ms-request-id: 4b7a44ea-1976-4dc2-95ef-1a7c80f82d3c
<< x-ms-routing-request-id: CANADAEAST:20170512T104031Z:4b7a44ea-
<< Cache-Control: no-cache
<< Content-Type: application/json; charset=utf-8
<< Content-Length: 2545
<< Expires: Thu Jan 01 02:00:00 EET 1970
2017-05-12 13:40:31,963 ERROR 106 o.j.a.a.h.AzureRateLimitRetryHandler
[ager-N7FStlcv-20] Cannot retry after rate limit error, no retry information
provided in the response
...
2017-05-12 13:40:31,967 DEBUG 126 o.a.b.l.j.JcloudsLocation [ager-N7FStlcv-20]
org.jclouds.azurecompute.arm.exceptions.AzureComputeRateLimitExceededException:
HTTP/1.1 429
{x-ms-ratelimit-remaining-subscription-writes=[1195]}
...
Caused by: org.jclouds.http.HttpResponseException: command: PUT
https://management.azure.com/subscriptions/<subscription
id>/resourcegroups/jclouds-westeurope/providers/Microsoft.Network/virtualNetworks/network-security-vnet-q8nvtq9kkr/subnets/web-tier
HTTP/1.1 failed with response: HTTP/1.1 429 ; content:
[{"Error":{"Details":[{"Code":"GatewayError","Message":"Error occured in
resource provider infrastructure
services.","Target":null}],"InnerError":"Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException:
A retryable error occured. --->
Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException: Error
occured in resource provider infrastructure services. --->
System.Threading.Tasks.TaskCanceledException: A task was canceled.\r\n at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)\r\n at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n
at
Microsoft.WindowsAzure.Networking.Nrp.Common.ReliableHttpClient.HttpClientWrapper.<SendAsync>d__4.MoveNext()
in
E:\\bt\\804732\\repo\\src\\sources\\Common\\ReliableHttpClient\\HttpClientWrapper.cs:line
80\r\n--- End of stack trace from previous location where exception was thrown
---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
task)\r\n at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)\r\n at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n
at
Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<CallNrpAndGetResponseWithRetries>d__23.MoveNext()
in
E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line
414\r\n--- End of stack trace from previous location where exception was
thrown ---\r\n at
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at
System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)\r\n at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n
at
Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<ForwardMessageToNrpAndProcessResponseAsync>d__20.MoveNext()
in
E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line
160\r\n --- End of inner exception stack trace ---\r\n --- End of inner
exception stack trace ---","Code":"RetryableError","Message":"A retryable error
occured.","Target":null}}]
at
org.jclouds.azurecompute.arm.handlers.AzureComputeErrorHandler.handleError(AzureComputeErrorHandler.java:45)
... 33 more
{noformat}
The response body expanded here for readability:
{noformat}
PUT
https://management.azure.com/subscriptions/341751b0-f348-45ce-9498-41cc68b4b45f/resourcegroups/jclouds-westeurope/providers/Microsoft.Network/virtualNetworks/network-security-vnet-q8nvtq9kkr/subnets/web-tier
HTTP/1.1
failed with response: HTTP/1.1 429 ; content:
[{"Error":{"Details":[{"Code":"GatewayError","Message":"Error occured in
resource provider infrastructure services.","Target":null}],
"InnerError":"Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException:
A retryable error occured.
---> Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException: Error
occured in resource provider infrastructure services.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at
Microsoft.WindowsAzure.Networking.Nrp.Common.ReliableHttpClient.HttpClientWrapper.<SendAsync>d__4.MoveNext()
in
E:\\bt\\804732\\repo\\src\\sources\\Common\\ReliableHttpClient\\HttpClientWrapper.cs:line
80
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at
Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<CallNrpAndGetResponseWithRetries>d__23.MoveNext()
in
E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line
414
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
task)
at
System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
at
Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<ForwardMessageToNrpAndProcessResponseAsync>d__20.MoveNext()
in
E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line
160
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
","Code":"RetryableError","Message":"A retryable error
occured.","Target":null}}]
{noformat}
---
Example 2:
{noformat}
Caused by: java.lang.IllegalStateException: {
"error": {
"code": "AnotherOperationInProgress",
"message": "Another operation on this or dependent resource is in
progress. To retrieve status of the operation use uri:
https://management.azure.com/subscriptions/<subscription
id>/providers/Microsoft.Network/locations/westeurope/operations/b43b79d9-7cb7-46ad-989e-5b8d0515249c?api-version=2017-03-01.",
"details": []
}
}
{noformat}
No wire logs available for this one.
---
Not all retriable requests get the {{Retry-After}} header.
> Azure ARM improve HTTP retry logic
> ----------------------------------
>
> Key: JCLOUDS-1294
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1294
> Project: jclouds
> Issue Type: Bug
> Components: jclouds-compute
> Affects Versions: 2.0.1
> Reporter: Svetoslav Neykov
> Labels: azurecompute-arm
>
> The retry logic in {{AzureRateLimitRetryHandler}} doesn't catch all requests
> that can be retried.
> {noformat}
> >> PUT https://management.azure.com/subscriptions/<subscription
> >> id>/resourcegroups/jclouds-westeurope/providers/Microsoft.Network/virtualNetworks/network-security-vnet-q8nvtq9kkr/subnets/web-tier?api-version=2017-03-01
> >> HTTP/1.1
> >> Accept: application/json
> >> Content-Type: application/json
> >> Content-Length: 48
> ...
> 2017-05-12 13:40:31,962 DEBUG 106
> o.j.i.ContentMetadataCodec$DefaultContentMetadataCodec [ager-N7FStlcv-20]
> Invalid Expires header (-1); should be in RFC-1123 format; treating as
> already expired: Error parsing data at 0
> 2017-05-12 13:40:31,962 DEBUG 106 o.j.h.o.OkHttpCommandExecutorService
> [ager-N7FStlcv-20] Receiving response 290079285: HTTP/1.1 429
> << HTTP/1.1 429
> << Connection: close
> << Date: Fri, 12 May 2017 10:40:30 GMT
> << OkHttp-Received-Millis: 1494585631962
> << OkHttp-Selected-Protocol: http/1.1
> << OkHttp-Sent-Millis: 1494585586349
> << Pragma: no-cache
> << Server: Microsoft-HTTPAPI/2.0
> << Strict-Transport-Security: max-age=31536000; includeSubDomains
> << x-ms-correlation-request-id: 4b7a44ea-1976-4dc2-95ef-1a7c80f82d3c
> << x-ms-ratelimit-remaining-subscription-writes: 1195
> << x-ms-request-id: 4b7a44ea-1976-4dc2-95ef-1a7c80f82d3c
> << x-ms-routing-request-id: CANADAEAST:20170512T104031Z:4b7a44ea-
> << Cache-Control: no-cache
> << Content-Type: application/json; charset=utf-8
> << Content-Length: 2545
> << Expires: Thu Jan 01 02:00:00 EET 1970
> 2017-05-12 13:40:31,963 ERROR 106 o.j.a.a.h.AzureRateLimitRetryHandler
> [ager-N7FStlcv-20] Cannot retry after rate limit error, no retry information
> provided in the response
> ...
> 2017-05-12 13:40:31,967 DEBUG 126 o.a.b.l.j.JcloudsLocation
> [ager-N7FStlcv-20]
> org.jclouds.azurecompute.arm.exceptions.AzureComputeRateLimitExceededException:
> HTTP/1.1 429
> {x-ms-ratelimit-remaining-subscription-writes=[1195]}
> ...
> Caused by: org.jclouds.http.HttpResponseException: command: PUT
> https://management.azure.com/subscriptions/<subscription
> id>/resourcegroups/jclouds-westeurope/providers/Microsoft.Network/virtualNetworks/network-security-vnet-q8nvtq9kkr/subnets/web-tier
> HTTP/1.1 failed with response: HTTP/1.1 429 ; content:
> [{"Error":{"Details":[{"Code":"GatewayError","Message":"Error occured in
> resource provider infrastructure
> services.","Target":null}],"InnerError":"Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException:
> A retryable error occured. --->
> Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException: Error
> occured in resource provider infrastructure services. --->
> System.Threading.Tasks.TaskCanceledException: A task was canceled.\r\n at
> System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
> task)\r\n at
> System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n
> at
> Microsoft.WindowsAzure.Networking.Nrp.Common.ReliableHttpClient.HttpClientWrapper.<SendAsync>d__4.MoveNext()
> in
> E:\\bt\\804732\\repo\\src\\sources\\Common\\ReliableHttpClient\\HttpClientWrapper.cs:line
> 80\r\n--- End of stack trace from previous location where exception was
> thrown ---\r\n at
> System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at
> System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
> task)\r\n at
> System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n
> at
> Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<CallNrpAndGetResponseWithRetries>d__23.MoveNext()
> in
> E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line
> 414\r\n--- End of stack trace from previous location where exception was
> thrown ---\r\n at
> System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at
> System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
> task)\r\n at
> System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n
> at
> Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<ForwardMessageToNrpAndProcessResponseAsync>d__20.MoveNext()
> in
> E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line
> 160\r\n --- End of inner exception stack trace ---\r\n --- End of inner
> exception stack trace ---","Code":"RetryableError","Message":"A retryable
> error occured.","Target":null}}]
> at
> org.jclouds.azurecompute.arm.handlers.AzureComputeErrorHandler.handleError(AzureComputeErrorHandler.java:45)
> ... 33 more
> {noformat}
> The response body expanded here for readability:
> {noformat}
> PUT
> https://management.azure.com/subscriptions/341751b0-f348-45ce-9498-41cc68b4b45f/resourcegroups/jclouds-westeurope/providers/Microsoft.Network/virtualNetworks/network-security-vnet-q8nvtq9kkr/subnets/web-tier
> HTTP/1.1
> failed with response: HTTP/1.1 429 ; content:
> [{"Error":{"Details":[{"Code":"GatewayError","Message":"Error occured in
> resource provider infrastructure services.","Target":null}],
> "InnerError":"Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException:
> A retryable error occured.
> ---> Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException:
> Error occured in resource provider infrastructure services.
> ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
> task)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
> task)
> at
> System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
> at
> Microsoft.WindowsAzure.Networking.Nrp.Common.ReliableHttpClient.HttpClientWrapper.<SendAsync>d__4.MoveNext()
> in
> E:\\bt\\804732\\repo\\src\\sources\\Common\\ReliableHttpClient\\HttpClientWrapper.cs:line
> 80
> --- End of stack trace from previous location where exception was thrown ---
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
> task)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
> task)
> at
> System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
> at
> Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<CallNrpAndGetResponseWithRetries>d__23.MoveNext()
> in
> E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line
> 414
> --- End of stack trace from previous location where exception was thrown ---
> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task
> task)
> at
> System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task
> task)
> at
> System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
> at
> Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<ForwardMessageToNrpAndProcessResponseAsync>d__20.MoveNext()
> in
> E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line
> 160
> --- End of inner exception stack trace ---
> --- End of inner exception stack trace ---
> ","Code":"RetryableError","Message":"A retryable error
> occured.","Target":null}}]
> {noformat}
> ---
> Not all retriable requests get the {{Retry-After}} header.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
