[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[
https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17571793#comment-17571793
]
George Ladutsko commented on CAMEL-16287:
-
Unfortunately, after fix (3.14.4) an application throws an exception:
{panel:title=Exception}
Caused by: software.amazon.awssdk.services.sqs.model.SqsException: MaxResults
is a mandatory parameter when you provide a value for NextToken. (Service: Sqs,
Status Code: 400, Request ID: 8653ea4e-3dfb-5ea6-89fc-b114d624928d)
at
software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:125)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:82)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:60)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:41)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[sdk-core-2.17.174.jar:na]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
~[sdk-core-2.17.174.jar:na]
at
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17542628#comment-17542628 ] Andrea Cosentino commented on CAMEL-16287: -- There is no plan for another 3.17.x release. It's not an LTS. It will be fixed in 3.18.x and LTS. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.14.4, 3.18.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17542586#comment-17542586 ] Zoltán Nébli commented on CAMEL-16287: -- Please treat this with high prio and possibly create a minor release like 3.17.1 because this is very annoying and affecting us also. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.14.4, 3.18.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534329#comment-17534329 ] Stephen Lawson commented on CAMEL-16287: Understood. I have a workaround for now. We are migrating a lot of apps from 2 to 3 so will target 3.18 when we start to make the move on all the apps. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.14.4, 3.18.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534327#comment-17534327 ] Andrea Cosentino commented on CAMEL-16287: -- Probably this will go in 3.18.0, not sure I'll have the time to fix this in this week. 3.14.4 will be in a longer time, so no rush. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.14.4, 3.18.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[
https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534324#comment-17534324
]
Stephen Lawson commented on CAMEL-16287:
Hi. Sorry to be a pain but the fix is not quite right (v 3.14.3) . When the
nextToken is null then no further calls to the listQueues() method should be
called. I think this would be better:
{code:java}
// check whether the queue already exists
boolean done = false;
while (!done) {
ListQueuesResponse listQueuesResult
=
client.listQueues(ListQueuesRequest.builder().maxResults(1000).build()); for
(String url : listQueuesResult.queueUrls()) {
if (url.endsWith("/" + configuration.getQueueName())) {
queueUrl = url;
LOG.trace("Queue available at '{}'.", queueUrl);
break;
}
}
if (listQueuesResult.nextToken() == null) {
done = true;
continue;
}
String token = listQueuesResult.nextToken();
listQueuesResult =
client.listQueues(ListQueuesRequest.builder().nextToken(token).build());
}{code}
This ensures the loop terminates before trying to do the next token. There is
probably a more elegant approach but this is the least intrusive.
> camel-aws2-sqs should use pagination for deciding which aws sqs queues it
> should create
> ---
>
> Key: CAMEL-16287
> URL: https://issues.apache.org/jira/browse/CAMEL-16287
> Project: Camel
> Issue Type: Bug
>Affects Versions: 2.25.3, 3.8.0
>Reporter: Gyorgy Abraham
>Assignee: Andrea Cosentino
>Priority: Major
> Fix For: 3.11.7, 3.14.3, 3.17.0
>
>
> There is an auto create queue feature for aws-sqs in camel-sqs component.
> Upon camel context creation / route building, it issues an API call to list
> all sqs queues, so it skips the creation for already existing ones. If camel
> context has a for example consumer route for an sqs queue that is not listed
> in the response of this API call, camel will try to to create it.
> However, this API only lists the first 1000 queues in the current account. If
> there are more then 1000 queues, camel-sqs will try to create a new one, and
> might fail. In our company's account, we have 1442 queues in our account and
> Camel tried to create a .fifo queue because it thought it wasnt already
> existing, resulting in application startup error.
> Relevant code:
> [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174]
> Relevant API docs at AWS:
> [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-]
> Solution is straightforward: use pagination and load all queues.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525430#comment-17525430 ] Andrea Cosentino commented on CAMEL-16287: -- We should start a release in a couple of weeks. Using a cache for checking the SQS queue existing on the account? I don't see a lot of value. If the component restart, the cache will need to be populated again. Persisting the cache for this kind of purpose is overkilling. So caching won't really help. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.11.7, 3.14.3, 3.17.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525260#comment-17525260 ] Stephen Lawson commented on CAMEL-16287: Hi. Perhaps a init cache stored on the client would help. Out of interest when is the next 3.14 LTS build? > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.11.7, 3.14.3, 3.17.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525097#comment-17525097 ] Andrea Cosentino commented on CAMEL-16287: -- For the moment I've added the 1000 max result to the listQueue invocation. I think we should find a better way to check the existence, but it's not straightforward > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.14.3, 3.17.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525038#comment-17525038 ] Andrea Cosentino commented on CAMEL-16287: -- Btw I'll have a look for sure. Thanks for reporting > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.14.3, 3.17.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525037#comment-17525037 ] Andrea Cosentino commented on CAMEL-16287: -- I don't quite getting the problem. The loop is working fine as far as I see. We could set a maxResult parameter for this operation or trying to check the existence of sqs queue with some test ro see if it exist or not. The workaround suggested needs some API invocation. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.14.3, 3.17.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525035#comment-17525035 ] Stephen Lawson commented on CAMEL-16287: Thanks for responding so quickly :). My only real workaround is to use the arn or url like this so that it does the lookup without the search: [aws2-sqs://[queue-name]?queueOwnerAWSAccountId=[account-id]|aws2-sqs://[queue-name]?queueOwnerAWSAccountId=[account-id]] Is it an easy fix to get built into the LTS version? > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.14.3, 3.17.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525022#comment-17525022 ] Claus Ibsen commented on CAMEL-16287: - Thanks Stephen. Ah yeah so the doc hint that if you set max to 1000, then the nextToken will return true if there are more rows, and hence the looping should work > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.14.3, 3.17.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525016#comment-17525016 ] Andrea Cosentino commented on CAMEL-16287: -- I'll have a look > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.14.3, 3.17.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[
https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17525013#comment-17525013
]
Stephen Lawson commented on CAMEL-16287:
[~davsclaus] When using camel-bom 3.14.2 and software.amazon.awssdk 2.17.172.
The pagination added in the latest code will only ever return 1K of results.
This is is because the AWS SDK requires that you pass a max result to trigger
the pagination behaviour. The camel code doesn't set the
{noformat}
// check whether the queue already exists
boolean done = false;
while (!done) {
ListQueuesResponse listQueuesResult =
client.listQueues();{noformat}
Perhaps changing this to:
client.listQueues(ListQueuesRequest.builder().maxResults(1).build())
[AWS SDK
Documentation|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/model/ListQueuesRequest.html#withMaxResults-java.lang.Integer-]
> camel-aws2-sqs should use pagination for deciding which aws sqs queues it
> should create
> ---
>
> Key: CAMEL-16287
> URL: https://issues.apache.org/jira/browse/CAMEL-16287
> Project: Camel
> Issue Type: Bug
>Affects Versions: 2.25.3, 3.8.0
>Reporter: Gyorgy Abraham
>Assignee: Andrea Cosentino
>Priority: Major
> Fix For: 3.10.0
>
>
> There is an auto create queue feature for aws-sqs in camel-sqs component.
> Upon camel context creation / route building, it issues an API call to list
> all sqs queues, so it skips the creation for already existing ones. If camel
> context has a for example consumer route for an sqs queue that is not listed
> in the response of this API call, camel will try to to create it.
> However, this API only lists the first 1000 queues in the current account. If
> there are more then 1000 queues, camel-sqs will try to create a new one, and
> might fail. In our company's account, we have 1442 queues in our account and
> Camel tried to create a .fifo queue because it thought it wasnt already
> existing, resulting in application startup error.
> Relevant code:
> [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174]
> Relevant API docs at AWS:
> [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-]
> Solution is straightforward: use pagination and load all queues.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320971#comment-17320971 ] Gyorgy Abraham commented on CAMEL-16287: Many thanks for the efforts everyone. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.10.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320946#comment-17320946 ] Andrea Cosentino commented on CAMEL-16287: -- Completed now. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.10.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17294363#comment-17294363 ] Andrea Cosentino commented on CAMEL-16287: -- You're right, the information provided wasn't complete and the behavior would be faulty anyway. So thanks for starting the discussion again. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.7.3, 3.9.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17294361#comment-17294361 ] Andrea Cosentino commented on CAMEL-16287: -- I was thinking about it a bit deeper: yes, setting it as false by default and then doing pagination is maybe acceptable in terms of startup performance. I'll work on this approach. If you have time to provide a PR, I'll gladly review it and merge it. If you don't have time, I'll take care of it. Let me re-open this. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.7.3, 3.9.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17294358#comment-17294358 ] Gyorgy Abraham commented on CAMEL-16287: I am pretty sad that you keep the functionality faulty. Anyways better to have auto create set to false, as you wrote. And I think the documentation should clearly state that if you enable it, and have 1000+ queues, there is this limitation. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.7.3, 3.9.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17293857#comment-17293857 ] Andrea Cosentino commented on CAMEL-16287: -- We already have other issues related to pagination. The autocreate queue option is true by default, but for historical reasons. I don't think introducing pagination is a good idea, because it means calling multiple time the API for listQueue with next token. So it will slow down the startup. My suggestion is to set the autocreateQueue to false. We can also switch all the autocreation features of AWS2-* components to false by default [~davsclaus] > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.7.3, 3.9.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17293859#comment-17293859 ] Andrea Cosentino commented on CAMEL-16287: -- We can close this as information provided. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.7.3, 3.9.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17293843#comment-17293843 ] Claus Ibsen commented on CAMEL-16287: - Are you able to work on a fix. And what happens if there are a lot of queus, will this slow down? Maybe you want to turn this off, because the queue already exsits (eg you dont want Camel to auto create queues as its some devops/admin job to do) > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.7.3, 3.9.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17293653#comment-17293653 ] Gyorgy Abraham commented on CAMEL-16287: Sorry for the confusion. It should be aws2-sqs. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.7.3, 3.9.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17293659#comment-17293659 ] Andrea Cosentino commented on CAMEL-16287: -- Ok, sounds good, thanks for the clarification. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.7.3, 3.9.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (CAMEL-16287) camel-aws2-sqs should use pagination for deciding which aws sqs queues it should create
[ https://issues.apache.org/jira/browse/CAMEL-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17293650#comment-17293650 ] Andrea Cosentino commented on CAMEL-16287: -- It's not clear if you are using camel-aws-sqs or aws2-sqs. From the description it seems version 2, from the title it seems version 1. By the way camel-aws-sqs has been removed and in 3.9.0 there will be only aws2-sqs, so I'll apply the logic to it. > camel-aws2-sqs should use pagination for deciding which aws sqs queues it > should create > --- > > Key: CAMEL-16287 > URL: https://issues.apache.org/jira/browse/CAMEL-16287 > Project: Camel > Issue Type: Bug >Affects Versions: 2.25.3, 3.8.0 >Reporter: Gyorgy Abraham >Assignee: Andrea Cosentino >Priority: Major > Fix For: 3.7.3, 3.9.0 > > > There is an auto create queue feature for aws-sqs in camel-sqs component. > Upon camel context creation / route building, it issues an API call to list > all sqs queues, so it skips the creation for already existing ones. If camel > context has a for example consumer route for an sqs queue that is not listed > in the response of this API call, camel will try to to create it. > However, this API only lists the first 1000 queues in the current account. If > there are more then 1000 queues, camel-sqs will try to create a new one, and > might fail. In our company's account, we have 1442 queues in our account and > Camel tried to create a .fifo queue because it thought it wasnt already > existing, resulting in application startup error. > Relevant code: > [https://github.com/apache/camel/blob/master/components/camel-aws2-sqs/src/main/java/org/apache/camel/component/aws2/sqs/Sqs2Endpoint.java#L174] > Relevant API docs at AWS: > [https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQS.html#listQueues-com.amazonaws.services.sqs.model.ListQueuesRequest-] > Solution is straightforward: use pagination and load all queues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
