Claus Ibsen created CAMEL-27179:
-----------------------------------

             Summary: camel-aws2-timestream - query has no next token header, 
results cannot be paged
                 Key: CAMEL-27179
                 URL: https://issues.apache.org/jira/browse/CAMEL-27179
             Project: Camel
          Issue Type: Improvement
          Components: camel-aws2
            Reporter: Claus Ibsen


h3. Problem
The {{query}} operation of {{aws2-timestream}} (without {{pojoRequest}}) builds 
the {{QueryRequest}} from two headers only: {{CamelAwsTimestreamQueryString}} 
and {{CamelAwsTimestreamQueryMaxRows}}. Timestream returns a result in pages, 
with a {{nextToken}} for the next page, but there is no header to send a next 
token, and none is set from the response. So only the first page of a result 
can be read. Setting {{CamelAwsTimestreamQueryMaxRows}} limits the page size, 
it does not give paging. The endpoint documentation already says that "the next 
page needs a QueryRequest with its nextToken and pojoRequest=true", that is 
Java code.

The Kamelet {{aws-timestream-query-sink}} cannot offer paging because of this.

h3. Where
{{components/camel-aws/camel-aws2-timestream/src/main/java/org/apache/camel/component/aws2/timestream/query/Timestream2QueryProducer.java}},
 {{query(...)}}: the {{else}} branch reads only {{QUERY_STRING}} and 
{{QUERY_MAX_ROWS}}, and the body is set to the {{QueryResponse}} without any 
header from it.

h3. Suggested change
One or both of:
* A header {{CamelAwsTimestreamQueryNextToken}}: read it into 
{{QueryRequest.Builder.nextToken(...)}}, and set it on the result from 
{{QueryResponse.nextToken()}} (absent on the last page), so that a route can 
loop over the pages, e.g. with {{loop doWhile}}.
* An endpoint option (e.g. {{queryAllPages}}, default {{false}}) that follows 
the pages with {{timestreamQueryClient.queryPaginator(...)}} and gives all the 
rows in one result.
Document the new header and option in {{Timestream2Constants}} and the 
component docs.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to