Claus Ibsen created CAMEL-27179:
-----------------------------------
Summary: camel-aws2-timestream - query has no next token header,
results cannot be paged
Key: CAMEL-27179
URL: https://issues.apache.org/jira/browse/CAMEL-27179
Project: Camel
Issue Type: Improvement
Components: camel-aws2
Reporter: Claus Ibsen
h3. Problem
The {{query}} operation of {{aws2-timestream}} (without {{pojoRequest}}) builds
the {{QueryRequest}} from two headers only: {{CamelAwsTimestreamQueryString}}
and {{CamelAwsTimestreamQueryMaxRows}}. Timestream returns a result in pages,
with a {{nextToken}} for the next page, but there is no header to send a next
token, and none is set from the response. So only the first page of a result
can be read. Setting {{CamelAwsTimestreamQueryMaxRows}} limits the page size,
it does not give paging. The endpoint documentation already says that "the next
page needs a QueryRequest with its nextToken and pojoRequest=true", that is
Java code.
The Kamelet {{aws-timestream-query-sink}} cannot offer paging because of this.
h3. Where
{{components/camel-aws/camel-aws2-timestream/src/main/java/org/apache/camel/component/aws2/timestream/query/Timestream2QueryProducer.java}},
{{query(...)}}: the {{else}} branch reads only {{QUERY_STRING}} and
{{QUERY_MAX_ROWS}}, and the body is set to the {{QueryResponse}} without any
header from it.
h3. Suggested change
One or both of:
* A header {{CamelAwsTimestreamQueryNextToken}}: read it into
{{QueryRequest.Builder.nextToken(...)}}, and set it on the result from
{{QueryResponse.nextToken()}} (absent on the last page), so that a route can
loop over the pages, e.g. with {{loop doWhile}}.
* An endpoint option (e.g. {{queryAllPages}}, default {{false}}) that follows
the pages with {{timestreamQueryClient.queryPaginator(...)}} and gives all the
rows in one result.
Document the new header and option in {{Timestream2Constants}} and the
component docs.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)