[
https://issues.apache.org/jira/browse/CAMEL-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17116011#comment-17116011
]
Jon Chase commented on CAMEL-14992:
-----------------------------------
I've changed the URI to accept the operation as a parameter instead of part of
the path.
I've also added a few additional endpoints: getQueryExecution, getQueryResults,
listQueryExecutions, startQueryExecution,
I can add support for more, but those are the interesting ones to me. :) I
will likely round out the query-centric endpoints (e.g. StopQueryExecution,
etc.). If there are specific requests for other endpoints, I'll happily add
them now, but otherwise I'm not planning on writing support for e.g
"UntagResource", "UpdateWorkGroup", etc (I don't use these myself). Either
way, it'd be easy to add support for those when desired.
I've tried to make `getQueryResults` similar to the JDBC component in that it
supports different return types:
`outputType=[StreamList,SelectList,S3Pointer]`:
{code:java}
/**
* Represents various ways to return query results from Athena. For example,
choose between a streaming iterator
* that will yield all results, a static list of rows, or a pointer to the
results in S3.
*/
public enum Athena2OutputType {
/**
* When using an endpoint that returns rows directly back to the caller, such
as {@code getQueryResults}, use
* AWS 2 Athena {@link GetQueryResultsIterable} to return a streaming list of
results. Returning a streaming
* result means that no API requests happen until the streaming result is
accessed.
*
* <p>This is the type to use if you need to process large result sets in
memory (as opposed to, e.g. an EMR job),
* as the iterator returned using this method will stream results from AWS a
page at a time, thus limiting the amount
* of memory consumed at any one point.
*/
StreamList,
/**
* Return a static list of rows. The amount of rows returned is limited to
the max response size of Athena's
* {@code GetQueryResults} (currently 1,000).
*
* @see <a
href="https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryResults.html";>GetQueryResults</a>
*/
SelectList,
/**
* Return the path to the results in S3. This may be preferred if you want
to pass the pointer to the results to
* another process for handling.
*/
S3Pointer
}
{code}
I've got some clean up to do, after which I'll put up a PR. I expect it to
take this week to find the time for the clean up.
Here's what the routes from the integration test look like now:
{code:java}
@Override
protected RouteBuilder createRouteBuilder() throws Exception {
return new RouteBuilder() {
@Override
public void configure() throws Exception {
// String s = "&accessKey=xxx&secretKey=yyy®ion=eu-west-1";
String s = "amazonAthenaClient=#athenaClient";
from("direct:athenaExecuteQueryTest")
.setBody(constant(
"SELECT * FROM ("
+ " VALUES"
+ " (1, 'a'),"
+ " (2, 'b')"
+ ") AS t (id, name)"))
.to("aws2-athena://label?operation=executeQuery&outputLocation=" +
s3Bucket + "&" + s);
from("direct:athenaExecuteQueryHandlesPermFailTest")
.setBody(constant(
"SELECT INVALID SQL"))
.to("aws2-athena://label?operation=executeQuery&outputLocation=" +
s3Bucket + "&" + s);
from("direct:athenaGetQueryExecutionTest")
.setBody(constant("SELECT 'foo'"))
.to("aws2-athena://label?operation=startQueryExecution&outputLocation=" +
s3Bucket + "&" + s)
.to("aws2-athena://label?operation=getQueryExecution&" + s);
from("direct:athenaGetQueryResultsAsStreamListTest")
.setBody(constant(
"SELECT * FROM ("
+ " VALUES"
+ " (1, 'a'),"
+ " (2, 'b')"
+ ") AS t (id, name)"))
.to("aws2-athena://label?operation=startQueryExecution&outputLocation=" +
s3Bucket + "&" + s)
.to("aws2-athena://label?operation=getQueryResults&outputType=StreamList&" + s);
from("direct:athenaGetQueryResultsAsStaticListTest")
.setBody(constant(
"SELECT * FROM ("
+ " VALUES"
+ " (1, 'a'),"
+ " (2, 'b')"
+ ") AS t (id, name)"))
.to("aws2-athena://label?operation=startQueryExecution&outputLocation=" +
s3Bucket + "&" + s)
.to("aws2-athena://label?operation=getQueryResults&outputType=SelectList&" + s);
from("direct:athenaGetQueryResultsAsS3PointerTest")
.setBody(constant("SELECT 1 as foo"))
.to("aws2-athena://label?operation=startQueryExecution&outputLocation=" +
s3Bucket + "&" + s)
.to("aws2-athena://label?operation=getQueryResults&outputType=S3Pointer&" + s);
from("direct:athenaListQueryExecutionsTest")
.setBody(constant("SELECT 1 AS foo"))
.to("aws2-athena://label?operation=startQueryExecution&outputLocation=" +
s3Bucket + "&" + s)
.log("ensured there will be a query execution to return from the
following listQueryExecutions call")
.to("aws2-athena://label?operation=listQueryExecutions&" + s);
from("direct:athenaStartQueryExecutionTest")
.setBody(constant("SELECT 1 AS foo"))
.to("aws2-athena://label?operation=startQueryExecution&outputLocation=" +
s3Bucket + "&" + s);
from("direct:athenaStartQueryExecutionAndGetQueryExecutionUsingHeaders")
.setBody(constant("SELECT 1"))
.to("aws2-athena://label?operation=startQueryExecution&outputLocation=" +
s3Bucket + "&" + s)
.to("aws2-athena://label?operation=getQueryExecution&" + s);
}
};
}
{code}
> Create a Camel AWS2 Athena component
> ------------------------------------
>
> Key: CAMEL-14992
> URL: https://issues.apache.org/jira/browse/CAMEL-14992
> Project: Camel
> Issue Type: New Feature
> Components: camel-aws2
> Reporter: Andrea Cosentino
> Assignee: Andrea Cosentino
> Priority: Major
> Fix For: 3.x
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
