voonhous commented on code in PR #19383:
URL: https://github.com/apache/hudi/pull/19383#discussion_r3802417773
##########
hudi-utilities/pom.xml:
##########
@@ -537,6 +537,12 @@
<artifactId>kinesis</artifactId>
<version>${aws.sdk.version}</version>
</dependency>
+ <!-- STS: assume-role credentials for reading a Kinesis stream in a
different AWS account. -->
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>sts</artifactId>
+ <version>${aws.sdk.version}</version>
+ </dependency>
Review Comment:
Checked the bundles: none ships `software.amazon.awssdk:kinesis` today
(`hudi-utilities-bundle` / `-slim-bundle` contain zero AWS SDK classes;
`hudi-aws-bundle` has `sts`, but relocated under
`org.apache.hudi.software.amazon.awssdk`, so it cannot satisfy the unrelocated
`StsClient` reference in the bundled `KinesisOffsetGen`). So `sts` is the same
pre-existing gap as `kinesis`, not a new one: whoever runs `JsonKinesisSource`
today already supplies the SDK themselves (typically
`software.amazon.awssdk:bundle`, which carries both).
#19425 already adds `software.amazon.awssdk:sts` (plus `aws-query-protocol`
/ `profiles`) to `hudi-utilities-bundle` for exactly this feature, so this PR
should not touch the bundle. Merge order does not matter: `sts` is already on
hudi-utilities' dependency tree via `hudi-aws` (`mvn dependency:tree` shows it
"omitted for duplicate"), so this declaration is direct-use hygiene only.
c1ce6df3c0e842e74fdb2ebfcf2510622bd95a32 fixes the pom comment accordingly; the
PR body's "not otherwise on the classpath" is inaccurate.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KinesisSourceConfig.java:
##########
@@ -76,6 +76,17 @@ public class KinesisSourceConfig extends HoodieConfig {
.withDocumentation("AWS secret key for Kinesis. Used when connecting to
custom endpoints (e.g., LocalStack). "
+ "If not set with endpoint, uses the default AWS credential
chain.");
+ public static final ConfigProperty<String> KINESIS_ROLE_ARN = ConfigProperty
+ .key(PREFIX + "role.arn")
+ .noDefaultValue()
+ .sinceVersion("1.2.0")
Review Comment:
Applied in c1ce6df3c0e842e74fdb2ebfcf2510622bd95a32, which also adds
`hoodie.streamer.source.kinesis.role.external.id` and `role.session.name`
(mirroring `HoodieAWSConfig`; the same gap HUDI-7699 / #11134 closed for the
Glue assume-role provider) and makes the new tests assert the selected
credentials provider via `serviceClientConfiguration().credentialsProvider()`
instead of `assertNotNull(client)`, which passed with the role-arn branch
deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
