suryadanny opened a new pull request, #19383: URL: https://github.com/apache/hudi/pull/19383
Tracking issue: apache/hudi#19382 ### Change Logs Kinesis has no resource-based policy, so reading a stream in a **different AWS account** than the application requires IAM role assumption (`sts:AssumeRole`). This adds an optional config `hoodie.streamer.source.kinesis.role.arn`: - **`KinesisSourceConfig`** — new `KINESIS_ROLE_ARN` key (no default). Naming aligns with `HoodieAWSConfig.hoodie.aws.role.arn`. - **`KinesisOffsetGen.createKinesisClient`** — when the ARN is set (and no static keys), attach an auto-refreshing `StsAssumeRoleCredentialsProvider` (AWS SDK v2). The base STS client uses the default credential chain (must be granted `sts:AssumeRole` on the ARN). Static access/secret keys still take precedence; empty/absent ARN preserves the existing same-account behavior. No external ID is used. - Provider is **cached per `region|roleArn`** so a long-lived streaming executor reuses one `StsClient` per distinct role instead of leaking one per `mapPartitions` task (AWS SDK v2 does not close a user-supplied credentials provider when the `KinesisClient` closes). - **`KinesisReadConfig`** — carry the ARN so executors rebuild the client with it (client is built inside `mapPartitions`, not on the driver). - **`JsonKinesisSource`** — thread the ARN through on both driver and executor paths. - **`hudi-utilities/pom.xml`** — add `software.amazon.awssdk:sts` (`StsAssumeRoleCredentialsProvider` is not otherwise on the classpath; `kinesis` doesn't pull it transitively). New unit tests (no Spark harness): the config-key contract, ARN round-trip + `null` preservation, ARN surviving Java serialization to executors, and the credential-provider branch selection (assume-role vs default-chain vs static-keys-take-precedence). ### Impact Opt-in; no behavior change when the config is unset. Enables reading cross-account Kinesis streams. ### Risk level (write none, low medium or high below) low — additive, gated behind a new no-default config; existing same-account path unchanged. ### Documentation Update Config is self-documenting via `withDocumentation`; will surface in the generated configuration docs. No separate doc page needed. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Change Logs and Impact described above - [x] Necessary unit tests added 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
