suryadanny opened a new issue, #19382: URL: https://github.com/apache/hudi/issues/19382
## Problem Amazon Kinesis has no resource-based (resource) policy, so a Hudi ingestion job cannot read a Kinesis stream that lives in a **different AWS account** than the application — the default credential chain (the job's own pod/node/task role) gets `AccessDenied`. The only way to read cross-account is IAM role assumption (`sts:AssumeRole`). Today `JsonKinesisSource` / `KinesisOffsetGen` build the `KinesisClient` with region + the default credential chain (or static access/secret keys for LocalStack-style endpoints), with no way to assume a role in the stream's account. ## Proposed enhancement Add an optional config `hoodie.streamer.source.kinesis.role.arn`. When set, the Kinesis client is built with an auto-refreshing `StsAssumeRoleCredentialsProvider` (AWS SDK v2) for that role; the base STS client uses the default credential chain (which must be granted `sts:AssumeRole` on the ARN). When empty/absent, behavior is unchanged (same-account read via the default chain). Static access/secret keys continue to take precedence. Naming aligns with the existing `HoodieAWSConfig.hoodie.aws.role.arn`. The assume-role provider is cached per `region|roleArn` so a long-lived streaming executor reuses one `StsClient` per distinct role rather than leaking one per `mapPartitions` task. ## Environment - Component: hudi-utilities (DeltaStreamer/Streamer Kinesis source) A PR implementing this is attached. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
