0AyanamiRei opened a new pull request, #61324:
URL: https://github.com/apache/doris/pull/61324
### What problem does this PR solve?
**_Overview_**:
This PR adds AWS MSK IAM authentication for Kafka Routine Load in Apache
Doris. You can connect to Amazon MSK using IAM credentials (including Assume
Role and cross-account) with SASL_SSL and OAUTHBEARER.
**_What It Solves_**:
1. Consume AWS MSK data from Doris via Routine Load.
2. Support three credential modes: explicit AK/SK, same-account Instance
Profile Assume Role, and cross-account AK/SK Assume Role.
3. Align with AWS MSK IAM (SigV4-signed OAUTHBEARER tokens).
**_SQL Examples_**
1. MSK IAM with explicit Access Key and Secret Key (same account)
```
CREATE ROUTINE LOAD my_msk_load ON my_db.my_table
COLUMNS (id, name, dt)
PROPERTIES (
"desired_concurrent_number" = "2",
"max_error_number" = "1000"
)
FROM KAFKA
(
"kafka_broker_list" =
"b-1.xxx.kafka.us-east-1.amazonaws.com:9098,b-2.xxx.kafka.us-east-1.amazonaws.com:9098",
"kafka_topic" = "my-topic",
"property.security.protocol" = "SASL_SSL",
"property.sasl.mechanism" = "OAUTHBEARER",
"aws.region" = "us-east-1",
"aws.access.key" = "AKIAIOSFODNN7EXAMPLE",
"aws.secret.key" = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
);
```
2. MSK IAM with Assume Role (e.g. EC2 Instance Profile, same account)
```
CREATE ROUTINE LOAD my_msk_load ON my_db.my_table
COLUMNS (id, name, dt)
PROPERTIES (
"desired_concurrent_number" = "2"
)
FROM KAFKA
(
"kafka_broker_list" = "b-1.xxx.kafka.us-east-1.amazonaws.com:9098",
"kafka_topic" = "my-topic",
"property.security.protocol" = "SASL_SSL",
"property.sasl.mechanism" = "OAUTHBEARER",
"aws.region" = "us-east-1",
"aws.msk.iam.role.arn" =
"arn:aws:iam::123456789012:role/MyMSKConsumerRole"
);
```
3. MSK IAM with cross-account Assume Role (AK/SK of account B to assume role
in account A)
```
CREATE ROUTINE LOAD my_msk_load ON my_db.my_table
COLUMNS (id, name, dt)
PROPERTIES (
"desired_concurrent_number" = "2"
)
FROM KAFKA
(
"kafka_broker_list" = "b-1.xxx.kafka.us-east-1.amazonaws.com:9098",
"kafka_topic" = "my-topic",
"property.security.protocol" = "SASL_SSL",
"property.sasl.mechanism" = "OAUTHBEARER",
"aws.region" = "us-east-1",
"aws.msk.iam.role.arn" =
"arn:aws:iam::111111111111:role/CrossAccountMSKRole",
"aws.access.key" = "AKIAIOSFODNN7EXAMPLE",
"aws.secret.key" = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
);
```
Validation rules (FE): When any aws.* property is set, aws.region is
required, and property.security.protocol must be SASL_SSL and
property.sasl.mechanism must be OAUTHBEARER. If you use explicit credentials,
both aws.access.key and aws.secret.key must be set together.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
