shibd opened a new pull request, #21004:
URL: https://github.com/apache/pulsar/pull/21004
### Motivation
Improve kinesis connector some config.
### Modifications
#### For the kinesis sink connector
1. Add `cloudwatchEndpoint` config, Because the AWS Kinesis producer will
periodically send log metrics to CloudWatch.
The current user can set `awsRegion` or `awsEndpoint`, but if the user just
sets `awsEndpoint`, he will encounter an error:
```
2023-08-08T21:16:36,307+0800 [kpl-daemon-0003] ERROR
com.amazonaws.services.kinesis.producer.LogInputStreamReader - [2023-08-08
21:16:36.302919] [0x00007725][0x0000000201a76280] [error] [main.cc:268] Could
not configure the region. It was not given in the config and we were unable to
retrieve it from EC2 metadata.
2023-08-08T21:16:37,316+0800 [kpl-daemon-0000] ERROR
com.amazonaws.services.kinesis.producer.KinesisProducer - Error in child process
com.amazonaws.services.kinesis.producer.IrrecoverableError: Child process
exited with code 1
at
com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:537)
~[amazon-kinesis-producer-0.14.0.jar:?]
at
com.amazonaws.services.kinesis.producer.Daemon.fatalError(Daemon.java:509)
~[amazon-kinesis-producer-0.14.0.jar:?]
```
#### For the kinesis source connector
1. Change `applicationName` default value is `pulsar-kinesis`, If you use
the original default values(`Apache Pulsar IO Connector`), you will encounter
an error:
```
Caused by: software.amazon.awssdk.services.dynamodb.model.DynamoDbException:
1 validation error detected: Value 'Apache Pulsar IO Connector' at 'tableName'
failed to satisfy constraint: Member must satisfy regular expression pattern:
[a-zA-Z0-9_.-]+ (Service: DynamoDb, Status Code: 400, Request ID:
FPG9PKE3V9TMEPUFBFFF7PGFEBVV4KQNSO5AEMVJF66Q9ASUAAJG)
at
software.amazon.awssdk.services.dynamodb.model.DynamoDbException$BuilderImpl.build(DynamoDbException.java:95)
~[dynamodb-2.10.56.jar:?]
at
software.amazon.awssdk.services.dynamodb.model.DynamoDbException$BuilderImpl.build(DynamoDbException.java:55)
~[dynamodb-2.10.56.jar:?]
```
#### For some refactor code:
1. Add a `KinesisSinkConfig.load()` method, and move the param verify logic
to it.
2. Add a `KinesisSourceConfig.load()` method, and move the param verify
logic to it.
3. Remove duplicate validation logic and useless tests
4. Remove `AwsCredentialProviderPlugin`, `AwsDefaultProviderChainPlugin` and
`STSAssumeRoleProviderPlugin` class, These classes have long been deprecated,
and there is no place to use them.
### Verifying this change
- Add `missCloudWatchEndpointTest` to cover `awsRegion` and
`cloudwatchEndpoint` is empty.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [x] `doc-required` <!-- Your PR changes impact docs and you will update
later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
- https://github.com/shibd/pulsar/pull/25
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
