YuriGusev opened a new pull request #17407:
URL: https://github.com/apache/flink/pull/17407
## What is the purpose of the change
This pull requests adds a new DynamoDB connector (sink) that can write to
DynamoDB in batch mode. There is a plan to add non-batch mode later (to support
update operations).
## Brief change log
A new module - flink-connectors --> flink-connector-dynamodb and end to end
tests module. Connector only supports batch mode at the moment.
## Verifying this change
This change added tests and verified as follows:
* *End to end tests against DynamoDB (local container)*
* *Unit and integration tests*
* *Manually verified the change by running:*
-- reading from s3 and writing to DynamoDB with low capacity (back
pressure check),
-- reading from s3 and writing to DynamoDB when DynamoDB throttles and
scales (backoff strategy check)
-- long running job for 4 hours (to check if it is stable)
-- writing a 4GB dataset and verifying no messages lost (by checking
number of records in DynamoDB)
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no (only new classes annotated with @Public)
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? JavaDocs, no other documentation
yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
