hlteoh37 opened a new pull request, #211:
URL: https://github.com/apache/flink-connector-aws/pull/211
## Purpose of the change
Add support for Flink 2.0 for `flink-connector-aws`. This includes:
1. Kinesis Source/Sink
2. Firehose Sink
3. DDB Streams Source
4. DDB Table Sink
5. SQS Sink
Refactoring done to ensure compatibility with Flink 2 can be summarised
below:
1. **Flink public Interface changes.** Sink.InitContext has been renamed to
WriterInitContext. ParameterTools base package has been migrated.
**Impact:** Every Sink connector exposes Sink.InitContext as a public
interface, because this is exposed in the method signature of the abstract
class, AsyncSinkBase. This means a breaking change for the connectors
repository to support Flink 2.x
**Proposal:** I don't see a simple way to decouple the AWS repository code
from being tied to either one of Flink 1.x and Flink 2.x. As such I suggest
that we split the branches, aws-connector-5.x supports flink-1.20.x and
aws-connector-6.x for flink-2.x. This means we have to maintain two active
branches (since 1.20 is LTS), but I don't see a better way around this!
2. **Removal of SinkFunction, SourceFunction interfaces.**
**Impact:** FlinkKinesisConsumer, FlinkKinesisProducer have to be removed.
**Proposal:** Propose that we remove the above, along with the SQL
counterparts, as we have replacements of KinesisStreamsSource and
KinesisStreamsSink as replacements.
**Proposal:** E2E tests and GSR/JSON format tests currently rely on these
interfaces, so they will need refactoring as well.
3. **Removal of support for JDK 8, and JDK 17 is now the recommended Java
version**
**Impact/Proposal:** CI/CD can remove testing for JDK 8, and ensure we test
against JDK 17
4. **Kryo is disabled by default on Flink 2**
**Impact/Proposal:** We should sanity check that none of the new states
falls back to Kryo. IF there are any, we should make it such that we can use
POJOSerializer instead of Kryo.
## Verifying this change
This change is already covered by existing tests, such as existing unit
tests, e2e tests.
Will run manual tests for each Source/Sink against local Flink cluster and
update here with results.
## Significant changes
*(Please check any boxes [x] if the answer is "yes". You can first publish
the PR and check them afterwards, for convenience.)*
- [ ] Dependencies have been added or upgraded
- [ ] Public API has been changed (Public API is any class annotated with
`@Public(Evolving)`)
- [ ] Serializers have been changed
- [ ] New feature has been introduced
- If yes, how is this documented? (not applicable / docs / JavaDocs / not
documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
