dannycranmer commented on code in PR #668: URL: https://github.com/apache/flink-web/pull/668#discussion_r1282887604
########## docs/content/posts/2023-08-03-connector-updates.md: ########## @@ -0,0 +1,68 @@ +--- +title: "DynamoDB, MongoDB and OpenSearch externalized connectors" Review Comment: ```suggestion title: "Announcing three new Apache Flink connectors, the new connector versioning strategy and externalization" ``` ########## docs/content/posts/2023-08-03-connector-updates.md: ########## @@ -0,0 +1,68 @@ +--- +title: "DynamoDB, MongoDB and OpenSearch externalized connectors" +date: "2023-08-03T12:00:00Z" +authors: +- elphastori: + name: "Elphas Toringepi" + twitter: "elphastori" +aliases: +- /news/2023/08/03/externalized-connectors.html +--- + +## New connectors + +We're excited to announce that Apache Flink now includes 3 new connectors for [DynamoDB](https://aws.amazon.com/dynamodb), [MongoDB](https://www.mongodb.com/) and [OpenSearch](https://opensearch.org/)! The connectors are available for both the [DataStream](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/) and [Table/SQL](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/overview/) APIs. Review Comment: ```suggestion We're excited to announce that Apache Flink now supports 3 new connectors: [Amazon DynamoDB](https://aws.amazon.com/dynamodb), [MongoDB](https://www.mongodb.com/) and [OpenSearch](https://opensearch.org/)! The connectors are available for both the [DataStream](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/) and [Table/SQL](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/overview/) APIs. ``` ########## docs/content/posts/2023-08-03-connector-updates.md: ########## @@ -0,0 +1,68 @@ +--- +title: "DynamoDB, MongoDB and OpenSearch externalized connectors" +date: "2023-08-03T12:00:00Z" +authors: +- elphastori: + name: "Elphas Toringepi" + twitter: "elphastori" +aliases: +- /news/2023/08/03/externalized-connectors.html +--- + +## New connectors + +We're excited to announce that Apache Flink now includes 3 new connectors for [DynamoDB](https://aws.amazon.com/dynamodb), [MongoDB](https://www.mongodb.com/) and [OpenSearch](https://opensearch.org/)! The connectors are available for both the [DataStream](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/) and [Table/SQL](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/overview/) APIs. + +- **[DynamoDB sink](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/dynamodb/)** - The DynamoDB sink writes to [Amazon DynamoDB](https://aws.amazon.com/dynamodb) using the [AWS v2 SDK for](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html) Java. Review Comment: ```suggestion - **[Amazon DynamoDB](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/dynamodb/)** - This connector includes a sink that provides at-least-once delivery guarantees. ``` ########## docs/content/posts/2023-08-03-connector-updates.md: ########## @@ -0,0 +1,68 @@ +--- +title: "DynamoDB, MongoDB and OpenSearch externalized connectors" +date: "2023-08-03T12:00:00Z" +authors: +- elphastori: + name: "Elphas Toringepi" + twitter: "elphastori" +aliases: +- /news/2023/08/03/externalized-connectors.html +--- + +## New connectors + +We're excited to announce that Apache Flink now includes 3 new connectors for [DynamoDB](https://aws.amazon.com/dynamodb), [MongoDB](https://www.mongodb.com/) and [OpenSearch](https://opensearch.org/)! The connectors are available for both the [DataStream](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/) and [Table/SQL](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/overview/) APIs. + +- **[DynamoDB sink](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/dynamodb/)** - The DynamoDB sink writes to [Amazon DynamoDB](https://aws.amazon.com/dynamodb) using the [AWS v2 SDK for](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html) Java. +- **[MongoDB connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/mongodb/)** - Flink now provides a [MongoDB](https://www.mongodb.com/) connector for reading and writing data from and to MongoDB collections with at-least-once guarantees. +- **[OpenSearch sink](https://github.com/apache/flink-connector-opensearch/blob/main/docs/content/docs/connectors/datastream/opensearch.md)** - This connector provides a sink that can request document actions to an [OpenSearch](https://opensearch.org/) Index. + +|Connector|Date Released|Supported Flink Versions| +|---|---|---| +|DynamoDB sink|12/2/2022|1.15+| Review Comment: ```suggestion |Amazon DynamoDB sink|12/2/2022|1.15+| ``` ########## docs/content/posts/2023-08-03-connector-updates.md: ########## @@ -0,0 +1,68 @@ +--- +title: "DynamoDB, MongoDB and OpenSearch externalized connectors" +date: "2023-08-03T12:00:00Z" +authors: +- elphastori: + name: "Elphas Toringepi" + twitter: "elphastori" +aliases: +- /news/2023/08/03/externalized-connectors.html +--- + +## New connectors + +We're excited to announce that Apache Flink now includes 3 new connectors for [DynamoDB](https://aws.amazon.com/dynamodb), [MongoDB](https://www.mongodb.com/) and [OpenSearch](https://opensearch.org/)! The connectors are available for both the [DataStream](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/) and [Table/SQL](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/overview/) APIs. + +- **[DynamoDB sink](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/dynamodb/)** - The DynamoDB sink writes to [Amazon DynamoDB](https://aws.amazon.com/dynamodb) using the [AWS v2 SDK for](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html) Java. +- **[MongoDB connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/mongodb/)** - Flink now provides a [MongoDB](https://www.mongodb.com/) connector for reading and writing data from and to MongoDB collections with at-least-once guarantees. +- **[OpenSearch sink](https://github.com/apache/flink-connector-opensearch/blob/main/docs/content/docs/connectors/datastream/opensearch.md)** - This connector provides a sink that can request document actions to an [OpenSearch](https://opensearch.org/) Index. Review Comment: ```suggestion - **[OpenSearch sink](https://github.com/apache/flink-connector-opensearch/blob/main/docs/content/docs/connectors/datastream/opensearch.md)** - This connector includes a sink that provides at-least-once guarantees ``` ########## docs/content/posts/2023-08-03-connector-updates.md: ########## @@ -0,0 +1,68 @@ +--- +title: "DynamoDB, MongoDB and OpenSearch externalized connectors" Review Comment: I am not 100% on this, what do you think? ########## docs/content/posts/2023-08-03-connector-updates.md: ########## @@ -0,0 +1,68 @@ +--- +title: "DynamoDB, MongoDB and OpenSearch externalized connectors" +date: "2023-08-03T12:00:00Z" +authors: +- elphastori: + name: "Elphas Toringepi" + twitter: "elphastori" +aliases: +- /news/2023/08/03/externalized-connectors.html +--- + +## New connectors + +We're excited to announce that Apache Flink now includes 3 new connectors for [DynamoDB](https://aws.amazon.com/dynamodb), [MongoDB](https://www.mongodb.com/) and [OpenSearch](https://opensearch.org/)! The connectors are available for both the [DataStream](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/) and [Table/SQL](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/overview/) APIs. + +- **[DynamoDB sink](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/dynamodb/)** - The DynamoDB sink writes to [Amazon DynamoDB](https://aws.amazon.com/dynamodb) using the [AWS v2 SDK for](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html) Java. +- **[MongoDB connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/mongodb/)** - Flink now provides a [MongoDB](https://www.mongodb.com/) connector for reading and writing data from and to MongoDB collections with at-least-once guarantees. Review Comment: ```suggestion - **[MongoDB connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/mongodb/)** - This connector includes a source and sink that provide at-least-once guarantees. ``` ########## docs/content/posts/2023-08-03-connector-updates.md: ########## @@ -0,0 +1,68 @@ +--- +title: "DynamoDB, MongoDB and OpenSearch externalized connectors" +date: "2023-08-03T12:00:00Z" +authors: +- elphastori: + name: "Elphas Toringepi" + twitter: "elphastori" +aliases: +- /news/2023/08/03/externalized-connectors.html +--- + +## New connectors + +We're excited to announce that Apache Flink now includes 3 new connectors for [DynamoDB](https://aws.amazon.com/dynamodb), [MongoDB](https://www.mongodb.com/) and [OpenSearch](https://opensearch.org/)! The connectors are available for both the [DataStream](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/) and [Table/SQL](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/overview/) APIs. + +- **[DynamoDB sink](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/dynamodb/)** - The DynamoDB sink writes to [Amazon DynamoDB](https://aws.amazon.com/dynamodb) using the [AWS v2 SDK for](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html) Java. +- **[MongoDB connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/mongodb/)** - Flink now provides a [MongoDB](https://www.mongodb.com/) connector for reading and writing data from and to MongoDB collections with at-least-once guarantees. +- **[OpenSearch sink](https://github.com/apache/flink-connector-opensearch/blob/main/docs/content/docs/connectors/datastream/opensearch.md)** - This connector provides a sink that can request document actions to an [OpenSearch](https://opensearch.org/) Index. + +|Connector|Date Released|Supported Flink Versions| +|---|---|---| +|DynamoDB sink|12/2/2022|1.15+| +|MongoDB connector|3/31/2023|1.16+| +|OpenSearch sink|12/21/2022|1.16+| + +## Externalized connectors + +In order to improve the overall connector ecosystem, we introduced the new connectors in separate repositories and moved existing connectors out of [Flink’s main repository](https://github.com/apache/flink). + +This decouples the connectors' release cycle from Flink’s release cycle, resulting in faster releases of connectors. It also allows the addition of newer connector features to older Flink versions and facilitates more contributions to connectors, among numerous other benefits Review Comment: ```suggestion The community have externalized connectors from [Flink’s main repository](https://github.com/apache/flink). This was driven to realise the following benefits: - **Faster releases of connectors:** New features can be added more quickly, bugs can be fixed immediately, and we can have faster security patches in case of direct or indirect (through dependencies) security flaws. - **Adding newer connector features to older Flink versions:** By having stable connector APIs, the same connector artifact may be used with different Flink versions. Thus, new features can also immediately be used with older Flink versions. - **More activity and contributions around connectors:** By easing the contribution and development process around connectors, we will see faster development and also more connectors. - **Documentation:** Standardized documentation and user experience for the connectors, regardless of where they are maintained. - **A faster Flink CI:** By not needing to build and test connectors, the Flink CI pipeline will be faster and Flink developers will experience fewer build stabilities (which mostly come from connectors). That should speed up Flink development. ``` ########## docs/content/posts/2023-08-03-connector-updates.md: ########## @@ -0,0 +1,68 @@ +--- +title: "DynamoDB, MongoDB and OpenSearch externalized connectors" +date: "2023-08-03T12:00:00Z" +authors: +- elphastori: + name: "Elphas Toringepi" + twitter: "elphastori" +aliases: +- /news/2023/08/03/externalized-connectors.html +--- + +## New connectors + +We're excited to announce that Apache Flink now includes 3 new connectors for [DynamoDB](https://aws.amazon.com/dynamodb), [MongoDB](https://www.mongodb.com/) and [OpenSearch](https://opensearch.org/)! The connectors are available for both the [DataStream](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/) and [Table/SQL](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/overview/) APIs. + +- **[DynamoDB sink](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/dynamodb/)** - The DynamoDB sink writes to [Amazon DynamoDB](https://aws.amazon.com/dynamodb) using the [AWS v2 SDK for](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html) Java. Review Comment: Let's reduce links vendor specific web pages ########## docs/content/posts/2023-08-03-connector-updates.md: ########## @@ -0,0 +1,68 @@ +--- +title: "DynamoDB, MongoDB and OpenSearch externalized connectors" +date: "2023-08-03T12:00:00Z" +authors: +- elphastori: + name: "Elphas Toringepi" + twitter: "elphastori" +aliases: +- /news/2023/08/03/externalized-connectors.html +--- + +## New connectors + +We're excited to announce that Apache Flink now includes 3 new connectors for [DynamoDB](https://aws.amazon.com/dynamodb), [MongoDB](https://www.mongodb.com/) and [OpenSearch](https://opensearch.org/)! The connectors are available for both the [DataStream](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/) and [Table/SQL](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/overview/) APIs. + +- **[DynamoDB sink](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/dynamodb/)** - The DynamoDB sink writes to [Amazon DynamoDB](https://aws.amazon.com/dynamodb) using the [AWS v2 SDK for](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html) Java. +- **[MongoDB connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/mongodb/)** - Flink now provides a [MongoDB](https://www.mongodb.com/) connector for reading and writing data from and to MongoDB collections with at-least-once guarantees. +- **[OpenSearch sink](https://github.com/apache/flink-connector-opensearch/blob/main/docs/content/docs/connectors/datastream/opensearch.md)** - This connector provides a sink that can request document actions to an [OpenSearch](https://opensearch.org/) Index. + +|Connector|Date Released|Supported Flink Versions| +|---|---|---| +|DynamoDB sink|12/2/2022|1.15+| +|MongoDB connector|3/31/2023|1.16+| +|OpenSearch sink|12/21/2022|1.16+| + +## Externalized connectors + +In order to improve the overall connector ecosystem, we introduced the new connectors in separate repositories and moved existing connectors out of [Flink’s main repository](https://github.com/apache/flink). + +This decouples the connectors' release cycle from Flink’s release cycle, resulting in faster releases of connectors. It also allows the addition of newer connector features to older Flink versions and facilitates more contributions to connectors, among numerous other benefits + +The following connectors have been moved to individual repositories: + +- [Kafka / Upsert-Kafka](https://github.com/apache/flink-connector-kafka) +- [Cassandra](https://github.com/apache/flink-connector-cassandra) +- [Elasticsearch](https://github.com/apache/flink-connector-elasticsearch/) +- [**MongoDB**](https://github.com/apache/flink-connector-mongodb) +- **[OpenSearch](https://github.com/apache/flink-connector-opensearch)** +- [RabbitMQ](https://github.com/apache/flink-connector-rabbitmq) +- [Google Cloud PubSub](https://github.com/apache/flink-connector-gcp-pubsub) +- [Pulsar](https://github.com/apache/flink-connector-pulsar/) +- [JDBC](https://github.com/apache/flink-connector-jdbc) +- [HBase](https://github.com/apache/flink-connector-hbase) +- [Hive](https://github.com/apache/flink-connector-hive) +- [AWS connectors](https://github.com/apache/flink-connector-aws): + - Firehose + - Kinesis + - **DynamoDB** + +### Versioning + +Connectors continue to use the same maven dependency `groupId` and `artificatId`, however, the JAR artifact `version` has “reset” and now uses the format, `<major>.<minor>.<patch>-<flink-major>.<flink-minor>`. For example, to use the DynamoDB connector for Flink 1.17, add the following dependency to your project: Review Comment: ```suggestion Connectors continue to use the same maven dependency `groupId` and `artificatId`, however, the JAR artifact `version` has changed and now uses the format, `<major>.<minor>.<patch>-<flink-major>.<flink-minor>`. For example, to use the DynamoDB connector for Flink 1.17, add the following dependency to your project: ``` ########## docs/content/posts/2023-08-03-connector-updates.md: ########## @@ -0,0 +1,68 @@ +--- +title: "DynamoDB, MongoDB and OpenSearch externalized connectors" +date: "2023-08-03T12:00:00Z" +authors: +- elphastori: + name: "Elphas Toringepi" + twitter: "elphastori" +aliases: +- /news/2023/08/03/externalized-connectors.html +--- + +## New connectors + +We're excited to announce that Apache Flink now includes 3 new connectors for [DynamoDB](https://aws.amazon.com/dynamodb), [MongoDB](https://www.mongodb.com/) and [OpenSearch](https://opensearch.org/)! The connectors are available for both the [DataStream](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/overview/) and [Table/SQL](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/overview/) APIs. + +- **[DynamoDB sink](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/dynamodb/)** - The DynamoDB sink writes to [Amazon DynamoDB](https://aws.amazon.com/dynamodb) using the [AWS v2 SDK for](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/home.html) Java. +- **[MongoDB connector](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/mongodb/)** - Flink now provides a [MongoDB](https://www.mongodb.com/) connector for reading and writing data from and to MongoDB collections with at-least-once guarantees. +- **[OpenSearch sink](https://github.com/apache/flink-connector-opensearch/blob/main/docs/content/docs/connectors/datastream/opensearch.md)** - This connector provides a sink that can request document actions to an [OpenSearch](https://opensearch.org/) Index. + +|Connector|Date Released|Supported Flink Versions| +|---|---|---| +|DynamoDB sink|12/2/2022|1.15+| +|MongoDB connector|3/31/2023|1.16+| +|OpenSearch sink|12/21/2022|1.16+| + +## Externalized connectors + +In order to improve the overall connector ecosystem, we introduced the new connectors in separate repositories and moved existing connectors out of [Flink’s main repository](https://github.com/apache/flink). + +This decouples the connectors' release cycle from Flink’s release cycle, resulting in faster releases of connectors. It also allows the addition of newer connector features to older Flink versions and facilitates more contributions to connectors, among numerous other benefits Review Comment: ```suggestion The community have externalized connectors from [Flink’s main repository](https://github.com/apache/flink). This was driven to realise the following benefits: - **Faster releases of connectors:** New features can be added more quickly, bugs can be fixed immediately, and we can have faster security patches in case of direct or indirect (through dependencies) security flaws. - **Adding newer connector features to older Flink versions:** By having stable connector APIs, the same connector artifact may be used with different Flink versions. Thus, new features can also immediately be used with older Flink versions. - **More activity and contributions around connectors:** By easing the contribution and development process around connectors, we will see faster development and also more connectors. - **Documentation:** Standardized documentation and user experience for the connectors, regardless of where they are maintained. - **A faster Flink CI:** By not needing to build and test connectors, the Flink CI pipeline will be faster and Flink developers will experience fewer build stabilities (which mostly come from connectors). That should speed up Flink development. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
