klam-shop opened a new pull request, #26159:
URL: https://github.com/apache/flink/pull/26159
…
<!--
- Fill out the template below to describe the changes contributed by the
pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean
verify` passes. You can set up Azure Pipelines CI to do that following [this
guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from
multiple issues.
- Each commit in the pull request has a meaningful commit message
(including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and
this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
Add a Format Option to the Debezium Format to optionally skip emitting the
UPDATE_BEFORE Rows when deserializing a Debezium message with op='u'.
This is helpful for Flink SQL applications that want to operate in UPSERT
(ChangelogMode=[I,UA,D]) mode and save on processing the UPDATE_BEFORE Rows
since the downstream sinks can handle it.
Note no changes are required for the encoder since Flink [encodes
UPDATE_BEFORE and UPDATE_AFTER as DELETE and INSERT Debezium
messages.](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/table/formats/debezium/)
## Brief change log
- Add `ENABLE_UPSERT_MODE` ConfigOption to `DebeziumAvroFormatFactory`
- Update `DebeziumAvroDeserializationSchema` to handle the new option
properly and skip emitting UPDATE_BEFORE when the option is enabled
- Add test coverage to `DebeziumAvroSerDeSchemaTest`
## Verifying this change
This change added tests and can be verified as follows:
- Added `testUpdateDataDeserializationWithUpsertMode` that deserializes an
update and checks no -U is emitted when upsert mode is enabled.
- Added `testSeDeSchemaWithUpsertMode` to test the format factory flag
functionality, and changelog mode is correctly set.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): /no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
