robg-eb opened a new pull request, #152:
URL: https://github.com/apache/flink-connector-aws/pull/152
## Purpose of the change
When `DynamoDbSink` is used with CDC sources, it fails to process DELETE
records and throws
```
org.apache.flink.connector.dynamodb.shaded.software.amazon.awssdk.services.dynamodb.model.DynamoDbException:
The provided key element does not match the schema
```
This is due to `DynamoDbSinkWriter` passing the whole DynamoDb Item as key
instead of the constructed primary key alone.
## Verifying this change
This change added tests and can be verified as follows:
- Added tests to `RowDataToAttributeValueConverterTest.java`:
- `testDeleteOnlyPrimaryKey` - Ensures that for a `DELETE` request, only
the (single) PK field is included
- `testDeleteOnlyPrimaryKeys` - Ensures that for a `DELETE` request with a
composite PK, both PK fields are included.
- `testPKIgnoredForInsert` - Ensures that PK is ignored when an `INSERT`
request is done, and all fields continue to be included as they have been in
the past.
- `testPKIgnoredForUpdateAfter` - Ensures that PK is ignored when an
`UPDATE_AFTER` request is done, and all fields continue to be included as they
have been in the past.
- Ran manual tests following the steps noted in
https://issues.apache.org/jira/browse/FLINK-35500 under "Steps To Reproduce".
Running the SQL statement as described in Step 6 now properly runs a DELETE in
DynamoDB.
## Significant changes
Previously, the `PRIMARY KEY` field had no significance for a DynamoDB Sink
via Table API. Now, the `PRIMARY KEY` is required when processing a CDC
stream that contains `DELETES`. This is not a 'breaking change' because the
previous behavior for processing a CDC stream containing `DELETES` was already
a failure (`The provided key element does not match the schema`). This change
now provides a clear exception informing users to specify a Primary Key to
avoid that failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
