riyarawat-amazon opened a new pull request, #253:
URL: https://github.com/apache/flink-connector-aws/pull/253
## Purpose of the change
When the DynamoDB Table API sink consumes a changelog (CDC) stream containing
DELETE records, it fails with:
```
DynamoDbException: The provided key element does not match the schema
```
because the sink sends the whole item as the delete key instead of only the
primary key. This PR builds the DELETE key from the declared `PRIMARY KEY`
(partition key and optional sort key), leaving PUT (INSERT/UPDATE_AFTER)
unchanged.
## Verifying this change
This change adds unit tests and was verified end to end:
- `RowDataToAttributeValueConverterTest` — `testDeleteOnlyPrimaryKey`,
`testDeleteOnlyPrimaryKeys`, `testPKIgnoredForInsert`,
`testPKIgnoredForUpdateAfter`.
- `RowDataElementConverterTest` — DELETE includes only the primary key,
DELETE
without a primary key throws a clear error, and INSERT/UPDATE_AFTER still
write the full item.
- `DynamoDbDynamicSinkFactoryTest` — primary key is read from the schema,
composite key order is preserved, `PARTITIONED BY` defaults to the primary
key
when unset, a primary key with more than two columns is rejected, and
mismatched `PARTITIONED BY` / `PRIMARY KEY` is rejected.
- Manually verified against a real DynamoDB table: a CDC stream of 4 inserts
+ 1
delete results in the deleted row being removed. The same job fails with
the
original error on unpatched `main`.
## Significant changes
- *(Bug fix)* DELETE requests now carry only the primary key.
- `PRIMARY KEY` is now required to process CDC DELETE records; a clear
`TableException` is thrown when it is missing. This is not a breaking
change,
as this scenario previously failed outright.
- When `PARTITIONED BY` is not specified it defaults to the `PRIMARY KEY`,
so CDC
deletes work without an explicit `PARTITIONED BY` clause. When both are
specified they must reference the same columns.
- Documentation updated for the `PRIMARY KEY` / CDC delete behavior.
Based on the original work in #152 by @robg-eb. The broader `PARTITIONED BY`
deprecation discussed there is intentionally left out of scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
