davsclaus opened a new pull request, #28893: URL: https://github.com/apache/camel/pull/28893
Fixes [CAMEL-26117](https://issues.apache.org/jira/browse/CAMEL-26117). ### 1. With the ExtractNewRecordState / ExtractNewDocumentState SMT the body and the operation header were null (all six connectors) **What was wrong:** the SMT replaces the change event envelope with the flattened row. `DebeziumEndpoint` only read the `after` and `op` fields of the envelope. The row has neither, so the body and `CamelDebeziumOperation` were `null` and the row was lost. **What changed:** in `DebeziumEndpoint`, a value that is a `Struct` without an `after` field is now the body. Debezium's own events (schema change, heartbeat, transaction metadata) are left out: their schema names start with `io.debezium.`. They keep a `null` body, so a MySQL/SQL Server/Oracle/Db2 schema change event with default settings behaves as before. The operation header is read from the `__op` field (`add.fields=op`) or the `__op` record header (`add.headers=op`) that the SMT adds with its default prefix. Without those options the record has no operation, so the header stays `null`. ### 2. debezium-mongodb: an update had a null body and its changes were lost **What was wrong:** `DebeziumMongodbEndpoint` fell back to the `patch` field when `after` was null. Debezium 2 removed `patch`, so this code never ran. With `captureMode=change_streams` (no full document lookup) an update had a null body, and the `updateDescription` of the event was not exposed. **What changed:** the `patch` fallback is removed. The `updateDescription` is set as a `Map` (`updatedFields`, `removedFields`, `truncatedArrays`) in a new `CamelDebeziumUpdateDescription` header. The header uses `applicableFor = "debezium-mongodb"`, so only the mongodb catalog lists it. The body stays the document JSON String, or `null` when there is none, as the body type in the catalog already says. A header and not the body because the body would otherwise sometimes be a String and sometimes a Struct. ### 3. DebeziumTypeConverter.toMap The javadoc said nested structs were not handled, but they were. The javadoc is fixed. A `Struct` inside a list (array field) or as a map value is now also converted to a `Map`. ### Tests - `DebeziumEndpointTest`: flattened row as body, operation from the `__op` field, operation from the `__op` header, and a schema change event that keeps a null body. `testIfHandlesUnknownSchema` asserted the null body for a Struct without `after`, which is this bug, so it now asserts the row is the body. - `DebeziumTypeConverterTest`: Structs in a list and a map. - New `DebeziumMongodbEndpointTest`: create with `after`, update without the full document (header contents, including a struct in `truncatedArrays`), and a document flattened by `ExtractNewDocumentState`. - Without the fix, 4 endpoint tests, the converter test and the mongodb update test fail. The schema change test passes either way; it guards that behaviour. - `mvn clean install -Psourcecheck` in `components/camel-debezium`: all modules pass (27 + 32 + 4 + 8 + 5 + 4 + 5 + 5 tests). ### Docs and generated files - The Message body section of the six component docs describes the SMT, and the mongodb one also describes the update header. The catalog doc copies are updated. - Regenerated `debezium-mongodb.json` (component and catalog) and `DebeziumMongodbEndpointBuilderFactory`. The mongodb `@BodyTypes` description now mentions the update header. - Upgrade guide 4.24: new `camel-debezium` entry. The body is no longer `null` with the SMT, the `patch` fallback is gone, there is a new header, and the converter change. ### Not changed - Only the default `__` prefix of the SMT is supported for the operation. A custom `add.fields.prefix` is not read. - With the SMT, `CamelDebeziumTimestamp`, `CamelDebeziumSourceMetadata` and `CamelDebeziumBefore` stay `null`, because the SMT removes those fields. - The note in the mongodb doc that `CamelDebeziumBefore` is not available is outdated, since Debezium 2+ has pre-image capture modes. It is left as it is. _Claude Code on behalf of Claus Ibsen (davsclaus)_ 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
