davsclaus opened a new pull request, #30618:
URL: https://github.com/apache/camel/pull/30618
Fixes CAMEL-27398, CAMEL-27399 and CAMEL-27400: three camel-debezium tickets
from the new debezium-postgresql-source Kamelet (apache/camel-kamelets#3363),
one commit each.
## CAMEL-27398: fail at startup when the offset file cannot be written
With the default file offset store, the consumer started when the folder of
`offsetStorageFileName` did not exist. Every save of the position then failed
with only a WARN, and a restart sent everything again, the snapshot included.
-
`EmbeddedDebeziumConfiguration.validateDebeziumEmbeddedEngineConfiguration()`
now checks the files as the engine gets them, the `additionalProperties`
included:
- offset file: the folder must exist and be writable, and the file, if it
exists, must be writable and not a folder;
- file schema history (`schemaHistoryInternal=...FileSchemaHistory`):
`schemaHistoryInternalFileFilename` must be set and writable.
- The error names the file, such as: `The file of offsetStorageFileName
(/data/offsets.dat) is in the folder /data, which does not exist: create it, or
use a file in a folder that exists. Debezium saves its position to this file:
without it, a restart sends all the changes again.`
- **One difference from the ticket:** for the schema history, a missing
folder is not an error. Debezium's `FileSchemaHistory` creates its folders
itself (`Files.createDirectories`), so only a folder that cannot be created
fails. Kafka's `FileOffsetBackingStore` does not create its folder, which is
why the offset check is stricter.
- The validation runs when the endpoint is created, as before, so a route
with such an endpoint fails at startup.
- The existing tests used paths such as `/test` and `/offset/file`; they
now use files under `target/`.
- `DebeziumConsumerEngineFailureTest` creates its `target/data` folder in
a `@BeforeAll`, as the endpoint is created before its `@BeforeEach`.
- Docs of the six connectors:
- the examples no longer use `/usr/offset-file-1.dat`, which would now
fail on most machines;
- a new section "Where the position is saved" covers the file store, the
file schema history, and the Kafka store (`offsetStorageTopic`, ... and
`additionalProperties.bootstrap.servers`).
- The catalog copies are updated, and the upgrade guide has an entry.
## CAMEL-27399: `DebeziumTypeConverter.toMap` keeps the field order
- The converter builds a `LinkedHashMap` in the order of the fields of the
Struct's schema (the column order of the table). Nested Structs are included,
also inside a list, and a map field keeps its order.
- Before, it was a `HashMap`, so the JSON had no defined order.
- New test `testToMapKeepsTheOrderOfTheFields`, and an upgrade guide note
(same entry).
## CAMEL-27400: depend on `debezium-storage-kafka` directly
- `camel-debezium-common` declares `io.debezium:debezium-storage-kafka` next
to `debezium-storage-file`. Before, it came only transitively via
`debezium-embedded`, with the same version and no exclusions, so nothing
changes today.
- With Kafka 4.x the embedded engine creates the Kafka offset stores through
its `OffsetStoreProvider`s. That includes the default file store
(`KafkaFileOffsetProvider`), not only the topic store
(`KafkaOffsetStoreProvider`).
- New test `DebeziumOffsetStoreProvidersTest` checks that both providers are
found. The new docs section covers the Kafka store.
## Tests
- New `EmbeddedDebeziumStorageFilesTest` (9 tests): a folder that exists, a
missing folder, a folder that cannot be written, a file that is a folder, the
`additionalProperties` override, another offset store, and the file schema
history (missing folders are fine, a missing file name fails, a folder that
cannot be created fails).
- All unit tests of the seven debezium modules pass. The common module was
also run from a clean `target/data` with no test retries.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]