david-streamlio opened a new issue, #67:
URL: https://github.com/apache/pulsar-connectors/issues/67
## Motivation
Debezium ships a Google Cloud Spanner source connector
(`io.debezium:debezium-connector-spanner`, present in the `debezium-bom` at our
pinned version), but this repository does not wrap it. We currently wrap 5 of
Debezium's 12 source connectors: MySQL, PostgreSQL, MongoDB, Oracle, and SQL
Server.
Spanner is a managed, horizontally-scaled database with a growing Pulsar
user base, and the connector itself wraps cleanly — the interesting work is in
testing.
## Proposal
Add a `debezium/spanner` module (`pulsar-io-debezium-spanner`) following the
structure of the existing Debezium modules:
- `DebeziumSpannerSource` extending `DebeziumSource`
- `include("debezium:pulsar-io-debezium-spanner")` in `settings.gradle.kts`
- `debezium-connector-spanner = { module =
"io.debezium:debezium-connector-spanner" }` in the version catalog
- NAR packaging + distribution wiring
- An integration test against the **Spanner emulator**
(`gcr.io/cloud-spanner-emulator/emulator`) so CI needs no GCP credentials
## Notes
- Spanner CDC works via **change streams**, which must be created with DDL
(`CREATE CHANGE STREAM ...`) as part of test setup. Confirm the emulator
supports change streams at the version we target — if it does not, the test
must be credential-gated, and we should avoid repeating the
`azure-data-explorer` situation (#49) where a credential-gated E2E test means
the connector is never actually exercised in CI.
- Spanner is a **multi-partition** connector, so it will likely hit the
`task.id` gap described in #61. That issue should land first.
- Use the `GenericContainer` pattern (see the Testcontainers version
conflict noted in #62).
_Identified by a scan of the Debezium connector catalog against our
connector inventory._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
