moneycat957 commented on issue #10443:
URL: https://github.com/apache/seatunnel/issues/10443#issuecomment-5491223737
I'd like to implement the **Amazon DocumentDB source** connector.
Checking the tracker against the current `dev` tree: Cosmos DB landed in
#11167, Airtable
in #10469, SingleStore is in flight as #10521, and Couchbase source was
picked up on
Aug 26. Amazon DocumentDB and Ingres are the two rows nobody has claimed
yet. I'd like to
take DocumentDB.
**Why a dedicated connector rather than pointing users at
`connector-mongodb`**
DocumentDB is MongoDB wire-compatible, so this is the obvious first
question. Three
reasons it still needs its own module:
1. There is already precedent for dedicated modules for AWS managed services
—
`connector-amazondynamodb` and `connector-amazonsqs`.
2. `connector-azurecosmosdb` was just merged even though Cosmos also exposes
a MongoDB
API, so a compatibility layer alone hasn't been treated as a reason to
skip a
dedicated connector.
3. It genuinely does not work out of the box today. DocumentDB requires TLS
with the
Amazon RDS CA bundle, it does **not** support `retryWrites` — which the
MongoDB driver
enables by default, so a plain `connector-mongodb` connection fails
immediately — and
it is missing aggregation stages that `connector-mongodb`'s pushdown path
can emit.
If maintainers would rather solve this by adding DocumentDB compatibility
options to
`connector-mongodb` instead of creating a new module, I'm happy to take that
route — I'd
just like the direction settled before writing code.
**V1 scope (source only)**
Following the shape @DanielLeens has asked for on the other slices in this
thread:
*In scope*
- `connector-amazondocumentdb`, source only. Sink methods stubbed, not
implemented.
- Connection options: cluster endpoint, port, database, collection,
credentials,
`tls` + CA bundle path, and `retryWrites=false` enforced with a clear
error if a user
overrides it.
- Basic type mapping from BSON to SeaTunnel row types, mirroring
`connector-mongodb`.
- Full-collection read with a configurable batch size. Optional projection.
- Unit tests for option validation and type mapping.
*Out of scope for V1*
- Sink
- CDC / change streams
- Catalog discovery and schema inference
- Multi-table source
- IAM authentication (username/password only in V1)
- Aggregation pipeline pushdown
Two questions before I start:
1. Is `connector-amazondocumentdb` the naming you want, to match
`connector-amazondynamodb` / `connector-amazonsqs`?
2. For E2E, DocumentDB has no official local emulator. Would you accept E2E
tests running
against a `mongo` Testcontainer with `retryWrites=false` as the closest
proxy, with the
DocumentDB-specific TLS path covered by unit tests only? Or would you
rather V1 ship
with unit tests alone and defer E2E?
I'll open the PR against `dev` within the four-week window.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
