nitinware opened a new pull request, #40648: URL: https://github.com/apache/beam/pull/40648
Adds CouchbaseIO, a Java connector for Couchbase Server. Fixes #18381. R: @kennknowles **Design discussion:** https://lists.apache.org/thread/cvzpfnw717zbnf7z6slfkfbp6zg6604g **Design doc:** https://docs.google.com/document/d/1ALlXaiVSa_Dn9jJxM9MsmhEhBLPW_FP8RswN8HG33iU ### Implementation The commits are organized by component to make the changes easier to review: 1. **Write:** Concurrent, batched UPSERT operations using `withIdFn` and `withContentFn`, with bounded in-flight concurrency and flushing in `@FinishBundle`. 2. **JsonObjectCoder:** UTF-8 JSON encoding for Couchbase SDK `JsonObject` values. 3. **Read:** Bounded SQL++ query execution as a single unit of work, with an optional parsing function (defaulting to `JsonObject`), named/positional parameters, scan consistency, and opt-in `withRedistribute()` applied after parsing. 4. **Schema-aware reads:** `readRows(Schema)` converts query results into Beam `Row` values using `RowJson`, respecting the supplied schema's nullability constraints. 5. **Documentation:** Class Javadocs, `CHANGES.md`, and the connector documentation table. 6. **Review improvements:** Display data, a build configuration comment, and an upsert idempotency test. ### Testing - **39 unit tests** across `CouchbaseIOTest` and `JsonObjectCoderTest`. - **15 Testcontainers integration tests** across `CouchbaseIOReadTest` and `CouchbaseIOWriteTest`, using `couchbase/server:enterprise-8.0.3`. - Tests cover authentication failures (the password must not appear in errors, `toString`, or display data), type mismatches, missing required fields, and repeated UPSERT operations. - Local validation passes: `./gradlew :sdks:java:io:couchbase:build` (Checkstyle, Javadoc, Checker Framework nullness validation, and dependency analysis). - `checkJavaLinkage` could not run locally because of a corporate repository proxy restriction; CI validation is pending. - TLS (`couchbases://`) is delegated to the Couchbase SDK but is not covered by an integration test. ### Known limitations / follow-ups - **Connection lifecycle:** One Couchbase `Cluster` per `DoFn` instance. JVM-wide sharing may be considered after benchmarking. - **Source parallelism:** SQL++ reads execute as a single unit of work. Partitioned reads are deferred. - **SchemaTransform / Managed I/O:** Deferred pending further community feedback (design doc ยง12, question 2). - **Additional capabilities:** Advanced write modes, durability configuration, dead-letter handling, and DCP-based CDC are future work. <!-- Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). - [ X ] Replace these instructions with a description of your change and why you have made it. - [ X ] Mention the appropriate issue in your description (for example: `addresses #123` or `fixes #456`). - [ X ] Update `CHANGES.md` with noteworthy changes. See [CONTRIBUTING.md]() for more details. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
