Hi All,

Just following up on the CouchbaseIO PR. The implementation is complete and
has been ready for review for about three weeks.

I'd appreciate a review whenever someone has a chance. The CI workflows are
also awaiting approval, and I'd be grateful if a maintainer could help with
that.

PR: https://github.com/apache/beam/pull/40648

Thanks,
Nitin

On Tue, 6 Oct 2026 at 11:16, nitin ware <[email protected]> wrote:

> Thanks Kenn, that clarifies it. To summarize what I'll take into the
> design doc:
>
>    -
>
>    read() with a parsing function, defaulting to the SDK's JsonObject
>    when none is provided
>    -
>
>    readRows(Schema), converting to Row in the read DoFn
>    -
>
>    opt-in withRedistribute(), applied after parsing so we can avoid
>    shuffling the raw JSON representation when a more compact encoding is
>    available
>
> I'll share the updated doc on this thread once it's revised.
>
> Nitin
>
> On Tue, 6 Oct 2026 at 10:36, Kenneth Knowles <[email protected]> wrote:
>
>>
>>
>> On Tue, Oct 6, 2026 at 11:31 AM nitin ware <[email protected]> wrote:
>>
>>> Thanks Kenn for taking a look at the design doc so quickly and for the
>>> detailed feedback. This is really helpful.
>>>
>>> > it is OK to have CouchBase SDK types in the new API, and possibly
>>> desirable.
>>>
>>> Makes sense. I'll revisit the API with that in mind. One option is for
>>> read() to accept a mapping function from the SDK's JsonObject to T, and for
>>> write() to accept functions that map T to a document ID and a JsonObject.
>>> This would also let users reuse their existing Couchbase conversion code.
>>>
>>
>> This matches some other IOs that have a "parsingFn". I'm sure there are
>> also users that just don't bother "parsing" so of course you will want
>> the identity function as a default, or something like that.
>>
>>
>> > The sooner you can convert a JSON string into something with a more
>>> compact encoding, the better.
>>>
>>> The mapping would happen inside the read DoFn, so the JSON
>>> representation wouldn't need to flow further through the pipeline. I'm also
>>> considering a readRows(Schema) API that converts to Row in the same DoFn.
>>> Would that be along the lines of what you had in mind, or is there another
>>> representation/pattern you'd recommend?
>>>
>>
>> Nope, I think this is exactly right. And the readRows is a good idea, if
>> it is practical. I know the Row/JSON mismatch can be severe sometimes,
>> though typically the JSON is just as structured as a nested Row structure
>> when you are dealing with big data sets (otherwise processing them is a
>> nightmare).
>>
>>
>>
>>> > you may want to have .withRedistribute to optionally build in a
>>> shuffle as KafkaIO does.
>>>
>>> Agreed. I'll look at following the KafkaIO pattern with an opt-in
>>> withRedistribute() and corresponding configuration for the number of
>>> redistribution keys.
>>>
>>
>> And possibly the interesting part here is that you might want to use a
>> more compact encoding (even BSON if nothing else) so that you don't
>> immediately shuffle the whole size of the JSON read.
>>
>> Anyhow, all of this will shake out, I'm sure.
>>
>> Kenn
>>
>>
>>>
>>> I'll update the design doc based on this feedback.
>>>
>>> Thanks,
>>> Nitin Ware
>>>
>>> On Tue, 6 Oct 2026 at 09:27, Kenneth Knowles <[email protected]> wrote:
>>>
>>>> Nice!
>>>>
>>>> Things that jump out at me immediately:
>>>>
>>>>    - it is OK to have CouchBase SDK types in the new API, and possibly
>>>>    desirable. If there are libraries that work with CouchBase SDK types, 
>>>> then
>>>>    you want to be able to write a DoFn that invokes them in a simple 
>>>> manner.
>>>>    So if the documents are already a specialized Beam type you'll have to
>>>>    convert them back. For example, if your frontend app has
>>>>    conversions between Couch documents and application-specific types that 
>>>> you
>>>>    want invoke in your DoFns. But if you are used to working directly
>>>>    with JSON string encoding you don't need this, of course.
>>>>    - The sooner you can convert a JSON string into something with a
>>>>    more compact encoding, the better. Ideally before you shuffle. If
>>>>    there is a common or standard way to do this, you might want to bake it 
>>>> in.
>>>>    - This design saves parallel reads for future work. That is fine,
>>>>    but of course for large amounts of data you will likely need it. The
>>>>    CouchBase SDK client may be able to read very fast to saturate a
>>>>    local buffer, but commits to shuffle will be more expensive and need
>>>>    parallelism. So then if parallel reads are less common, you may want to
>>>>    have .withRedistribute to optionally build in a shuffle as KafkaIO
>>>>    does.
>>>>
>>>> Kenn
>>>>
>>>> On Tue, Oct 6, 2026 at 10:02 AM nitin ware <[email protected]> wrote:
>>>>
>>>>> Hi Beam Community,
>>>>>
>>>>> I've picked up https://github.com/apache/beam/issues/18381 and would
>>>>> like to add a Couchbase connector to the Java SDK. Before starting the
>>>>> implementation, I'd like to get feedback from the community on the 
>>>>> proposed
>>>>> design and initial scope.
>>>>>
>>>>> Design doc: link
>>>>> <https://docs.google.com/document/d/1ALlXaiVSa_Dn9jJxM9MsmhEhBLPW_FP8RswN8HG33iU/edit?usp=sharing>
>>>>>
>>>>> Couchbase is an established distributed document database, but Beam
>>>>> currently has no native Couchbase connector. CouchbaseIO would provide a
>>>>> reusable integration instead of requiring users to build custom DoFns
>>>>> around the Couchbase SDK.
>>>>>
>>>>> The first version is intentionally small:
>>>>>
>>>>>    -
>>>>>
>>>>>    read() runs a user-supplied SQL++ query and streams the results as
>>>>>    a bounded PCollection<CouchbaseDocument>. The query is not
>>>>>    partitioned in v1; partitioned reads are proposed as the first 
>>>>> follow-up.
>>>>>    -
>>>>>
>>>>>    write() upserts documents with bounded concurrency. Retrying the
>>>>>    same keyed documents results in the same end state; the connector does 
>>>>> not
>>>>>    claim exactly-once semantics.
>>>>>    -
>>>>>
>>>>>    CouchbaseDocument is a small Beam-owned type containing a document
>>>>>    ID and JSON body, keeping Couchbase SDK types out of the public API.
>>>>>    -
>>>>>
>>>>>    The connector includes connection configuration, TLS,
>>>>>    bucket/scope/collection selection, credential redaction, unit tests, 
>>>>> and
>>>>>    integration tests against Couchbase.
>>>>>
>>>>> The design doc also describes potential follow-up work, including
>>>>> partitioned reads, additional write modes, durability, dead-letter output,
>>>>> Row/schema support, SchemaTransform/Managed I/O, and DCP-based streaming.
>>>>>
>>>>> I've listed a few open questions in the design doc, particularly
>>>>> around the v1 scope, element type, and whether SchemaTransform/Managed I/O
>>>>> should be included in the initial implementation. I'd appreciate feedback
>>>>> on those or any other aspects of the proposed design.
>>>>>
>>>>> Please reply on this thread rather than in the design doc comments so
>>>>> that the discussion remains archived on the mailing list.
>>>>>
>>>>> Once there's some agreement on the direction, I'll start sending
>>>>> small PRs.
>>>>>
>>>>> Thanks,
>>>>> Nitin Ware (@nitinware <https://github.com/nitinware>)
>>>>>
>>>>

Reply via email to