Hi all, I've put together a design doc for a native PostgreSQL connector for the Go SDK (postgresio). It covers three transforms: a sink using staged COPY with an UNNEST fallback for transaction poolers, a bounded batch read over server-side cursors with optional range partitioning, and a CDC source built on logical replication and pgoutput. All three are also registered as SchemaTransforms, making them accessible from Python and Beam YAML.
Design Doc: https://docs.google.com/document/d/1X8NtzpGor8Atn1yg5WT8JR5pLTObAbtFXov1f-WFjzQ/edit?usp=sharing Tracking Issue: https://github.com/apache/beam/issues/39959 An initial implementation is already written (generated) as a 7-PR stack, starting with the sink implementation at https://github.com/apache/beam/pull/40166. The design doc focuses on the API and design; the "Implementation status" section details what the current test suite covers. I would especially appreciate feedback on three key areas: 1. URN Overlap: postgres_read:v1 and postgres_write:v1 are currently implemented by the Java JDBC providers, with Managed routing Python to the Java expansion service. The Go providers register these same URNs with a superset config schema. Should we use separate Go URNs, establish a shared contract with a documented compatibility subset, or keep Go registration out of Managed for now? 2. Verification Bar: The suite is hermetic—only 8 of 448 test functions interact with a real server, and none run in CI. Is live-server integration testing required before merging, or is an experimental package with a hermetic suite and documented follow-up testing acceptable? 3. PR Split: PR 3 (CDC layer which is the largest) is ~21k lines across 59 files, which is difficult to review. I split the stack so each commit compiles and passes tests independently, but could not find a finer split that preserves this property for CDC section. Suggestions are welcome. The doc also covers a few smaller questions regarding lib/pq vs. pgx/pglogrepl, construction-time panics vs. error-returning constructors, and whether the Arrow output format should be included in the initial scope. If you have bandwidth to review the design doc, and the initial PR 1 which is the most self-contained starting point (core configuration and the sink). I would appreciate your feedback! Thanks, Jignesh Shah (new to Beam)
