GitHub user singhpratech created a discussion: Apache Cloudberry through ODBC into Apache Arrow (adbcBridge 0.1.0) — is multi-argument unnest a safe bulk-ingest path on Cloudberry?
adbcBridge is a small Apache-2.0 ADBC driver written in C11 that loads an ODBC driver and exposes it through the Arrow ADBC C ABI: block-cursor reads straight into Arrow record batches, bulk ingest, metadata, partitioned parallel reads. I ran Apache Cloudberry through it as one of 46 databases in a single compatibility workload, and it is one of the few entries that needed no driver quirks and no tolerance flags at all — so I wanted to share the entry here and ask one question at the end. **What was verified** (Apache Cloudberry 2.1.0-incubating, `woblerr/cloudberry` image; driver psqlodbc 16, PostgreSQL wire): - Linux: PASS - macOS arm64: PASS (amd64 emulated) - Windows x64: PASS (compose service unchanged, 3 GB / shm 1 GB) **What the compatibility entry records:** an MPP cluster of PostgreSQL 14 segments behind one coordinator, driven by the `postgres` entry's types unchanged (`INTEGER`, `DOUBLE PRECISION`, `VARCHAR`, `BYTEA`, `DATE`, `TIMESTAMP`, `NUMERIC(10,3)`, `BOOLEAN` all behave as on stock PostgreSQL) and, unlike CockroachDB, needing no `PRIMARY KEY`. Since it reports `SQL_DBMS_NAME` "PostgreSQL" behind the same `psqlodbcw.so`, no driver-name quirk *could* be correct here without also firing on real PostgreSQL; the only place Cloudberry is named in the bridge is a fork test on the `version()` banner. Extra steps cover what the standard workload cannot tell apart from PostgreSQL: a `DISTRIBUTED BY` table whose bulk-ingested rows occupy both segments plus an aggregate merged on the coordinator (`Gather Motion 2:1`, GPORCA), and append-optimized column-oriented storage read from `pg_am` as `ao_column`. Full entry: https://github.com/singhpratech/adbcbridge/blob/main/docs/COMPATIBILITY.md **The question.** For real PostgreSQL the bridge ingests through a multi-argument `unnest` form — `INSERT INTO t SELECT * FROM unnest($1::bigint[], $2::text[], …)` with one array parameter per column — and it only turns that on when `version()` is a PostgreSQL banner carrying no fork marker, so Cloudberry (banner `PostgreSQL 14.4 (Apache Cloudberry 2.1.0-incubating …)`) keeps the plain multi-row `INSERT` path. Probing Cloudberry directly, the `unnest` form works exactly as on PostgreSQL — from array literals and from bound `bigint[]`/`text[]` parameters, into heap, append-optimized row and append-optimized column tables, across the segments — and server-side it lands 5,000 rows roughly an order of magnitude faster than the multi-row `INSERT` it keeps (~530k vs ~33k rows/s, bare SQL on a shared host). Is relying on multi-argument `unnest` expansion something Cloudberry considers stable across releases, the same way PostgreSQL does — i.e. is it reasonable to let Cloudberry through that gate? If there is a caveat (GPORCA vs the planner, distribution of the unnested rows, anything on the 2.x roadmap), I would rather know before flipping it. Trying it (Python; Rust, Go, Java and C# are on the docs site): ``` pip install adbcbridge ``` ```python import adbcbridge with adbcbridge.connect(uri="Driver=psqlodbcw.so;Servername=127.0.0.1;Port=5432;Database=mydb;Username=gpadmin;Password=…;") as conn: with conn.cursor() as cur: cur.execute("SELECT ...") table = cur.fetch_arrow_table() # a pyarrow.Table ``` Links: repository https://github.com/singhpratech/adbcbridge · docs https://adbcbridge.org/docs/ · upstream notes https://github.com/singhpratech/adbcbridge/blob/main/docs/UPSTREAM.md · PyPI https://pypi.org/project/adbcbridge/ It is a 0.1.0. If the entry says something wrong about Cloudberry, or you run a version I didn't, an issue on the repository with the details is the most useful thing you could send. GitHub link: https://github.com/apache/cloudberry/discussions/1939 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
