GitHub user igor-suhorukov added a comment to the discussion: [DISCUSSION] Making Apache Cloudberry an Agent-Native Analytical Database
@MisterRaindrop great progress on `lance_fdw`. One suggestion for the read path before deciding between A/B/C: have you looked at using **[pg_duckdb](https://github.com/duckdb/pg_duckdb) together with DuckDB's [`lance` extension](https://duckdb.org/docs/stable/core_extensions/lance)**, either instead of a hand-written FDW or alongside it? ### Why it might be worth it - **Lance reading is already solved upstream.** DuckDB's `lance` extension was built with LanceDB and DuckDB Labs. It reads and writes Lance datasets from local paths and S3. It also exposes `lance_vector_search`, `lance_fts` and `lance_hybrid_search` as table functions, with a `prefilter` option. That covers Phase 1–2 and the hybrid search planned for later. - **pg_duckdb supports PG 14**, which matches Cloudberry's kernel. It can load DuckDB extensions and run DuckDB SQL from inside Postgres. - **Less to maintain.** Several things would become someone else's job: Lance format changes, type mapping (structs, maps), S3 credentials, and FTS and hybrid scoring. We also wouldn't need to pin and package the `lance-c` 0.x ABI, which upstream says is unstable. - **Iceberg comes with the same engine.** DuckDB already reads Iceberg and Parquet, which fits the Iceberg + Lance picture from the original proposal. - **Phase 4 write-back.** DuckDB can write Lance (`COPY ... TO ... (FORMAT lance)`), so AI-derived features could go back into Lance as well as into native tables. Rough sketch (exact pg_duckdb syntax to be verified): ```sql SELECT duckdb.install_extension('lance'); SELECT r['id'], r['content'], r['_distance'] FROM duckdb.query($$ SELECT id, content, _distance FROM lance_vector_search( 's3://bucket/documents.lance', 'embedding', [0.1, 0.2, ...]::FLOAT[768], k = 20, prefilter = true) $$) r; ``` ### Caveats - **MPP.** As far as I know, pg_duckdb has not been ported to or tested on Cloudberry. It takes over planning and runs the query inside a single backend. On Cloudberry that most likely means the QD only, which loses the fragment-to-segment assignment and distributed Top-K you've built. For retrieval-style queries the Top-K result is small, so QD-only may be acceptable. For large scans joined with distributed tables it probably isn't. - **Two engines.** Memory used by DuckDB sits outside Cloudberry's memory accounting and resource groups. Type mapping between the two engines also adds a layer that has to be maintained. ### Suggestion A quick benchmark would settle this: 1. Prefiltered vector Top-K 2. A full Lance scan joined with a native distributed table Run both with pg_duckdb + `lance` on the QD and with `lance_fdw`. If pg_duckdb is good enough for (1), it could be the low-maintenance default for retrieval and MCP tools. `lance_fdw` would then focus on what it does better: MPP scans and distributed Top-K. GitHub link: https://github.com/apache/cloudberry/discussions/1967#discussioncomment-18873689 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
