GitHub user my-ship-it added a comment to the discussion: [DISCUSSION] Making Apache Cloudberry an Agent-Native Analytical Database
Thanks for the write-up. The "agent decides what, Cloudberry decides how" boundary makes sense to me, and MCP-first for Phase 0 is the right call. A few points I think need answers *before* Phase 1, since they decide whether Phases 2–3 are feasible on an MPP engine: 1. **Segment ↔ fragment mapping.** With `mpp_execute 'all segments'`, a naive FDW makes every segment scan the whole Lance dataset and the result is silently multiplied by the segment count (we have hit exactly this with `postgres_fdw` on Cloudberry). Each segment must scan only its own subset of fragments (e.g. `fragment_id % num_segments == gp_segment_id`). This should be an explicit Phase 1 deliverable, since distributed Top-K in Phase 3 depends on it. 2. **Global index vs. partitioned scan.** Lance's vector index is built over the whole dataset, but MPP wants each segment to search only its fragments. Whether an IVF/PQ search can be restricted to a fragment subset with acceptable recall needs a quick spike. If it can't, Phase 2/3 collapse to either coordinator-only index scan (no MPP) or per-segment brute force (no index). 3. **Answer question 3 explicitly.** Cloudberry already has pgvector with per-segment indexes and `ORDER BY dist LIMIT k` via Gather Merge, which *is* distributed vector search. Lance's real differentiators are: data stays in the lake, lazy-loaded multimodal blobs / versioned datasets, and sharing the same files with Hudi/Paimon/Fluss. Worth stating up front, because "why not just pgvector?" will be the first review question. Also, the revenue-drop + tickets example is a *pre-filtered* ANN query; the filter has to be pushed into Lance, so I'd make that the Phase 2 acceptance test rather than plain `distance + LIMIT`. 4. **Separate extension, not in-tree.** Lance is Rust with no stable C API, so the FDW will need a Rust cdylib + FFI layer. Keeping it out of the core build (the way `cloudberry_fdw` is) avoids a long toolchain discussion and moves faster. So a clear yes on question 2. Happy to discuss the fragment-assignment design in (1) further. GitHub link: https://github.com/apache/cloudberry/discussions/1967#discussioncomment-18342691 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
