GitHub user MisterRaindrop added a comment to the discussion: [DISCUSSION] Making Apache Cloudberry an Agent-Native Analytical Database
Thanks — these are exactly the execution constraints that need to be made explicit. I checked the current Cloudberry and Lance implementations and also ran a small local spike. 1. **Segment-to-fragment assignment** Agreed. This should be part of Phase 1, not deferred to distributed Top-K. The QD should pin one Lance dataset version, enumerate its fragments, assign each fragment exactly once, and pass the assignments to the QEs through `fdw_private`. This is similar to the existing PXF pattern: the dispatcher obtains and serializes the fragment list, while each segment keeps only its assigned subset. `fragment_id % num_segments` is sufficient for an initial correctness prototype, although a size-aware assignment would eventually handle uneven fragments better. The Phase 1 acceptance criteria should include: - no duplicate rows with `mpp_execute 'all segments'`; - no missing fragments; - all QEs reading the same pinned dataset version; - stable behavior when the number of fragments differs from the number of segments. 2. **Fragment-scoped and distributed ANN** I ran a local spike, and the current Lance model makes this feasible. The important distinction is that normal scans can be assigned by data fragment, while indexed ANN should normally be assigned by whole physical **index segments**. A Lance logical index can contain multiple physical segments, each covering a disjoint fragment subset. Lance also exposes APIs for restricting a query to selected index segment UUIDs. This is the same ownership model used by Lance-Ray's distributed vector search. In a small test with 32,000 vectors, four data fragments, and two physical IVF_FLAT index segments: - a pure C program linked against `lance-c v0.1.9`; - each index-segment-scoped query returned rows only from its covered fragments; - there were zero fragment-ownership violations; - merging the two local Top-K result sets matched Lance's global Top-K result for all 12 test queries. I also tested a prefiltered ANN query through the C API. All returned rows satisfied the scalar predicate, and the Top-5 matched an exact filtered search in that test. A separate IVF_PQ experiment confirmed that recall depends materially on `nprobes`, refinement, and per-worker oversampling. Therefore, this establishes functional feasibility, not yet acceptable MPP/object-store performance. For Phase 3, the likely execution model is: - QD pins the dataset snapshot and reads index-segment metadata; - whole index segments are assigned to QEs; - fragments not covered by an index use a flat-search fallback; - each QE returns an oversampled local candidate set; - Cloudberry performs the final distance-ordered global Top-K merge. The next spike should run this inside Cloudberry and measure recall against an exact filtered search, along with S3 I/O, skew, and latency. 3. **pgvector versus Lance** Agreed. Cloudberry already has distributed pgvector execution. One terminology detail is that the observed Cloudberry plan uses `Gather Motion 3:1` with a `Merge Key`, rather than PostgreSQL's `Gather Merge`, but the substance of the point is correct. The distinction I would make is: - pgvector: vectors stored and indexed in Cloudberry native relational tables; - Lance: external, versioned vector/multimodal datasets on object storage, shared with other lakehouse and ML engines; - Cloudberry: SQL joins, aggregation, MPP planning, and global result merging across those sources. So Lance is not justified merely by distributed vector search. Its value is avoiding ingestion and duplication of lake-resident multimodal data. I also agree that Phase 2's acceptance test should be a prefiltered ANN query, not only plain `distance + LIMIT`. 4. **Extension and C interface** I agree with keeping this as a separate extension, but the interface situation has changed: there is now an official [`lance-c`](https://github.com/lance-format/lance-c) project. Its C API supports dataset scans, Arrow streams, SQL/Substrait filters, fragment restriction, vector search, index-segment enumeration, and index-segment-scoped search. I verified the read and ANN paths from a pure C11 consumer using the official v0.1.9 binary. There are still two important qualifications: - `lance-c` is implemented on top of Rust, although a Cloudberry FDW can link against the resulting `liblance_c` without implementing its own Rust FFI layer; - the project explicitly treats its 0.x ABI as unstable across minor releases, so an extension should pin and package a tested `lance-c` version. The v0.1.9 C API can build uncommitted distributed index segments, but I did not find a C API for committing those segments as one logical index. That does not block a read-only FDW consuming existing datasets and indexes, but distributed index construction would currently need another coordinator path or an additional upstream C API. So my current conclusion is: **yes to a separate extension, using the official version-pinned `lance-c` interface, with fragment ownership in Phase 1 and prefiltered ANN recall as the Phase 2 gate.** GitHub link: https://github.com/apache/cloudberry/discussions/1967#discussioncomment-18343644 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
