debabsah commented on issue #3842: URL: https://github.com/apache/gravitino/issues/3842#issuecomment-5422816048
Hi @jerryshao, following up on my June note. Since there is no near-term plan on your side, I put together a design proposal for this epic and would like the PMC's steer before any code is written. The short version: add a `VECTOR` catalog kind so a vector database can be governed as a first-class catalog peer to `RELATIONAL`, `FILESET`, `MESSAGING`, and `MODEL`, behind the existing `CatalogProvider` / `BaseCatalog` SPI and discovered via `ServiceLoader`, mirroring how `catalog-model` is put together. Milvus first. Scope for v1 is metadata and governance only: register, list, load, and drop collections, and expose each collection's field schema and index metadata as governed entities. Explicitly no vector search, no query federation, and no write path in v1, and explicitly disjoint from the Lance lakehouse-format work. The questions I actually need answered before building anything: 1. Should this be a dedicated `VectorCatalog` SPI with a `Collection` entity, which makes VECTOR a true kind, or reuse `TableCatalog` / `Table` with a vector connector, which is less code but is arguably just another relational connector? 2. For the embedding field, carry `LIST(FLOAT)` plus dimensionality and metric in properties for v1, or add a native type to the type system? 3. ANN index metadata in properties for v1, or extend the `Index` abstraction? 4. Milvus first, pgvector first, or both? 5. Can you confirm metadata-only governance is the right v1 scope, consistent with MODEL? I am not claiming the epic, and I know a new `Catalog.Type` is an `@Evolving` API change that needs PMC buy-in. If the direction is wrong, or if someone on the PMC would rather own the API change and have me take a later phase, that works too. Happy to open a discussion thread or a dev@ mail if that is the better venue for this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
