debabsah commented on issue #3842:
URL: https://github.com/apache/gravitino/issues/3842#issuecomment-4796661962
Hi @jerryshao, thanks for the note that this is open to pick up. I would
like to take it on, and because it introduces a new catalog kind rather than a
connector for an existing one, I want to start with a design rather than code.
I would propose a short design doc covering:
- entity mapping: how a vector database's collections and indexes map onto
Gravitino's catalog / schema / table / column model, and where vector-specific
concepts (index type, dimensionality, distance metric) belong;
- the new VECTOR value on Catalog.Type and a catalog-vector-<engine> module
behind the existing CatalogProvider / BaseCatalog SPI, mirroring catalog-jdbc-*
and catalog-lakehouse-*;
- scope: strictly vector-database-as-a-governed-catalog (the metadata and
governance plane), kept separate from the Lance format work already in flight
so there is no overlap.
For the first engine, would you prefer Milvus (matching the RAG use case
@zhoukangcn raised) or pgvector (simplest, JDBC-backed) as the reference
implementation? I am happy to write the design doc for review first and only
begin implementation once the model is agreed. Would a design doc on this issue
(or a linked sub-issue) work, or is there a preferred process for adding a new
kind?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
