GitHub user petrizhang edited a discussion: Relify: Parquet-native vector indexes with DataFusion
Hi everyone! I’m building [Relify](https://github.com/petrizhang/relify), an early-stage Python and Rust project that stores IVF vector indexes as ordinary Parquet datasets and queries them directly with DataFusion. This keeps vector search inside the relational plan, allowing results to be filtered, joined, and aggregated without a separate vector database. ```python import relify from relify.datafusion import col, functions session = relify.connect("./relify-data") session.register_parquet( "documents", relify.datasets.uri("documents"), ) documents = session.table("documents") documents.create_index( "documents_embedding", column="embedding", key=["document_id"], config=relify.IVF(nlist=3), ) documents.wait_for_index("documents_embedding") query = ( documents.search([0.2, 0.0], column="embedding") .where("tenant_id = 42 AND status = 'published'") .nprobes(3) .limit(3) .select(["document_id", "title"]) ) hits = session.to_dataframe(query) document_stats = session.read_parquet( relify.datasets.uri("document_stats") ) result = ( hits.join(document_stats, on="document_id") .aggregate( "category", [ functions.count(col("document_id")).alias("matches"), functions.avg(col("_distance")).alias("avg_distance"), functions.max(col("popularity")).alias("max_popularity"), ], ) .sort("category") ) print(result.to_pydict()) ``` The longer-term vision is an embedded multimodal lakehouse built on DataFusion, supporting both writes and queries in one local system. I’d love to hear your feedback and whether anyone is interested in this direction. GitHub link: https://github.com/apache/datafusion/discussions/24131 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
