Andrew Kyle Purtell created PHOENIX-7998:
--------------------------------------------
Summary: Vector Indexes
Key: PHOENIX-7998
URL: https://issues.apache.org/jira/browse/PHOENIX-7998
Project: Phoenix
Issue Type: New Feature
Components: core
Reporter: Andrew Kyle Purtell
Vector similarity search, identifying the top _k_ rows whose high dimensional
vector embeddings are closest to a query vector, when compared with a given
distance metric, is an essential database workload for machine learning
embeddings, semantic search, recommendation engines, and retrieval-augmented
generation. Modern data platforms across relational, document, and distributed
systems provide native vector search capabilities.
This issue proposes native vector indexing and similarity search capabilities
for Apache Phoenix. Phoenix on HBase is well structured to support distributed
vector search. Horizontal partitioning across regions and distributed top-_k_
aggregation across shards are already implemented through HBase region splits
and Phoenix server-side order by, limit, and top-_N_ push down execution.
The implementation introduces a fixed-dimension {{VECTOR}} type utilizing
Phoenix's existing packed fixed-width array storage, built-in vector distance
functions, and an exact top-_k_ search that executes via the existing
server-side top-_N_ scatter-gather engine. This establishes the SQL interface
and provides exact nearest-neighbor search and retrieval capabilities. Building
on this, it implements a clustering vector index, in which posting lists map
directly to contiguous range scans on a covered global index keyed by centroid
identifiers. This approach reuses Phoenix's global index mutation pipeline,
IndexMaintainer, verified write protocols, read repair mechanisms, and range
scan infrastructure. It also implements support for optional per-region
immutable HNSW proximity graphs generated during the flush and compaction
lifecycles, similar in many respects to Phoenix local indexes, providing high
recall at low query latency. Finally, it adds a half-precision data type,
scalar quantization, binary quantization, product quantization, and an on-disk
graph layout for very high scale deployments.
Design Document:
https://docs.google.com/document/d/139E87IC2b6UG5T0UaxOK64qAjPt2u-ePB7DirWDm3v0/edit?tab=t.0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
