[
https://issues.apache.org/jira/browse/PHOENIX-7998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated PHOENIX-7998:
-----------------------------------------
Description:
Vector similarity search uses mathematical embeddings to identify records by
semantic meaning and conceptual proximity rather than exact keyword matches. In
a hybrid document database like Apache Phoenix, native vector search allows
applications to execute unified queries that combine AI-driven semantic
retrieval with relational filters and nested BSON document predicates. This
eliminates the operational overhead, data replication, and synchronization lag
of maintaining external vector databases while leveraging Phoenix's existing
scale-out query parallelism.
This design integrates native vector similarity search into Apache Phoenix
across relational columns and BSON documents using fixed-dimension vector
types, typed path extraction expressions, and standard SQL distance functions.
Exact search pushes distance evaluation and bounded candidate heaps down to
RegionServers for distributed top-N execution, while approximate retrieval maps
Inverted File centroid posting lists into contiguous global index ranges
traversed via multi-range skip scans. Server-side coprocessors maintain
centroid assignments and verified-write consistency across relational mutations
and atomic BSON updates, background compactions observe centroid drift to
trigger asynchronous rebuilds, and the optimizer balances exact evaluation,
covered index scanning, and adaptive probing within an extensible architecture.
Design Document:
https://docs.google.com/document/d/139E87IC2b6UG5T0UaxOK64qAjPt2u-ePB7DirWDm3v0/edit?tab=t.0
was:
Vector similarity search, identifying the top _k_ rows whose high dimensional
vector embeddings are closest to a query vector, when compared with a given
distance metric, is an essential database workload for machine learning
embeddings, semantic search, recommendation engines, and retrieval-augmented
generation. Modern data platforms across relational, document, and distributed
systems provide native vector search capabilities.
This issue proposes native vector indexing and similarity search capabilities
for Apache Phoenix. Phoenix on HBase is well structured to support distributed
vector search. Horizontal partitioning across regions and distributed top-_k_
aggregation across shards are already implemented through HBase region splits
and Phoenix server-side order by, limit, and top-_N_ push down execution.
The implementation introduces a fixed-dimension {{VECTOR}} type utilizing
Phoenix's existing packed fixed-width array storage, built-in vector distance
functions, and an exact top-_k_ search that executes via the existing
server-side top-_N_ scatter-gather engine. This establishes the SQL interface
and provides exact nearest-neighbor search and retrieval capabilities. Building
on this, it implements a clustering vector index, in which posting lists map
directly to contiguous range scans on a covered global index keyed by centroid
identifiers. This approach reuses Phoenix's global index mutation pipeline,
IndexMaintainer, verified write protocols, read repair mechanisms, and range
scan infrastructure. It also implements support for optional per-region
immutable HNSW proximity graphs generated during the flush and compaction
lifecycles, similar in many respects to Phoenix local indexes, providing high
recall at low query latency. Finally, it adds a half-precision data type,
scalar quantization, binary quantization, product quantization, and an on-disk
graph layout for very high scale deployments.
Design Document:
https://docs.google.com/document/d/139E87IC2b6UG5T0UaxOK64qAjPt2u-ePB7DirWDm3v0/edit?tab=t.0
> Vector Indexes
> --------------
>
> Key: PHOENIX-7998
> URL: https://issues.apache.org/jira/browse/PHOENIX-7998
> Project: Phoenix
> Issue Type: New Feature
> Components: core
> Reporter: Andrew Kyle Purtell
> Priority: Major
>
> Vector similarity search uses mathematical embeddings to identify records by
> semantic meaning and conceptual proximity rather than exact keyword matches.
> In a hybrid document database like Apache Phoenix, native vector search
> allows applications to execute unified queries that combine AI-driven
> semantic retrieval with relational filters and nested BSON document
> predicates. This eliminates the operational overhead, data replication, and
> synchronization lag of maintaining external vector databases while leveraging
> Phoenix's existing scale-out query parallelism.
> This design integrates native vector similarity search into Apache Phoenix
> across relational columns and BSON documents using fixed-dimension vector
> types, typed path extraction expressions, and standard SQL distance
> functions. Exact search pushes distance evaluation and bounded candidate
> heaps down to RegionServers for distributed top-N execution, while
> approximate retrieval maps Inverted File centroid posting lists into
> contiguous global index ranges traversed via multi-range skip scans.
> Server-side coprocessors maintain centroid assignments and verified-write
> consistency across relational mutations and atomic BSON updates, background
> compactions observe centroid drift to trigger asynchronous rebuilds, and the
> optimizer balances exact evaluation, covered index scanning, and adaptive
> probing within an extensible architecture.
> Design Document:
> https://docs.google.com/document/d/139E87IC2b6UG5T0UaxOK64qAjPt2u-ePB7DirWDm3v0/edit?tab=t.0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
