leaves12138 commented on code in PR #61:
URL:
https://github.com/apache/paimon-vector-index/pull/61#discussion_r3629927505
##########
docs/api.html:
##########
@@ -0,0 +1,273 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to you under the Apache License, Version 2.0.
+-->
+<!doctype html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <meta name="description" content="Unified Paimon Vector Index APIs for Rust,
C, C++, Java, and Python, including warm-up and metadata filter pushdown.">
+ <title>API and Language Bindings · Paimon Vector Index</title>
+ <link rel="stylesheet" href="styles.css">
+ <script src="docs.js" defer></script>
+</head>
+<body>
+ <a class="skip-link" href="#main">Skip to content</a>
+ <header class="site-header"><div class="header-inner">
+ <a class="brand" href="index.html" aria-label="Paimon Vector Index
documentation home"><span class="brand-mark">VI</span><span>Paimon Vector
Index</span></a>
+ <nav class="site-nav" data-site-nav aria-label="Documentation"><a
href="index.html">Overview</a><a href="api.html" aria-current="page">API</a><a
href="development.html">Development</a><a href="ivf-flat.html">IVF-FLAT</a><a
href="ivf-pq.html">IVF-PQ</a><a href="ivf-rq.html">IVF-RQ</a><a
href="ivf-hnsw-flat.html">HNSW-FLAT</a><a
href="ivf-hnsw-sq.html">HNSW-SQ</a></nav>
+ <div class="header-actions"><button class="icon-button" type="button"
data-theme-toggle aria-label="Switch color theme">◐</button><button
class="nav-toggle" type="button" data-nav-toggle aria-expanded="false"
aria-label="Open navigation">☰</button></div>
+ </div></header>
+
+ <main id="main"><div class="page-shell">
+ <section class="hero detail-hero">
+ <p class="eyebrow">One lifecycle · five language surfaces</p>
+ <h1>API and language bindings</h1>
+ <p class="hero-lead">The Rust core, C FFI, C++ RAII layer, Java/JNI, and
Python ctypes package share one train → write → serialize → detect → search
model. Build options select the index type; Readers discover it from the file
magic.</p>
+ <div class="badge-row"><span class="badge strong">Unified
lifecycle</span><span class="badge">Positional I/O</span><span
class="badge">Single and batch search</span><span class="badge">Roaring64
prefilter</span></div>
+ </section>
+
+ <div class="doc-layout">
+ <aside class="toc" aria-label="On this page"><strong>On this
page</strong><a href="#workspace">Modules</a><a
href="#lifecycle">Lifecycle</a><a href="#params">Search parameters</a><a
href="#warmup">Warm-up</a><a href="#rust">Rust</a><a href="#c">C FFI</a><a
href="#cpp">C++</a><a href="#java">Java/JNI</a><a href="#python">Python</a><a
href="#filter">Filter pushdown</a></aside>
+ <article class="article">
+ <section class="article-section" id="workspace">
+ <h2>Public integration layers</h2>
+ <div
class="table-wrap"><table><thead><tr><th>Module</th><th>Role</th><th>Primary
entry point</th></tr></thead><tbody>
+ <tr><td><code>core</code></td><td>Pure Rust indexes, file
Readers/Writers, and
benchmarks</td><td><code>paimon_vindex_core::index</code></td></tr>
+ <tr><td><code>ffi</code></td><td>C ABI over the Rust core; builds
<code>libpaimon_vindex_ffi</code></td><td>Generated
<code>include/paimon_vindex.h</code></td></tr>
+ <tr><td><code>include</code></td><td>C++ RAII
wrapper</td><td><code>paimon_vindex.hpp</code></td></tr>
+ <tr><td><code>jni</code> + <code>java</code></td><td>JNI
implementation and Java
API</td><td><code>org.apache.paimon.index.vector</code></td></tr>
+ <tr><td><code>python</code></td><td>Pure Python package loading
the C FFI through ctypes</td><td><code>paimon_vindex</code></td></tr>
+ </tbody></table></div>
+ <p>The top-level Cargo workspace contains <code>core</code>,
<code>ffi</code>, and <code>jni</code>. The Python package loads the shared FFI
library at runtime.</p>
+ </section>
+
+ <section class="article-section" id="lifecycle">
+ <h2>Shared lifecycle</h2>
+ <div class="flow" aria-label="Unified API lifecycle"><div
class="flow-step"><small>01</small><strong>Create a Trainer<br>Parse and
validate options</strong></div><div
class="flow-step"><small>02</small><strong>Submit one or more<br>training
batches</strong></div><div class="flow-step"><small>03</small><strong>Finish
training and<br>create a one-shot Writer</strong></div><div
class="flow-step"><small>04</small><strong>Add row IDs / vectors<br>and write
the file</strong></div><div class="flow-step"><small>05</small><strong>Detect
file magic<br>and execute searches</strong></div></div>
+ <ul><li>Vectors are contiguous <code>f32</code> values; length must
equal <code>vector_count × dimension</code>.</li><li>Training data may arrive
in batches, but the Trainer accumulates the full set. Batching avoids oversized
language-level arrays; it does not reduce native peak memory.</li><li>A Writer
may receive production vectors in multiple batches. Row-ID count must equal
vector count.</li><li>Readers expose metadata, single-query search, batch
search, and Roaring64-filtered variants.</li><li>Files carry their type and
model sections. Callers do not pass index options again when opening a
Reader.</li></ul>
+ </section>
+
+ <section class="article-section" id="params">
+ <h2>Shared search parameters</h2>
+ <div
class="table-wrap"><table><thead><tr><th>Parameter</th><th>Applies
to</th><th>Description</th></tr></thead><tbody>
+ <tr><td><code>top_k</code></td><td>All indexes</td><td>Number of
nearest neighbors returned for each query.</td></tr>
+ <tr><td><code>nprobe</code></td><td>All indexes</td><td>Number of
IVF lists probed. Larger values usually improve recall while increasing I/O and
compute.</td></tr>
+ <tr><td><code>ef_search</code></td><td>IVF-HNSW-FLAT /
SQ</td><td>Per-list graph-search breadth. <code>0</code> resolves to
<code>max(top_k, 32)</code>.</td></tr>
+ <tr><td><code>query_bits</code></td><td>IVF-RQ
only</td><td><code>0</code> keeps the float-query byte-LUT path; <code>4</code>
/ <code>8</code> use quantized-query bit-plane scans.</td></tr>
+ </tbody></table></div>
+ <div class="callout warning"><strong>Stored RQ bits and query bits
are different</strong>IVF-RQ currently writes 1-bit vector codes.
<code>query_bits</code> is a per-request query quantization setting. Current
Readers reject reserved multi-bit file payloads.</div>
+ </section>
+
+ <section class="article-section" id="warmup">
+ <h2>Search warm-up</h2>
+ <p>After opening a Reader and before repeated searches, call the
language-specific warm-up method. It builds process-local caches without
changing the file or results.</p>
+ <div
class="table-wrap"><table><thead><tr><th>Language</th><th>Method</th><th>Current
behavior</th></tr></thead><tbody>
+ <tr><td>Rust</td><td><code>optimize_for_search</code></td><td
rowspan="5">IVF-PQ builds residual-L2 precomputed tables. IVF-HNSW-SQ builds SQ
decode LUTs for filtered scans and fallback. Other indexes primarily preload
metadata.</td></tr>
+
<tr><td>C</td><td><code>paimon_vindex_reader_optimize_for_search</code></td></tr><tr><td>C++</td><td><code>optimize_for_search</code></td></tr><tr><td>Java</td><td><code>optimizeForSearch</code></td></tr><tr><td>Python</td><td><code>optimize_for_search</code></td></tr>
+ </tbody></table></div>
+ <p>The IVF-HNSW-SQ LUT is not used by normal unfiltered graph
traversal, so warm-up is not expected to accelerate that path.</p>
+ </section>
+
+ <section class="article-section" id="rust">
+ <h2>Rust</h2>
+ <div class="code-block"><span class="code-label">Rust · train,
write, and search</span><pre><code>use std::fs::File;
+
+use paimon_vindex_core::distance::MetricType;
+use paimon_vindex_core::hnsw::HnswBuildParams;
+use paimon_vindex_core::index::{
+ VectorIndexConfig, VectorIndexReader, VectorIndexTrainer,
+ VectorIndexWriter, VectorSearchParams,
+};
+use paimon_vindex_core::io::PosWriter;
+
+let config = VectorIndexConfig::IvfHnswSq {
+ dimension: 128,
+ nlist: 1024,
+ metric: MetricType::L2,
+ hnsw: HnswBuildParams::default(),
+};
+
+let training = VectorIndexTrainer::train(
+ config, &training_vectors, training_count)?;
+let mut writer = VectorIndexWriter::new(training);
+writer.add_vectors(&row_ids, &vectors, vector_count)?;
+
+let mut file = File::create("vectors.pvindex")?;
+let mut out = PosWriter::new(&mut file);
+writer.write(&mut out)?;
+
+let file = File::open("vectors.pvindex")?;
+let mut reader = VectorIndexReader::open(file)?;
+reader.optimize_for_search()?;
+let params = VectorSearchParams::with_ef_search(10, 16, 80);
+let (ids, distances) = reader.search(&query, params)?;</code></pre></div>
+ <div class="code-block"><span class="code-label">Rust · other
configurations</span><pre><code>VectorIndexConfig::IvfFlat {
+ dimension: 128, nlist: 1024, metric: MetricType::L2,
+};
+VectorIndexConfig::IvfPq {
+ dimension: 128, nlist: 1024, m: 16,
+ metric: MetricType::L2, use_opq: false,
+};
+VectorIndexConfig::IvfRq {
+ dimension: 128, nlist: 1024, metric: MetricType::L2,
+};
+VectorIndexConfig::IvfHnswFlat {
+ dimension: 128, nlist: 1024, metric: MetricType::L2,
+ hnsw: HnswBuildParams::default(),
+};</code></pre></div>
+ </section>
+
+ <section class="article-section" id="c">
+ <h2>C FFI</h2>
+ <p>The C ABI builds <code>libpaimon_vindex_ffi</code>; cbindgen
produces the public header. Functions return <code>0</code> on success and
<code>-1</code> on failure. <code>paimon_vindex_last_error()</code> exposes a
thread-local error string.</p>
Review Comment:
The `0`/`-1` convention only applies to status-returning operations.
`paimon_vindex_trainer_open`, `paimon_vindex_trainer_finish`,
`paimon_vindex_writer_open`, and `paimon_vindex_reader_open` return a handle or
null; the free functions return `void`; and `paimon_vindex_last_error()`
returns a string pointer. Could we describe the status-returning and
handle-returning conventions separately?
##########
docs/development.html:
##########
@@ -0,0 +1,126 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to you under the Apache License, Version 2.0.
+-->
+<!doctype html>
+<html lang="en">
+<head>
+ <meta charset="utf-8">
+ <meta name="viewport" content="width=device-width, initial-scale=1">
+ <meta name="description" content="Paimon Vector Index repository layout,
development checks, cross-language tests, and ANN benchmarks.">
+ <title>Development and Benchmarks · Paimon Vector Index</title>
+ <link rel="stylesheet" href="styles.css">
+ <script src="docs.js" defer></script>
+</head>
+<body>
+ <a class="skip-link" href="#main">Skip to content</a>
+ <header class="site-header"><div class="header-inner">
+ <a class="brand" href="index.html" aria-label="Paimon Vector Index
documentation home"><span class="brand-mark">VI</span><span>Paimon Vector
Index</span></a>
+ <nav class="site-nav" data-site-nav aria-label="Documentation"><a
href="index.html">Overview</a><a href="api.html">API</a><a
href="development.html" aria-current="page">Development</a><a
href="ivf-flat.html">IVF-FLAT</a><a href="ivf-pq.html">IVF-PQ</a><a
href="ivf-rq.html">IVF-RQ</a><a href="ivf-hnsw-flat.html">HNSW-FLAT</a><a
href="ivf-hnsw-sq.html">HNSW-SQ</a></nav>
+ <div class="header-actions"><button class="icon-button" type="button"
data-theme-toggle aria-label="Switch color theme">◐</button><button
class="nav-toggle" type="button" data-nav-toggle aria-expanded="false"
aria-label="Open navigation">☰</button></div>
+ </div></header>
+
+ <main id="main"><div class="page-shell">
+ <section class="hero detail-hero"><p class="eyebrow">Build · test ·
measure</p><h1>Development and benchmarks</h1><p class="hero-lead">Run the
standard Rust checks, exercise the C/C++, Java, and Python integrations, and
measure ANN and filtered-query behavior with reproducible workloads.</p><div
class="badge-row"><span class="badge strong">Cargo workspace</span><span
class="badge">CMake smoke tests</span><span class="badge">Maven /
pytest</span><span class="badge">Criterion benchmarks</span></div></section>
+
+ <div class="doc-layout">
+ <aside class="toc" aria-label="On this page"><strong>On this
page</strong><a href="#workspace">Repository layout</a><a href="#rust">Rust
checks</a><a href="#ann">ANN benchmark</a><a href="#filter-bench">Filter
benchmark</a><a href="#c">C FFI</a><a href="#cpp">C++</a><a
href="#java">Java/JNI</a><a href="#python">Python</a><a href="#format">Format
compatibility</a></aside>
+ <article class="article">
+ <section class="article-section" id="workspace">
+ <h2>Repository layout</h2>
+ <div
class="table-wrap"><table><thead><tr><th>Directory</th><th>Contents</th><th>Primary
verification</th></tr></thead><tbody>
+ <tr><td><code>core/</code></td><td>Indexes, quantizers, HNSW, I/O,
format fixtures, and benchmarks</td><td><code>cargo test -p
paimon-vindex-core</code></td></tr>
+ <tr><td><code>ffi/</code></td><td>C ABI and shared
library</td><td>C smoke test</td></tr>
+ <tr><td><code>include/</code></td><td>C++ RAII header</td><td>C++
smoke test</td></tr>
+ <tr><td><code>jni/</code></td><td>Rust JNI bridge</td><td>Java
Maven tests</td></tr>
+ <tr><td><code>java/</code></td><td>Java public API and
tests</td><td><code>mvn test</code></td></tr>
+ <tr><td><code>python/</code></td><td>ctypes package and pytest
suite</td><td><code>pytest</code></td></tr>
+ <tr><td><code>c/</code> / <code>cpp/</code></td><td>Integration
smoke-test projects</td><td>CMake</td></tr>
+ <tr><td><code>docs/</code></td><td>Index, API, and development
documentation</td><td>Static link and structure checks</td></tr>
+ </tbody></table></div>
+ </section>
+
+ <section class="article-section" id="rust">
+ <h2>Standard Rust checks</h2>
+ <div class="code-block"><span
class="code-label">Shell</span><pre><code>cargo fmt --all
+cargo test --workspace
+cargo clippy --workspace --all-targets</code></pre></div>
+ <p>Before submitting a change, keep formatting, workspace tests, and
Clippy across all targets clean. Changes to binary layouts should also run the
core storage-format fixtures.</p>
+ </section>
+
+ <section class="article-section" id="ann">
+ <h2>ANN benchmark</h2>
+ <p><code>ann_bench</code> compares IVF-PQ, IVF-RQ, IVF-HNSW-FLAT,
and IVF-HNSW-SQ under one workload. It reports build time, Reader open/load
time, first-query latency, batch throughput, and serialized index size.</p>
+ <div class="code-block"><span class="code-label">Shell ·
defaults</span><pre><code>cargo bench -p paimon-vindex-core --bench ann_bench
-- --nocapture</code></pre></div>
+ <div class="code-block"><span class="code-label">Shell · explicit
workload</span><pre><code>ANN_N=100000 ANN_NQ=1000 ANN_D=128 ANN_K=10 \
+ANN_NLIST=256 ANN_NPROBE=16 ANN_PQ_M=16 \
+ANN_HNSW_M=20 ANN_HNSW_EF_CONSTRUCTION=150 \
+ANN_HNSW_EF_SEARCH=80 ANN_RQ_QUERY_BITS=0 \
+cargo bench -p paimon-vindex-core --bench ann_bench --
--nocapture</code></pre></div>
+ <div
class="table-wrap"><table><thead><tr><th>Variable</th><th>Meaning</th></tr></thead><tbody><tr><td><code>ANN_N
/ ANN_NQ</code></td><td>Indexed vector count / query
count</td></tr><tr><td><code>ANN_D / ANN_K</code></td><td>Dimension / top
K</td></tr><tr><td><code>ANN_NLIST / ANN_NPROBE</code></td><td>IVF list count /
probed lists</td></tr><tr><td><code>ANN_PQ_M</code></td><td>IVF-PQ subquantizer
count</td></tr><tr><td><code>ANN_HNSW_*</code></td><td>HNSW build and query
parameters</td></tr><tr><td><code>ANN_RQ_QUERY_BITS</code></td><td>IVF-RQ query
quantization: 0 / 4 / 8</td></tr></tbody></table></div>
+ <p><code>disk_scope=index_bytes</code> means the serialized
vector-index file itself, excluding the outer Paimon file and manifest.</p>
+ </section>
+
+ <section class="article-section" id="filter-bench">
+ <h2>IVF-HNSW-SQ filter benchmark</h2>
+ <p>The filter-heavy benchmark measures batch filtered search before
and after warm-up and verifies that both paths return identical results. It
isolates the SQ decode LUT used by scan and fallback paths.</p>
+ <div class="code-block"><span class="code-label">Shell ·
defaults</span><pre><code>cargo bench -p paimon-vindex-core \
+ --bench ivfhnswsq_filter_bench -- --nocapture</code></pre></div>
+ <div class="code-block"><span class="code-label">Shell · explicit
workload</span><pre><code>FILTER_BENCH_N=50000 FILTER_BENCH_NQ=500
FILTER_BENCH_D=128 \
+FILTER_BENCH_NLIST=64 FILTER_BENCH_NPROBE=32 \
+FILTER_BENCH_EF_SEARCH=80 FILTER_BENCH_FILTER_STRIDES=1,4,16,64 \
+cargo bench -p paimon-vindex-core \
+ --bench ivfhnswsq_filter_bench -- --nocapture</code></pre></div>
+ <p><code>FILTER_BENCH_FILTER_STRIDES</code> simulates different
selectivities through row-ID strides. Analyze direct scans, graph search, and
insufficient-result backfill separately.</p>
+ </section>
+
+ <section class="article-section" id="c">
+ <h2>C FFI smoke test</h2>
+ <p>Build the release shared library, then compile the C test against
the generated header.</p>
+ <div class="code-block"><span
class="code-label">Shell</span><pre><code>cargo build --release -p
paimon-vindex-ffi
+cmake -S c -B c/build
+cmake --build c/build
+LD_LIBRARY_PATH=target/release c/build/test_vindex</code></pre></div>
+ <div class="callout"><strong>Platform
note</strong><code>LD_LIBRARY_PATH</code> is the Linux loader variable. Use the
equivalent loader configuration on other platforms.</div>
+ </section>
+
+ <section class="article-section" id="cpp">
+ <h2>C++ smoke test</h2>
+ <p>The C++ test reuses the same FFI shared library and validates the
RAII header.</p>
+ <div class="code-block"><span
class="code-label">Shell</span><pre><code>cargo build --release -p
paimon-vindex-ffi
+cmake -S cpp -B cpp/build
+cmake --build cpp/build
+LD_LIBRARY_PATH=target/release cpp/build/test_vindex_cpp</code></pre></div>
+ </section>
+
+ <section class="article-section" id="java">
+ <h2>Java / JNI tests</h2>
+ <p>Run the Java API tests from the JNI Java module:</p>
+ <div class="code-block"><span
class="code-label">Shell</span><pre><code>mvn -f java/pom.xml
test</code></pre></div>
+ <p>The suite covers option validation, Reader/Writer lifecycle,
native panic boundaries, handle safety, metadata, single and batch search, and
filter overloads.</p>
Review Comment:
`mvn -f java/pom.xml test` only runs `VectorIndexJavaApiTest` through the
Maven exec plugin. The native validation, panic-boundary, and handle-safety
classes are standalone `main` programs that CI runs only after building
`paimon-vindex-jni`. Please either include those build/run commands here or
narrow this coverage claim to what the documented Maven command actually
executes.
##########
README.md:
##########
@@ -23,436 +23,48 @@
[actions]:
https://github.com/apache/paimon-vector-index/actions?query=branch%3Amain
Apache Paimon Vector Index is a pure Rust vector indexing library designed for
-Apache Paimon and data lake storage such as S3, HDFS, and OSS. Index readers
use
-seek-based positional I/O so query execution can read only the parts of an
index
-file needed by the selected IVF lists.
+Apache Paimon and data lake storage such as S3, HDFS, and OSS. Its seek-based
+readers load only the IVF lists selected by a query.
-The project is no longer limited to IVF-PQ. The unified writer and reader APIs
-support multiple index families across Rust, C FFI, Java/JNI, and Python:
+The library supports IVF-FLAT, IVF-PQ, IVF-RQ, IVF-HNSW-FLAT, and
+IVF-HNSW-SQ through shared Rust, C, C++, Java/JNI, and Python APIs.
-| Index type | Summary | Best fit |
-| --- | --- | --- |
-| `IVF_FLAT` | IVF partitioning with uncompressed vectors. | Baseline recall
and simple storage. |
-| `IVF_PQ` | IVF with product quantization and optional OPQ rotation. |
Compact indexes with fast approximate scans. |
-| `IVF_RQ` | IVF with 1-bit RaBitQ-style rotated residual quantization. | Very
compact high-dimensional indexes with low training cost. |
-| `IVF_HNSW_FLAT` | IVF partitioning with an HNSW graph inside each list over
raw vectors. | Higher recall within probed IVF lists. |
-| `IVF_HNSW_SQ` | IVF partitioning with per-list HNSW and scalar-quantized
vectors. | HNSW-style search with smaller vector storage. |
+## Documentation
-All index types share:
+- [Index selection and architecture](docs/index.html): compare all index
Review Comment:
These links currently open GitHub blob views, where the committed HTML is
shown as source rather than as the styled site. I also could not find a Pages
workflow, Pages configuration, or an `asf-site` branch in this repository.
Since this PR removes most of the user-facing README content in favor of these
pages, could we either publish `docs/` and link to the deployed URL, document a
usable preview path, or keep the entry-point documentation in Markdown?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
