leaves12138 commented on code in PR #62:
URL:
https://github.com/apache/paimon-vector-index/pull/62#discussion_r3651353690
##########
include/paimon_vindex.hpp:
##########
@@ -328,11 +360,24 @@ class Writer {
class Reader {
public:
- explicit Reader(InputFile input) :
input_(std::make_shared<InputFile>(std::move(input))) {
+ explicit Reader(InputFile input)
+ : Reader(
+ std::move(input),
+ StorageProfile::Auto,
+ static_cast<size_t>(4ULL * 1024 * 1024 * 1024)) {}
+
+ Reader(InputFile input, StorageProfile profile, size_t memory_budget_bytes)
+ : input_(std::make_shared<InputFile>(std::move(input))) {
PaimonVindexInputFile raw;
raw.ctx = input_.get();
raw.read_ranges_fn = detail::input_read_ranges;
- handle_ = paimon_vindex_reader_open(raw);
+ raw.preferred_alignment_bytes = input_->preferred_alignment_bytes;
+ raw.preferred_window_bytes = input_->preferred_window_bytes;
+ raw.max_ranges_per_read = input_->max_ranges_per_read;
+ PaimonVindexReaderOptions options;
+ options.storage_profile = static_cast<uint32_t>(profile);
+ options.memory_budget_bytes = memory_budget_bytes;
+ handle_ = paimon_vindex_reader_open_with_options(raw, options);
Review Comment:
Could we serialize native-handle operations in the C++ `Reader` as the Java
wrapper now does (and cover destruction/move assignment as well)? The new
DiskANN reader mutates lazy initialization, caches, workers, calibrated width,
and search statistics during search, while each C entry point converts the same
raw handle into a Rust `&mut VectorIndexReader`. Two C++ threads calling
`search` on one `Reader` therefore create simultaneous mutable Rust references
and can corrupt the reader; I reproduced that failure through the Python
wrapper, which reaches the same C ABI. The callback thread-safety note only
makes concurrent callbacks from one search safe; it does not make concurrent
calls on the reader handle safe. If synchronization belongs to callers instead,
the C and C++ APIs need an explicit non-thread-safe contract so users do not
reasonably share this read-only-looking object across search threads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
