leaves12138 commented on code in PR #62:
URL:
https://github.com/apache/paimon-vector-index/pull/62#discussion_r3650874727
##########
jni/src/stream.rs:
##########
@@ -79,6 +80,28 @@ impl SeekRead for JniSeekableStream {
copy_java_buffers(&mut env, &buffers, ranges)
}
+
+ fn try_clone_reader(&self) -> io::Result<Option<Self>> {
Review Comment:
Before making the JNI reader cloneable for batched search, please bound the
local references created by one `pread` call. Dropping the Rust `JByteArray`
wrapper does not call `DeleteLocalRef`, so every `new_byte_array` remains live
until the native frame returns; `copy_java_buffers` then creates another local
reference per range with `get_object_array_element`. IVF search can pass one
range per probed/unique list, and the Java default `maxRangesPerRead() == 0`
means unlimited. I reproduced this with 50,000 single-row IVF lists and
`nprobe=50,000`: `-Xcheck:jni` reported the local-reference count growing to
roughly 100,000 during one search. This retains every Java byte array and can
overflow/cripple the local-reference table on large probes or batches. Please
chunk JNI range batches and/or use local frames / explicit `delete_local_ref`
in both loops.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
