JingsongLi commented on code in PR #101:
URL:
https://github.com/apache/paimon-vector-index/pull/101#discussion_r4489406903
##########
python/paimon_vindex/__init__.py:
##########
@@ -386,6 +386,123 @@ def __del__(self):
pass
+@dataclass(frozen=True)
+class PreparedTrainingInfo:
+ dimension: int
+ nlist: int
+ sample_count: int
+ calibration_count: int
+ vectors_seen: int
+ iterations: int
+ restarts: int
+ seed: int
+ metric: str
+
+
+class PreparedIvfSqTraining:
+ """Owned IVF-SQ sample for external centroid training.
+
+ Samples are already preprocessed (normalized for cosine). Fit centers with
+ squared L2 even for IP/cosine indexes. Finishing calibrates residual SQ on
+ CPU and consumes the native state, including on native validation failure.
+ """
+
+ def __init__(self, handle):
+ self._native_handle_lock = _NativeHandleLock()
Review Comment:
[P2] Guard the full prepared-handle ownership transfer
`prepare_training()` has already obtained a native prepared handle and
freed/closed its trainer before constructing this wrapper. If
`_NativeHandleLock()` raises `MemoryError` here, neither the lock nor `_handle`
is installed; `__del__()` calls `close()`, which raises `AttributeError` and
suppresses it, leaving the native training reservoir permanently leaked. With
the real dylib, injecting this allocation failure and running `gc.collect()`
produced `created=1, freed=0, trainer_closed=True, trainer_handle=None`. This
is particularly costly in a training worker already under memory pressure.
Please guard wrapper construction at the raw-handle ownership boundary,
including failure in `__new__`, and free the prepared handle exactly once on
`BaseException`. Add fault-injection coverage before and after handle
assignment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]