zhiqiang-hhhh opened a new pull request, #56649:
URL: https://github.com/apache/doris/pull/56649
### What problem does this PR solve?
This pull request introduces new metrics for monitoring the in-memory state
and construction of ANN (Approximate Nearest Neighbor) indexes, and integrates
these metrics into the lifecycle of vector indexes. It also ensures that
metrics are updated appropriately when indexes are created, destroyed, loaded,
or modified.
**Metrics and Monitoring Enhancements:**
* Added three new metrics to `DorisMetrics`: `ann_index_in_memory_cnt`,
`ann_index_in_memory_rows_cnt`, and `ann_index_construction`, to track the
number of in-memory ANN indexes, the number of rows in memory, and the number
of ongoing index constructions, respectively.
[[1]](diffhunk://#diff-878c2670a099f34646a2d514e473068af8a5c43784e0450555a2e2b79e8f27caR248-R250)
[[2]](diffhunk://#diff-878c2670a099f34646a2d514e473068af8a5c43784e0450555a2e2b79e8f27caR414-R416)
[[3]](diffhunk://#diff-86758ef39527aa2e06d142689e6b13e5a89872ca7aaffb398d1e491749efbd52R256-R258)
* Registered the new metrics in the metrics registry and included their
prototypes for proper initialization and tracking.
[[1]](diffhunk://#diff-878c2670a099f34646a2d514e473068af8a5c43784e0450555a2e2b79e8f27caR414-R416)
[[2]](diffhunk://#diff-878c2670a099f34646a2d514e473068af8a5c43784e0450555a2e2b79e8f27caR248-R250)
**Integration with ANN Index Lifecycle:**
* Modified the `VectorIndex` base class to increment and decrement
`ann_index_in_memory_cnt` upon construction and destruction, respectively,
ensuring accurate tracking of in-memory index objects.
[[1]](diffhunk://#diff-b8d71062ef0c17d6d0ddb9690fefa383984d2bfb1ecbde1a34fe88c224e741a7L77-R78)
[[2]](diffhunk://#diff-4186cdac2d23477966f836c52ec5f5bc6cc6d9c015ffc0b09616b1fe47eedd57R65-R72)
* Updated `FaissVectorIndex` to:
- Inherit from the updated `VectorIndex` and override the destructor to
decrement `ann_index_in_memory_rows_cnt` based on the number of rows in the
index.
[[1]](diffhunk://#diff-db0f703076eba6a2e532676979b1bfb928a2ffda3123593779d49b10b143d533L73-R80)
[[2]](diffhunk://#diff-65584774f4b13280356d4cfc388080b0ae4215dff6ed79b1c711fae83f2e8cbaR173-R174)
- Increment `ann_index_in_memory_rows_cnt` when vectors are added or when
an index is loaded, and update `ann_index_construction` during index
construction.
[[1]](diffhunk://#diff-db0f703076eba6a2e532676979b1bfb928a2ffda3123593779d49b10b143d533R173-R176)
[[2]](diffhunk://#diff-db0f703076eba6a2e532676979b1bfb928a2ffda3123593779d49b10b143d533R495)
**Codebase Maintenance:**
* Added necessary includes for `doris_metrics.h` and `metrics.h` in relevant
files to support the new metrics functionality.
[[1]](diffhunk://#diff-4186cdac2d23477966f836c52ec5f5bc6cc6d9c015ffc0b09616b1fe47eedd57R20)
[[2]](diffhunk://#diff-db0f703076eba6a2e532676979b1bfb928a2ffda3123593779d49b10b143d533R44)
[[3]](diffhunk://#diff-878c2670a099f34646a2d514e473068af8a5c43784e0450555a2e2b79e8f27caR35)
<img width="1104" height="1220" alt="image"
src="https://github.com/user-attachments/assets/f1a800fc-0d3a-4f24-b264-0ed5e043fc54";
/>
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
