Doris-Breakwater commented on issue #69816: URL: https://github.com/apache/doris/issues/69816#issuecomment-6490396710
Breakwater-GitHub-Analysis-Slot: slot_260f1ddb23e0 The reported ~0.1 s versus ~20 s latency difference deserves investigation, and the request for per-replica ANN observability is well motivated. However, the supplied evidence does **not yet establish index corruption/staleness or fragment-failure-driven failover**. These should remain hypotheses rather than the confirmed root cause. I checked the local source at the exact reported commit, `917b20be83878e98f94e3ebf7efccc80346e3e50`. This is a source-based assessment; I have not reproduced the workload or inspected its actual index files/profiles. The issue currently has no labels. **What the code establishes** 1. **Replica health is not ANN health.** `ADMIN SHOW REPLICA STATUS` checks backend/replica availability, data version, schema hash and replica flags; it does not validate each segment's ANN files. Thus `OK` does not establish ANN coverage or readability. Also, `VersionNum` comes from `getTotalVersionCount()`: it is a count of versions/rowsets, not an index generation or a monotonic measure of compaction progress. The values 21/14/12 alone cannot establish that A is further compacted than B/C. See [MetadataViewer](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/catalog/MetadataViewer.java#L191-L232). 2. **File timestamps do not establish index freshness relative to data.** ANN indexes belong to individual segments; replicas can have different rowset/segment layouts after independent compaction. A valid pre-restart index can be loaded without being rebuilt. Directory-wide `.idx` counts/mtimes need to be mapped to the *active* rowsets, segment IDs and ANN index ID, excluding stale files. There is also an intentional build-skip path for segments below `ann_index_build_min_segment_rows` (default 0 in this commit), so an absent index can have a policy explanation. See [ANN writer](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/storage/index/ann/ann_index_writer.cpp#L135-L160) and [ANN reader](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/storage/index/ann/ann_index_reader.cpp#L58-L106). 3. **A missing/unloadable ANN index can cause local brute-force fallback, without a fragment failure.** The Top-N segment path enables reading vector data, increments `ann_fall_back_brute_force_cnt`, and returns success when the index is unavailable or cannot load. Other search errors can still propagate; the actual status is needed to distinguish them. Crucially, **inner-product Top-N requires `DESC`** for this index path. The abbreviated SQL in the report omits it; literal ascending order falls back even with a healthy index. Please provide the complete executed SQL. Neither mechanism alone explains why only A receives scans. See [Top-N execution](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/storage/segment/segment_iterator.cpp#L883-L991). 4. **The two quoted logs do not prove failover.** `Query ... does not exists, failed to cancel it` means the cancellation handler could not find a query context. `id_file_map not found` explicitly handles a BE with no scan range by returning empty blocks and success; it is not proof that A executed the tablet scan. `SimpleScheduler.getLocation()` selects an available/non-blacklisted backend; it is not an ANN-aware execution-retry routine. In the inspected shared-nothing statement retry path, retry eligibility is principally an RPC exception, and a retry receives a **new query ID** with an audit-log mapping. Matching the same query ID across these two messages therefore does not demonstrate a retry. See [cancellation handler](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/runtime/fragment_mgr.cpp#L705-L722), [row-ID handler](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/exec/rowid_fetcher.cpp#L561-L571), [ scheduler](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/qe/SimpleScheduler.java#L245-L275), and [statement retries](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java#L1017-L1110). 5. **`BUILD INDEX` has a relevant limitation, but “only builds new rowsets” is inaccurate.** It scans existing active local rowsets and skips those whose schema already declares the requested index ID, including ANN. An empty candidate list means no rowsets qualified under that metadata test; it does not certify physical index integrity. Separately, the reported cancellation threshold is supported: `OBTAIN_LOCK_FAILED` gets additional attempts, then cancellation at 10 failures. The actual failed lock and job/task history are still needed to connect that event to this incident. See [candidate selection](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/storage/tablet/tablet.cpp#L1296-L1326), [ANN metadata recognition](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/storage/tablet/tablet_schema.cpp#L1473-L1482), and [retry threshold](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e 50/fe/fe-core/src/main/java/org/apache/doris/alter/IndexChangeJob.java#L420-L440). **Evidence needed to resolve the cause** - Complete SQL, `EXPLAIN VERBOSE`, exact FE and all BE build versions, client concurrency/retry behavior, and session settings, especially `use_fix_replica`, `enable_query_cache`, `skip_missing_version`, and any backend-selection settings. In this commit, enabling query cache with Nereids can select a fixed replica even when `use_fix_replica` was initially -1; this is a concrete alternative to investigate, not an established explanation. See [replica selection](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java#L511-L523). - Full profiles for the same query on A, B and C, with scan-instance backend addresses and query IDs. Preserve `AnnIndexTopNSearchCnt`, `AnnIndexFallbackBruteForceCnt`, `AnnIndexLoadCosts`, `AnnIndexTopNSearchCosts`, rows/bytes read and I/O timing. These [existing query counters](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/exec/operator/olap_scan_operator.cpp#L339-L394) provide execution evidence, although they are not a persistent per-replica health summary. - FE audit/execution logs and all three BE logs for one fast and one slow query, including the **first error/status**, cancellation reason, scan assignment, and any old-to-new retry query-ID mapping. Include `Failed to load ann index, will fallback to brute force search` messages if present. - For the same tablet and observation time: replica status, active rowset version intervals/IDs, per-segment row counts, index IDs/files, effective ANN build configuration, and compaction status/history before and after restart. Include complete `SHOW BUILD INDEX` output and the corresponding BE lock-failure messages. Repeated ~20 s timings weaken a simple one-time warm-up explanation but do not eliminate cache capacity/eviction, repeated load failure, I/O waits or brute-force execution. **Suggested maintainer next steps** Treat this as two investigations: replica scan placement, and per-replica ANN execution/coverage. On a controlled reproduction, keep all BEs running and compare the same SQL with session-scoped replica selection, verifying the actual scan BE in each profile. Then test ordinary selection separately. Preserve this evidence before rebuilding indexes or changing compaction behavior. The observability proposal would be more actionable as per-replica **active-segment/row coverage, intentional skip reasons, load/build errors, and last validation time**, rather than equality of a single “index version” across replicas with different physical layouts. Automatic repair should distinguish intentional omissions from missing/unreadable required indexes. Backoff/retry handling for build lock contention can be evaluated separately once the blocking lock is known. No specific fix or upgrade is established by the evidence currently available. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
