Doris-Breakwater commented on issue #70440: URL: https://github.com/apache/doris/issues/70440#issuecomment-6766299348
Breakwater-GitHub-Analysis-Slot: slot_4456213adab7 Initial triage: the source supports the reported policy-propagation gaps and their ability to cause query/load failures. The proposed two PRs are reasonable to review separately, with explicit limits on what each guarantees. Suggested routing: FE report/agent-task handling and BE inverted-index policy lifecycle. The issue currently has no labels. **Verified from source** I inspected `4.1.4` (`ad35a140c7fd`), `4.1.5-release` (`917b20be8387`), master `08231e3a6359`, and branch-4.1 at `6acf3dad10bd`. The relevant propagation mechanism is present in all four; this does not mean their entire analyzer implementations are identical. - On restart, BE policies start in process-local maps. Heartbeat handling can notify report workers before the heartbeat response has been processed by the FE. `ReportHandler` computes the policy diff and submits a batch, while `AgentBatchTask.run()` skips an existing backend whose `isAlive()` is false. This report-driven path does not register the push in `AgentTaskQueue` for automatic resend; a later report provides reconciliation. See [report diff and dispatch](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/master/ReportHandler.java#L473-L550) and [the alive check](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/task/AgentBatchTask.java#L171-L235). - CREATE/DROP update FE metadata and journal it, but do not initiate a BE push. Thus successful DDL does not establish immediate BE visibility. See [CREATE](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java#L142-L187) and [DROP](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicyMgr.java#L422-L456). - `Backend.isQueryAvailable()` / `isLoadAvailable()` do not check policy readiness. Custom-analyzer creation reaches `IndexPolicyMgr::get_policy_by_name()`, which throws when the name is absent; writer initialization wraps that exception as the reported analyzer-creation error. See [BE lookup](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/runtime/index_policy/index_policy_mgr.cpp#L106-L134). **What remains unverified** The first push being skipped is a credible race, not an inevitable ordering on every restart. One detail to refine: `HeartbeatMgr` handles each response inside its futures loop; it does not wait until all responses have been processed before marking a particular BE alive. Report processing can nevertheless win that race. Two diff messages ten seconds apart alone do not prove why the first attempt failed. Ten seconds is the default report interval, not a guaranteed recovery upper bound under queueing or RPC failures. The before/after counts are useful reporter-provided evidence; I have not independently run those workloads or reviewed the patches. **PR review recommendations** 1. Scope the alive-check exception to report-triggered policy synchronization. Keep the backend-existence check and preserve other task types' behavior, including mixed batches. Repeated delivery of the same policy ID is skipped on BE, but that does not establish correct ordering for concurrent CREATE/DROP/recreate operations. This change removes the identified skip condition; it does not by itself establish policy readiness before scheduling. 2. Push committed DDL changes only from the active master, outside metadata locks, retaining report reconciliation. Define timeout behavior after the DDL is already durable, and cover dependent tokenizer/filter policies and concurrent report-driven pushes. 3. Distinguish **queued** from **applied**. `submit_tasks` queues work, and `push_index_policy_callback` applies it asynchronously without a completion report ([worker submission](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/agent/task_worker_pool.cpp#L554-L564), [policy callback](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/be/src/agent/task_worker_pool.cpp#L1823-L1827)). Waiting for enqueueing can reduce failures but cannot guarantee immediate post-DDL queries/loads succeed. Any stronger guarantee needs application confirmation or a readiness barrier, including required policy dependencies. Please open the proposed PRs with the exact analyzer/tokenizer/filter DDL, table DDL, query/load scripts, patch commits, and FE/BE build hashes. For one failing restart, attach a timestamped timeline correlated by backend ID/start time: first INDEX_POLICY report, diff, task submission/skip or RPC failure, FE alive transition, BE policy application, and first failing/successful request. Include the configured report interval and FE/BE counts. A query profile is not necessary to establish this propagation race. Regression coverage should force the report-before-alive ordering and delay BE policy application, then exercise immediate CREATE → INSERT/MATCH on multiple BEs, dependent policies, DROP/recreate ordering, failed delivery/reconciliation, and master failover/replay without duplicate broadcasts. A successful probe on one BE does not demonstrate cluster-wide readiness. Until then, wait for application of the required policies and dependencies on every target BE before resuming dependent traffic, and use bounded retries appropriate to query/load semantics. A fixed ten-second sleep is not a readiness check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
