matanper opened a new issue, #70440: URL: https://github.com/apache/doris/issues/70440
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4 (cloud mode). master (`08231e3a635`), `branch-4.1` and `4.1.5-release` have the same code. ### Description Index policies (custom analyzers, tokenizers and token filters from `CREATE INVERTED INDEX ANALYZER/TOKENIZER/TOKEN_FILTER`) are held only in memory on the BE (`IndexPolicyMgr`). Their only delivery path is the BE's periodic report: every `report_index_policy_interval_seconds` (default 10) the BE reports its policy set, `ReportHandler` diffs it against the FE's, and the FE sends a `PUSH_INDEX_POLICY` agent task. This causes two windows in which queries and loads fail with `[E33] Policy not found with name: <analyzer>`. **1. After a BE restart, the first push is dropped.** - The first heartbeat wakes the BE's report worker (`HeartbeatServer`, `need_report`), and the BE reports an empty policy set. - The FE dispatches the push through `AgentBatchTask`, whose `run()` skips any backend that is not alive (`backend == null || !backend.isAlive()`). - The BE is not alive yet: `HeartbeatMgr` sets `isAlive` only after processing every heartbeat response. The push is discarded without a retry. - As soon as the BE becomes alive, `isQueryAvailable()` is true and fragments are scheduled on it. Every query that builds an analyzer for a custom-analyzer index fails until the next report, about 10 s later. The same goes for loads: the memtable flush fails with `[E-6011] inverted index create analyzer failed: [E33] Policy not found`, so stream loads fail and must be retried. - During rolling restarts of a cluster that uses custom analyzers, every BE restart produces about 10 s of these failures. **2. After `CREATE`/`DROP` of a policy, nothing is pushed.** - `IndexPolicyMgr.createIndexPolicy` / `dropIndexPolicy` only write the edit log. Live BEs learn about the change at their next report, up to 10 s later. - So an `INSERT` or a MATCH query run right after `CREATE INVERTED INDEX ANALYZER` fails with E33. ### Solution 1. For the `PUSH_INDEX_POLICY` batch triggered by a BE's own report, skip the `isAlive` check in `AgentBatchTask`. The report itself shows the BE process is up, and pushes are idempotent on the BE (`apply_policy_changes` skips policies it already has). Every other agent task keeps the check. 2. After a policy DDL commits on the master, push the change to all alive BEs right away, and keep the report diff as the reconciliation path. Followers and edit-log replay must not push. We have patches for both and can open PRs. For (2), the DDL waits a short, configurable time until each alive backend has queued the push, and the report diff still covers any backend that misses it. ### How to Reproduce? 1. Create a custom analyzer (for example an ngram tokenizer plus a lowercase token filter) and a table with an inverted index that uses it (`PROPERTIES("analyzer" = "<name>")`). 2. Run a loop of `MATCH_ANY` queries and small stream loads against the table. 3. `kill -9` one BE and restart it. From the moment the FE marks the BE alive until about 10 s later, queries fail with `[E33] Policy not found` and loads with `[E-6011]`. On the master FE, `After diff index policy result` is logged twice per restart, 10 s apart. For (2): run `CREATE INVERTED INDEX ANALYZER ...` and immediately an `INSERT` into a table whose index uses it. The insert fails with E33. ### Anything Else? With fix (1), on a single-FE, single-BE cluster with 7 BE restarts per variant, counted from BE start until 60 s after it was marked alive: - queries failing with E33: 8,449 of 44,824 before, 0 of 42,286 after; - loads failing with E-6011: 322 of 2,092 before, 0 of 2,074 after. With fix (2), we ran `CREATE INVERTED INDEX ANALYZER`, `CREATE TABLE`, then an immediate `INSERT`, 10 times. The first insert failed with E33 every time before the fix (it took 7.3–8.0 s until one succeeded), and succeeded on the first try every time after (0.05–0.18 s). ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
