This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 76a5240d0ad [docs](k8s) Clarify decoupled cluster operations and index
builds (#4655)
76a5240d0ad is described below
commit 76a5240d0ad1e1a56fff9c3b3f6520a9dbf21460
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Tue Nov 17 11:38:56 2026 +0800
[docs](k8s) Clarify decoupled cluster operations and index builds (#4655)
Kubernetes users running compute-storage decoupled clusters need to size
the FE heap separately from the Pod memory limit and distinguish a
temporary BE restart from permanent decommission. The existing index
examples also direct Doris 4.1.5 cloud users to run an unnecessary build
for tokenized indexes.
This change clarifies six existing documentation topics:
- FE heap sizing, with a 4 GiB heap in the 8 GiB Pod example.
- MetaService HTTP token configuration and its relationship to SQL
credentials.
- Node-wide BE initialization and the `skipDefaultSystemInit` switch.
- Planned BE restarts, shutdown timing, and Doris 4.1.5 image
compatibility.
- Cloud index backfill syntax, tokenized-index schema changes, and
Japanese tokenization limits.
- TopN threshold tuning and its memory, CPU, and remote I/O costs.
## Versions
- [x] dev
- [x] 4.x
- [x] 3.x
- [ ] 2.1 or older (not covered by version/language sync gate)
General operational and TopN guidance is synchronized across active
versions. Doris 4.1.5 restart, index-build, and tokenizer details are
scoped to dev and 4.x. The index instructions explicitly distinguish the
4.1.5 behavior from newer SNII formats.
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [x] Checked by AI
- [ ] Test Cases Built
- [x] Updated required version and language counterparts, or explained
why not
- [x] If only one language changed, confirmed whether source/translation
counterparts need sync
Both languages are updated. Existing paths, front matter, and navigation
are preserved.
## Validation
- All 32 changed Markdown files compiled with the Docusaurus Markdown
processor.
- Front matter, explicit anchors, and topic coverage checked.
- Documentation governance compared with the pre-change baseline: zero
new findings; the existing 169 findings, including 19 errors, are
unchanged.
- `git diff --check` passed.
No full website build or live Kubernetes/SQL execution was performed.
Co-authored-by: morningman <[email protected]>
---
.../separating-storage-compute/config-cg.md | 41 ++++++++++++++++++++++
.../separating-storage-compute/config-fe.md | 10 +++++-
.../separating-storage-compute/config-ms.md | 10 +++++-
.../topn-optimization.md | 12 +++++++
.../index/inverted-index/custom-analyzer.md | 12 +++----
docs/table-design/index/inverted-index/overview.md | 37 ++++++++++++++++++-
.../separating-storage-compute/config-cg.md | 41 ++++++++++++++++++++++
.../separating-storage-compute/config-fe.md | 10 +++++-
.../separating-storage-compute/config-ms.md | 10 +++++-
.../topn-optimization.md | 12 +++++++
.../index/inverted-index/custom-analyzer.md | 12 +++----
.../table-design/index/inverted-index/overview.md | 37 ++++++++++++++++++-
.../separating-storage-compute/config-cg.md | 21 +++++++++++
.../separating-storage-compute/config-fe.md | 10 +++++-
.../separating-storage-compute/config-ms.md | 10 +++++-
.../topn-optimization.md | 12 +++++++
.../separating-storage-compute/config-cg.md | 41 ++++++++++++++++++++++
.../separating-storage-compute/config-fe.md | 10 +++++-
.../separating-storage-compute/config-ms.md | 10 +++++-
.../topn-optimization.md | 12 +++++++
.../index/inverted-index/custom-analyzer.md | 12 +++----
.../table-design/index/inverted-index/overview.md | 37 ++++++++++++++++++-
.../separating-storage-compute/config-cg.md | 21 +++++++++++
.../separating-storage-compute/config-fe.md | 10 +++++-
.../separating-storage-compute/config-ms.md | 10 +++++-
.../topn-optimization.md | 12 +++++++
.../separating-storage-compute/config-cg.md | 41 ++++++++++++++++++++++
.../separating-storage-compute/config-fe.md | 10 +++++-
.../separating-storage-compute/config-ms.md | 10 +++++-
.../topn-optimization.md | 12 +++++++
.../index/inverted-index/custom-analyzer.md | 12 +++----
.../table-design/index/inverted-index/overview.md | 37 ++++++++++++++++++-
32 files changed, 550 insertions(+), 44 deletions(-)
diff --git
a/docs/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
b/docs/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
index 7802d7f147a..3166d5fe4a0 100644
--- a/docs/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
+++ b/docs/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
@@ -38,6 +38,27 @@ spec:
| `image` | The image address for deploying the BE service. Use the images
provided by the [Apache Doris official image
registry](https://hub.docker.com/r/apache/doris) |
| `replicas` | The number of BE service nodes in the compute group |
+### Node system initialization {#be-system-initialization}
+
+By default, `skipDefaultSystemInit` is `false`. Doris Operator adds a
privileged BE init container that runs:
+
+```shell
+sysctl -w vm.max_map_count=2000000 && swapoff -a
+```
+
+These commands change the underlying Linux node's settings, including settings
shared by other workloads. With kind on a desktop, the affected Linux kernel
can belong to the container runtime's virtual machine. This initialization is
not limited to the BE process.
+
+When node administrators have already prepared the required settings, set
`skipDefaultSystemInit: true` on the existing compute-group entry. Keep the
other fields and compute groups in your manifest:
+
+```yaml
+spec:
+ computeGroups:
+ - uniqueId: cg1
+ skipDefaultSystemInit: true
+```
+
+For Doris 3.0.4 and later, this switch also injects `SKIP_CHECK_ULIMIT=true`,
which skips startup checks for some system parameters. Verify the node settings
yourself before enabling the switch.
+
## Scenario 2: Deploy Multiple Compute Groups for Business Isolation
The `DorisDisaggregatedCluster` resource supports deploying multiple compute
groups, and each compute group is independent of the others. The following
example shows the configuration for deploying two compute groups named `cg1`
and `cg2`:
@@ -434,3 +455,23 @@ In Operator 26.0.1, removing an entire `computeGroups`
entry uses the BE drop pa
After each change, check the compute group's phase in the custom resource, Pod
readiness, and `SHOW BACKENDS` together. Confirm the target node count, `Alive
= true` for retained nodes, removal of retired nodes, and successful queries
and loads through the intended compute group.
If the operation stalls, inspect `kubectl describe ddc`, Pod Events, PVC
binding, and Operator logs. `Pending` Pods often require checking scheduling or
storage; `Decommissioning` requires checking Doris decommission progress;
`ScaleDownFailed` requires checking the Operator's SQL error and FE
connectivity or permissions. Resolve the reported cause before retrying or
forcing resource deletion. See [Status and
Troubleshooting](../doris-operator/status-and-troubleshooting.md).
+
+## Planned BE restarts {#planned-be-restarts}
+
+For a temporary restart, retain the BE registration and use graceful shutdown.
In Doris 4.1.5 compute-storage decoupled mode, `DECOMMISSION BACKEND` is a
persistent scale-in operation: restarting the Pod does not cancel it, and
`CANCEL DECOMMISSION BACKEND` is unsupported. Use the scale-in workflow above
when permanently removing a BE.
+
+The rejection of plain `DROP BACKEND` is an intentional safeguard; see [DROP
BACKEND](../../../sql-manual/sql-statements/cluster-management/instance-management/DROP-BACKEND.md)
for the explicit `DROPP BACKEND` syntax. For permanent removal, prefer the
Operator scale-in workflow to keep Doris metadata and Kubernetes resources
consistent.
+
+Doris Operator installs a BE `preStop` hook. The stock Doris 4.1.5 image's
hook calls `stop_be.sh --grace`; the BE reports its shutdown state through
heartbeats and waits for running queries. Before restarting:
+
+1. Confirm that the other BEs in the compute group are healthy and have enough
capacity.
+2. Check the generated Pod's `terminationGracePeriodSeconds`. It must cover
the query wait, the post-shutdown delay, and execution overhead. Doris 4.1.5
defaults to `grace_shutdown_wait_seconds=120` and
`grace_shutdown_post_delay_seconds=30`. Operator 26.0.1 defaults the BE Pod
grace period to 200 seconds, but an explicit `grace_shutdown_wait_seconds` in
`be.conf` also changes the generated Pod value; check both settings together.
+3. Use normal Pod deletion, which runs `preStop`, rather than force deletion:
+
+ ```shell
+ kubectl delete pod ${be_pod} -n ${namespace}
+ ```
+
+4. Wait for the replacement Pod to become Ready and for `SHOW BACKENDS` to
report the BE alive with its shutdown state cleared. Verify queries and loads
before restarting another BE.
+
+Operator 26.0.1 also has a two-phase graceful rollout that requires a
compatible BE image. The stock Doris 4.1.5 image does not meet that flow's
compatibility check, so upgrading the Operator alone does not enable it for
this image. The existing `preStop` hook remains the graceful-stop mechanism.
Graceful shutdown reduces disruption; it cannot protect running queries after
an abrupt node loss or forced termination.
diff --git
a/docs/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
b/docs/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
index 67e8d078002..9bd9645837b 100644
--- a/docs/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
+++ b/docs/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
@@ -46,6 +46,14 @@ spec:
Apply this configuration to the [`DorisDisaggregatedCluster` resource you want
to
deploy](./install-doris-cluster.md#configure-the-dorisdisaggregatedcluster-resource).
+### JVM heap and Pod memory {#fe-jvm-memory}
+
+Kubernetes `requests.memory` controls scheduling, while `limits.memory` limits
the container's total memory. Doris Operator does not calculate the FE JVM heap
from either value. The heap, Metaspace, thread stacks, direct memory, and other
native allocations must fit within the container limit.
+
+Before lowering the memory limit, copy `fe.conf` from the image you deploy and
adjust both `-Xms` and `-Xmx` in its JVM options. For JDK 17 images, edit
`JAVA_OPTS_FOR_JDK_17` and preserve its other options. For example, the
ConfigMap below uses a 4 GiB heap within the 8 GiB container limit above; size
production heaps for the metadata and workload. Mount the file through
`feSpec.configMaps` as described in Custom startup configuration, then restart
the FE to apply it.
+
+If Kubernetes reports `OOMKilled`, compare the effective JVM options with the
Pod memory limit. Leave room for non-heap memory when increasing the limit;
when reducing the heap, do not lower only `-Xmx` while leaving a larger `-Xms`.
+
## Configure the number of Follower nodes
The FE service has two roles, with the following responsibilities:
@@ -93,7 +101,7 @@ data:
# Log dir
LOG_DIR = ${DORIS_HOME}/log
# For jdk 17, this JAVA_OPTS will be used as default JVM options
- JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false
-Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR
-Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED"
+ JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false
-Xmx4096m -Xms4096m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR
-Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED"
# INFO, WARN, ERROR, FATAL
sys_log_level = INFO
# NORMAL, BRIEF, ASYNC
diff --git
a/docs/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
b/docs/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
index 6832d7e0372..4441412cf0e 100644
--- a/docs/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
+++ b/docs/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
@@ -135,7 +135,7 @@ data:
brpc_listen_port = 5000
brpc_num_threads = -1
brpc_idle_timeout_sec = 30
- http_token = greedisgood9999
+ http_token = <custom-http-token>
# // doris txn config
label_keep_max_second = 259200
@@ -177,6 +177,14 @@ Parameter description:
Apply the above configuration to the [DorisDisaggregatedCluster
resource](./install-doris-cluster.md#2-modify-key-configurations) you want to
deploy.
+### Customize the management HTTP token {#ms-http-token}
+
+`http_token` authenticates MetaService management HTTP API requests. Its
default value is `greedisgood9999`. Replace `<custom-http-token>` in the
example with your own value and keep the rest of your `doris_cloud.conf`
settings. Store the file under the ConfigMap key `doris_cloud.conf` and
reference that ConfigMap through `spec.metaService.configMaps`, as shown above.
+
+Apply the same token to all MetaService replicas and restart them to load the
startup configuration. API clients must pass that value in the `token` query
parameter. Doris Operator also reads `http_token` from the referenced MS
ConfigMap for its management HTTP requests; changing only a file inside a Pod
leaves the Operator's configuration out of sync.
+
+FE and BE metadata RPCs do not require this management HTTP token in their
configuration. The SQL management credentials in `spec.authSecret` are separate
from `http_token`.
+
## 5. Configure service probe timeouts
Doris Operator provides two timeout parameters for storage-compute decoupled
cluster services:
diff --git
a/docs/query-acceleration/optimization-technology-principle/topn-optimization.md
b/docs/query-acceleration/optimization-technology-principle/topn-optimization.md
index 90db0e8f44b..c444433ec6b 100644
---
a/docs/query-acceleration/optimization-technology-principle/topn-optimization.md
+++
b/docs/query-acceleration/optimization-technology-principle/topn-optimization.md
@@ -327,6 +327,18 @@ Confirm that TOPN optimization is not enabled on the MOR
table. For MOR tables,
The policy controls only phase 2 of TOPN lazy materialization for Doris
internal tables. Phase 1, other operators, and external-table reads may still
write to File Cache. Use `TopNLazyMaterializationSecondPhaseWriteCacheBytes` to
determine whether phase 2 performed cache writes instead of relying on the
query's total write volume.
+### Can I raise the 1024 threshold for larger LIMIT queries?
{#larger-topn-limits}
+
+Yes. `topn_opt_limit_threshold` defaults to 1024 and controls several TopN
optimization paths. It is not the maximum supported SQL `LIMIT`, and different
paths do not all use the same inclusive boundary. To evaluate a query with
`LIMIT 5000`, first raise the threshold in the current session:
+
+```sql
+SET topn_opt_limit_threshold = 10000;
+```
+
+Compare the same query before and after the change with `EXPLAIN` and Profile.
A larger candidate set can increase sort CPU, peak memory, and the rows fetched
during two-phase materialization. In compute-storage decoupled mode, scattered
second-phase reads can also increase remote I/O. Table type, sort keys,
selectivity, and the other optimization conditions still apply; raising the
threshold does not force every TopN optimization to activate.
+
+Check the query latency, scan rows, peak memory, and, where available,
second-read time and remote-read bytes under representative concurrency.
Restore the default with `SET topn_opt_limit_threshold = 1024;` after the
experiment. Avoid increasing the global value until the workload benefits have
been measured.
+
## Quick Reference for Related Parameters
| Desired Effect | Setting |
diff --git a/docs/table-design/index/inverted-index/custom-analyzer.md
b/docs/table-design/index/inverted-index/custom-analyzer.md
index dfd0f888d7a..0ed0ad78e3c 100644
--- a/docs/table-design/index/inverted-index/custom-analyzer.md
+++ b/docs/table-design/index/inverted-index/custom-analyzer.md
@@ -310,18 +310,16 @@ SHOW ALTER TABLE COLUMN WHERE TableName='articles';
### Building an Index
-After adding an index, you must build the index for existing data:
+In compute-storage integrated mode, build an added inverted index for existing
data explicitly:
```sql
--- Build a specific index (non-cloud mode)
BUILD INDEX idx_content_chinese ON articles;
+SHOW BUILD INDEX WHERE TableName = 'articles';
+```
--- Build all indexes (cloud mode)
-BUILD INDEX ON articles;
+In Doris 4.1.5 compute-storage decoupled mode, adding the tokenized index in
the previous example builds historical data through schema change. Wait for
`SHOW ALTER TABLE COLUMN` to report `FINISHED`; an additional `BUILD INDEX` for
that index is unnecessary and is rejected as not needed.
--- Check the build progress
-SHOW BUILD INDEX WHERE TableName='articles';
-```
+Separate historical backfill is supported for non-tokenized inverted indexes
and NGram BloomFilter indexes added with `enable_add_index_for_new_data=true`.
In this case, the cloud syntax is `BUILD INDEX ON table_name`, without an index
name. See the [deployment-specific build
example](./overview.md#cloud-index-build). This Doris 4.1.5 behavior does not
describe the SNII build path in newer kernels.
### Key Notes
diff --git a/docs/table-design/index/inverted-index/overview.md
b/docs/table-design/index/inverted-index/overview.md
index 0a087888a49..b856863f2c6 100644
--- a/docs/table-design/index/inverted-index/overview.md
+++ b/docs/table-design/index/inverted-index/overview.md
@@ -220,10 +220,37 @@ CREATE TABLE t (
-- Create on an existing table
CREATE INDEX idx_content ON docs(content) USING INVERTED PROPERTIES ("parser"
= "chinese");
--- Build the index for existing data
+-- Compute-storage integrated mode: build the index for existing data
BUILD INDEX idx_content ON docs;
```
+#### Existing data in compute-storage decoupled mode {#cloud-index-build}
+
+In Doris 4.1.5, adding a tokenized inverted index, such as the
`parser="chinese"` index above, builds historical data through a schema-change
job. Wait for the job to reach `FINISHED`; a separate `BUILD INDEX` is not
needed for that index:
+
+```sql
+SHOW ALTER TABLE COLUMN WHERE TableName = 'docs';
+```
+
+For non-tokenized inverted indexes (`parser="none"`) and NGram BloomFilter
indexes, the default `enable_add_index_for_new_data=false` also builds
historical data through schema change. Set it to `true` when you want to add
the index for new data first and backfill historical data separately. Run the
following example in a session with an available compute group:
+
+```sql
+CREATE TABLE cloud_index_example (
+ id INT,
+ tag VARCHAR(64)
+) DUPLICATE KEY(id)
+DISTRIBUTED BY HASH(id) BUCKETS 1;
+
+INSERT INTO cloud_index_example VALUES (1, 'doris'), (2, 'database');
+SET enable_add_index_for_new_data = true;
+CREATE INDEX idx_tag ON cloud_index_example(tag)
+USING INVERTED PROPERTIES("parser" = "none");
+BUILD INDEX ON cloud_index_example;
+SHOW BUILD INDEX WHERE TableName = 'cloud_index_example';
+```
+
+The build is asynchronous. Wait for `SHOW BUILD INDEX` to report `FINISHED`
before expecting historical rows to use the index. The cloud syntax omits the
index name: `BUILD INDEX idx_tag ON cloud_index_example` is rejected. These
Doris 4.1.5 instructions describe its existing index formats, not the separate
SNII build path in newer kernels.
+
### Drop an index
```sql
@@ -237,6 +264,14 @@ SHOW CREATE TABLE docs;
SHOW INDEX FROM docs;
```
+## Frequently asked questions {#inverted-index-faq}
+
+### Does Doris 4.1.5 support Japanese tokenization and typo tolerance?
+
+Doris 4.1.5 does not support the built-in Kuromoji tokenizer. Evaluate the
`icu` or `unicode` parser with representative Japanese text and inspect its
output with `TOKENIZE()`; their segmentation is different from Kuromoji
morphological analysis. Check the documentation for the exact deployed release
before selecting a newer parser.
+
+Prefix, regular-expression, and phrase-slop matching are supported by the
[search operators](./search-operators.md). They do not provide edit-distance
spelling correction: a misspelled term is not automatically expanded to nearby
dictionary terms.
+
## Further reading
### Core documentation
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
index d67d2ba41b2..6f1b8eb4189 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
@@ -38,6 +38,27 @@ spec:
| `image` | 部署 BE 服务的镜像地址。请使用 [Apache Doris
官方镜像仓库](https://hub.docker.com/r/apache/doris) 提供的镜像 |
| `replicas` | 计算组内 BE 服务节点的数量 |
+### 节点系统初始化 {#be-system-initialization}
+
+`skipDefaultSystemInit` 默认为 `false`。Doris Operator 会添加以特权模式运行的 BE init
container,执行:
+
+```shell
+sysctl -w vm.max_map_count=2000000 && swapoff -a
+```
+
+这些命令修改底层 Linux 节点的系统设置,会影响共享该设置的其他工作负载。在桌面环境使用 kind 时,受影响的 Linux
内核可能属于容器运行时的虚拟机。初始化的作用范围不限于 BE 进程。
+
+如果节点管理员已完成所需的系统配置,可以在已有计算组条目中设置 `skipDefaultSystemInit:
true`。编辑清单时保留其他字段及其他计算组:
+
+```yaml
+spec:
+ computeGroups:
+ - uniqueId: cg1
+ skipDefaultSystemInit: true
+```
+
+对于 Doris 3.0.4 及以上版本,该开关还会注入
`SKIP_CHECK_ULIMIT=true`,跳过部分系统参数的启动检查。启用前应自行确认节点设置满足运行要求。
+
## 场景 2:部署多个计算组实现业务隔离
`DorisDisaggregatedCluster` 资源支持部署多套计算组,每套计算组之间相互独立。以下示例展示了部署名称为 `cg1` 和 `cg2`
两套计算组的配置:
@@ -434,3 +455,23 @@ Operator 26.0.1 在移除整个 `computeGroups` 条目时使用 BE 直接移除
每次变更后,结合自定义资源中的计算组阶段、Pod 就绪状态和 `SHOW BACKENDS` 验证结果。确认节点数量符合目标、保留节点的 `Alive =
true`、退役节点已被移除,并通过预期计算组执行查询和导入验证。
如果操作停滞,检查 `kubectl describe ddc`、Pod Events、PVC 绑定状态和 Operator 日志。Pod 为
`Pending` 时通常需要检查调度或存储;`Decommissioning` 需要检查 Doris 退役进度;`ScaleDownFailed` 需要检查
Operator 记录的 SQL 错误以及 FE
连通性和权限。解决报告的原因后再重试,不要直接强制删除资源。参见[状态与故障排查](../doris-operator/status-and-troubleshooting.md)。
+
+## 计划重启 BE {#planned-be-restarts}
+
+临时重启应保留 BE 的注册信息,并使用优雅停止。在 Doris 4.1.5 存算分离模式下,`DECOMMISSION BACKEND`
是持久化的缩容操作:重启 Pod 不会取消该状态,且不支持 `CANCEL DECOMMISSION BACKEND`。永久移除 BE
时,应使用上方的缩容流程。
+
+普通 `DROP BACKEND` 被拒绝是主动保护;显式删除语法 `DROPP BACKEND` 参见 [DROP
BACKEND](../../../sql-manual/sql-statements/cluster-management/instance-management/DROP-BACKEND.md)。永久删除节点时,优先使用
Operator 的缩容流程,使 Doris 元数据和 Kubernetes 资源保持一致。
+
+Doris Operator 会配置 BE 的 `preStop` hook。Doris 4.1.5 原始镜像中的 hook 调用 `stop_be.sh
--grace`;BE 通过心跳上报 shutdown 状态,并等待正在运行的查询退出。重启前:
+
+1. 确认计算组内其他 BE 健康且有足够容量。
+2. 检查生成的 Pod 的 `terminationGracePeriodSeconds`,使其覆盖查询等待、退出后的延迟及执行开销。Doris
4.1.5 默认
`grace_shutdown_wait_seconds=120`、`grace_shutdown_post_delay_seconds=30`。Operator
26.0.1 默认给 BE Pod 设置 200 秒宽限期,但在 `be.conf` 中显式设置 `grace_shutdown_wait_seconds`
也会改变生成的 Pod 值,应同时检查两处设置。
+3. 正常删除 Pod 以执行 `preStop`,不要强制删除:
+
+ ```shell
+ kubectl delete pod ${be_pod} -n ${namespace}
+ ```
+
+4. 等待替换 Pod 变为 Ready,并通过 `SHOW BACKENDS` 确认 BE 存活且 shutdown
状态已清除。验证查询和导入正常后,再重启另一个 BE。
+
+Operator 26.0.1 另有两阶段优雅滚动流程,需要兼容的 BE 镜像。Doris 4.1.5 原始镜像不满足该流程的兼容性检查,因此仅升级
Operator 无法让该镜像启用新流程;已有的 `preStop` hook
仍是其优雅停止机制。优雅停止可以减少影响,但无法保护节点突然丢失或被强制终止时正在运行的查询。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
index b9a54adacbd..0fd7c35ce2d 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
@@ -46,6 +46,14 @@ spec:
将上述配置信息更新到[需要部署的 `DorisDisaggregatedCluster`
资源](./install-doris-cluster.md#配置-dorisdisaggregatedcluster-资源)中。
+### JVM 堆与 Pod 内存 {#fe-jvm-memory}
+
+Kubernetes 的 `requests.memory` 用于调度,`limits.memory` 限制容器总内存。Doris Operator
不会根据这两个值自动计算 FE 的 JVM 堆大小。堆、Metaspace、线程栈、直接内存及其他本地内存分配都必须计入容器内存预算。
+
+降低内存限制前,先从待部署镜像中复制 `fe.conf`,同时调整 JVM 参数中的 `-Xms` 和 `-Xmx`。使用 JDK 17 的镜像应修改
`JAVA_OPTS_FOR_JDK_17`,并保留其中的其他参数。例如,下方 ConfigMap 在上方 8 GiB 容器限制内使用 4 GiB
堆;生产环境应按元数据规模和负载确定堆大小。按“自定义启动配置”中的步骤,通过 `feSpec.configMaps` 挂载文件并重启 FE,使其生效。
+
+如果 Kubernetes 报告 `OOMKilled`,请对比实际 JVM 参数与 Pod
内存限制。提高容器限制后仍需为非堆内存留出空间;降低堆大小时,不能只减小 `-Xmx` 而保留更大的 `-Xms`。
+
## 配置 Follower 节点数量
FE 服务包含两种角色,分工如下:
@@ -93,7 +101,7 @@ data:
# Log dir
LOG_DIR = ${DORIS_HOME}/log
# For jdk 17, this JAVA_OPTS will be used as default JVM options
- JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false
-Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR
-Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED"
+ JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false
-Xmx4096m -Xms4096m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR
-Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED"
# INFO, WARN, ERROR, FATAL
sys_log_level = INFO
# NORMAL, BRIEF, ASYNC
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
index 4d42d4bced6..8954fee9c5c 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
@@ -135,7 +135,7 @@ data:
brpc_listen_port = 5000
brpc_num_threads = -1
brpc_idle_timeout_sec = 30
- http_token = greedisgood9999
+ http_token = <custom-http-token>
# // doris txn config
label_keep_max_second = 259200
@@ -177,6 +177,14 @@ spec:
将上述配置更新到需要部署的 [DorisDisaggregatedCluster
资源](./install-doris-cluster.md#3-配置-dorisdisaggregatedcluster-资源)中。
+### 自定义管理 HTTP token {#ms-http-token}
+
+`http_token` 用于校验 MetaService 管理 HTTP API 的请求,默认值为 `greedisgood9999`。请将示例中的
`<custom-http-token>` 替换为自己的值,并保留 `doris_cloud.conf` 中的其他配置。将文件放在 ConfigMap 的
`doris_cloud.conf` 键下,再按上方步骤通过 `spec.metaService.configMaps` 引用该 ConfigMap。
+
+所有 MetaService 副本应使用相同 token,并重启以加载启动配置。API 客户端通过 `token` 查询参数传入该值。Doris
Operator 也会从引用的 MS ConfigMap 中读取 `http_token`,用于管理 HTTP 请求;只修改 Pod 内的文件会导致
Operator 与 MS 配置不一致。
+
+FE、BE 的元数据 RPC 不需要在其配置中设置这个管理 HTTP token。`spec.authSecret` 中的 SQL 管理凭据与
`http_token` 相互独立。
+
## 5. 配置服务探测超时
Doris Operator 为存算分离集群服务提供两种超时参数配置:
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/query-acceleration/optimization-technology-principle/topn-optimization.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/query-acceleration/optimization-technology-principle/topn-optimization.md
index 7d40eb919a3..4bd6ab0cc40 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/query-acceleration/optimization-technology-principle/topn-optimization.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/query-acceleration/optimization-technology-principle/topn-optimization.md
@@ -327,6 +327,18 @@ Profile 还提供分 BE
指标。`TopNLazyMaterializationSecondPhasePerBackend`
该策略只控制 Doris 内表的 TOPN 延迟物化第二阶段。第一阶段、其他算子和外表读取仍可能写入 File Cache。请使用
`TopNLazyMaterializationSecondPhaseWriteCacheBytes`
判断第二阶段是否发生写入,不要使用整个查询的写入量代替该指标。
+### 能否提高 1024 阈值以优化更大的 LIMIT? {#larger-topn-limits}
+
+可以。`topn_opt_limit_threshold` 默认为 1024,控制多条 TopN 优化路径。它不是 SQL `LIMIT`
的最大值,各条路径对边界值是否包含的判断也不完全相同。例如,评估 `LIMIT 5000` 的查询时,可以先提高当前会话的阈值:
+
+```sql
+SET topn_opt_limit_threshold = 10000;
+```
+
+对同一查询调整前后的 `EXPLAIN` 和 Profile 进行比较。候选集变大可能增加排序
CPU、峰值内存及两阶段物化需要回读的行数。在存算分离模式下,离散的第二阶段读取还可能增加远端
I/O。表类型、排序键、选择性及其他优化条件仍需满足;提高阈值不会强制启用所有 TopN 优化。
+
+在具有代表性的并发负载下,对比查询延迟、扫描行数、峰值内存,以及可用的第二阶段回读时间和远端读取字节数。实验结束后可用 `SET
topn_opt_limit_threshold = 1024;` 恢复默认值。确认实际负载受益后,再考虑提高全局值。
+
## 相关参数对照速查
| 想要的效果 | 设置方式 |
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/custom-analyzer.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/custom-analyzer.md
index 10f4b448890..5ad373201d8 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/custom-analyzer.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/custom-analyzer.md
@@ -311,18 +311,16 @@ SHOW ALTER TABLE COLUMN WHERE TableName='articles';
### 构建索引
-添加索引后,需要为已有数据构建索引:
+在存算一体模式下,新增倒排索引后,应显式为历史数据构建索引:
```sql
--- 构建指定索引(非云模式)
BUILD INDEX idx_content_chinese ON articles;
+SHOW BUILD INDEX WHERE TableName = 'articles';
+```
--- 构建所有索引(云模式)
-BUILD INDEX ON articles;
+在 Doris 4.1.5 存算分离模式下,前一个示例中新增的分词索引会通过 Schema Change 构建历史数据。等待 `SHOW ALTER
TABLE COLUMN` 显示 `FINISHED` 即可,再为该索引执行 `BUILD INDEX` 会被判断为不需要构建而拒绝。
--- 查看构建进度
-SHOW BUILD INDEX WHERE TableName='articles';
-```
+通过 `enable_add_index_for_new_data=true` 添加的不分词倒排索引和 NGram BloomFilter
索引支持单独补建历史数据。此时云模式语法为 `BUILD INDEX ON
table_name`,不指定索引名。参见[按部署模式构建索引的示例](./overview.md#cloud-index-build)。上述 Doris
4.1.5 行为不描述较新内核中的 SNII 构建路径。
### 关键说明
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/overview.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/overview.md
index 537a014c23b..9874bab26c0 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/overview.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/table-design/index/inverted-index/overview.md
@@ -220,10 +220,37 @@ CREATE TABLE t (
-- 在现有表上创建
CREATE INDEX idx_content ON docs(content) USING INVERTED PROPERTIES ("parser"
= "chinese");
--- 为现有数据构建索引
+-- 存算一体模式:为现有数据构建索引
BUILD INDEX idx_content ON docs;
```
+#### 存算分离模式下的历史数据 {#cloud-index-build}
+
+在 Doris 4.1.5 中,新增分词倒排索引(如上方 `parser="chinese"` 的索引)会通过 Schema Change
任务构建历史数据。等待任务达到 `FINISHED` 即可,该索引不需要再单独执行 `BUILD INDEX`:
+
+```sql
+SHOW ALTER TABLE COLUMN WHERE TableName = 'docs';
+```
+
+对于不分词倒排索引(`parser="none"`)和 NGram BloomFilter 索引,默认的
`enable_add_index_for_new_data=false` 也会通过 Schema Change
构建历史数据。如果希望先为新增数据添加索引,再单独补建历史数据,可以将该变量设为 `true`。请在已选择可用计算组的会话中执行以下示例:
+
+```sql
+CREATE TABLE cloud_index_example (
+ id INT,
+ tag VARCHAR(64)
+) DUPLICATE KEY(id)
+DISTRIBUTED BY HASH(id) BUCKETS 1;
+
+INSERT INTO cloud_index_example VALUES (1, 'doris'), (2, 'database');
+SET enable_add_index_for_new_data = true;
+CREATE INDEX idx_tag ON cloud_index_example(tag)
+USING INVERTED PROPERTIES("parser" = "none");
+BUILD INDEX ON cloud_index_example;
+SHOW BUILD INDEX WHERE TableName = 'cloud_index_example';
+```
+
+构建任务异步执行。等待 `SHOW BUILD INDEX` 显示 `FINISHED` 后,历史数据才能使用该索引。云模式语法不指定索引名,`BUILD
INDEX idx_tag ON cloud_index_example` 会被拒绝。以上 Doris 4.1.5
说明针对该版本已有的索引格式,不描述较新内核中独立的 SNII 构建路径。
+
### 删除索引
```sql
@@ -237,6 +264,14 @@ SHOW CREATE TABLE docs;
SHOW INDEX FROM docs;
```
+## 常见问题 {#inverted-index-faq}
+
+### Doris 4.1.5 是否支持日文分词和拼写容错?
+
+Doris 4.1.5 不支持内置 Kuromoji 分词器。可以用具有代表性的日文文本评估 `icu` 或 `unicode` 分词器,并通过
`TOKENIZE()` 检查输出;它们的分词方式与 Kuromoji 形态素分析不同。选择较新的分词器前,应核对实际部署版本的文档。
+
+[搜索算子](./search-operators.md)支持前缀、正则及短语 slop
匹配,但这些能力不提供基于编辑距离的拼写纠错,不会自动将拼错的词扩展为词典中相近的词。
+
## 延伸阅读
### 核心文档
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
index 494b6dc30f2..da39d9e2611 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
@@ -19,6 +19,27 @@ spec:
```
`${beImage}` 为部署 BE 服务的镜像地址,请使用 [apache doris
官方镜像仓库](https://hub.docker.com/r/apache/doris)提供的镜像。`${uniqueId}`
为计算组的唯一标识也是计算组的名称,匹配规则为`[a-zA-Z][0-9a-zA-Z_]+`。replicas 为计算组内 BE 服务节点的数量。
+### 节点系统初始化 {#be-system-initialization}
+
+`skipDefaultSystemInit` 默认为 `false`。Doris Operator 会添加以特权模式运行的 BE init
container,执行:
+
+```shell
+sysctl -w vm.max_map_count=2000000 && swapoff -a
+```
+
+这些命令修改底层 Linux 节点的系统设置,会影响共享该设置的其他工作负载。在桌面环境使用 kind 时,受影响的 Linux
内核可能属于容器运行时的虚拟机。初始化的作用范围不限于 BE 进程。
+
+如果节点管理员已完成所需的系统配置,可以在已有计算组条目中设置 `skipDefaultSystemInit:
true`。编辑清单时保留其他字段及其他计算组:
+
+```yaml
+spec:
+ computeGroups:
+ - uniqueId: cg1
+ skipDefaultSystemInit: true
+```
+
+对于 Doris 3.0.4 及以上版本,该开关还会注入
`SKIP_CHECK_ULIMIT=true`,跳过部分系统参数的启动检查。启用前应自行确认节点设置满足运行要求。
+
## 配置多计算组
`DorisDisaggregatedCluster` 资源支持部署多套计算组,每套计算组之间相互独立。以下展示了部署名称为 `cg1` 和 `cg2`
两套计算组的配置示例:
```yaml
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
index 66e713bfab1..b38f19c745e 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
@@ -22,6 +22,14 @@ spec:
```
将上述配置信息更新到[需要部署的 `DorisDisaggregatedCluster`
资源](./install-doris-cluster.md#3-配置-dorisdisaggregatedcluster-资源)中。
+### JVM 堆与 Pod 内存 {#fe-jvm-memory}
+
+Kubernetes 的 `requests.memory` 用于调度,`limits.memory` 限制容器总内存。Doris Operator
不会根据这两个值自动计算 FE 的 JVM 堆大小。堆、Metaspace、线程栈、直接内存及其他本地内存分配都必须计入容器内存预算。
+
+降低内存限制前,先从待部署镜像中复制 `fe.conf`,同时调整 JVM 参数中的 `-Xms` 和 `-Xmx`。使用 JDK 17 的镜像应修改
`JAVA_OPTS_FOR_JDK_17`,并保留其中的其他参数。例如,下方 ConfigMap 在上方 8 GiB 容器限制内使用 4 GiB
堆;生产环境应按元数据规模和负载确定堆大小。按“自定义启动配置”中的步骤,通过 `feSpec.configMaps` 挂载文件并重启 FE,使其生效。
+
+如果 Kubernetes 报告 `OOMKilled`,请对比实际 JVM 参数与 Pod
内存限制。提高容器限制后仍需为非堆内存留出空间;降低堆大小时,不能只减小 `-Xmx` 而保留更大的 `-Xms`。
+
## 配置 Follower 数量
FE 服务有 Follower 和 Observer 两种角色,Follower 负责 sql 解析任务和元数据的管理和存储。Observer 主要负责
sql 解析任务,分担 Follower 的查询和写入负载任务。Doris 使用 bdbje 存储系统管理元数据,bdbje 底层实现类似 paxos
协议算法。
分布式部署中,需要配置多个 Follower 节点参与分布式环境下元数据管理工作。
@@ -52,7 +60,7 @@ Doris Operator 通过 Kubernetes 的 ConfigMap 挂载 FE 启动配置。配置
# Log dir
LOG_DIR = ${DORIS_HOME}/log
# For jdk 17, this JAVA_OPTS will be used as default JVM options
- JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false
-Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR
-Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED"
+ JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false
-Xmx4096m -Xms4096m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR
-Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED"
# INFO, WARN, ERROR, FATAL
sys_log_level = INFO
# NORMAL, BRIEF, ASYNC
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
index 69c98fd2c98..6b156d143be 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
@@ -67,7 +67,7 @@ Doris-Operator 通过 ConfigMap 挂载组件的启动配置文件。Doris-Operat
brpc_listen_port = 5000
brpc_num_threads = -1
brpc_idle_timeout_sec = 30
- http_token = greedisgood9999
+ http_token = <custom-http-token>
# // doris txn config
label_keep_max_second = 259200
@@ -104,6 +104,14 @@ Doris-Operator 通过 ConfigMap 挂载组件的启动配置文件。Doris-Operat
在 Kubernetes 部署中,定制化 MetaService 启动配置时请不要填写 `fdb_cluster` 配置,Doris Operator
会自动处理相关信息。
:::
+### 自定义管理 HTTP token {#ms-http-token}
+
+`http_token` 用于校验 MetaService 管理 HTTP API 的请求,默认值为 `greedisgood9999`。请将示例中的
`<custom-http-token>` 替换为自己的值,并保留 `doris_cloud.conf` 中的其他配置。将文件放在 ConfigMap 的
`doris_cloud.conf` 键下,再按上方步骤通过 `spec.metaService.configMaps` 引用该 ConfigMap。
+
+所有 MetaService 副本应使用相同 token,并重启以加载启动配置。API 客户端通过 `token` 查询参数传入该值。Doris
Operator 也会从引用的 MS ConfigMap 中读取 `http_token`,用于管理 HTTP 请求;只修改 Pod 内的文件会导致
Operator 与 MS 配置不一致。
+
+FE、BE 的元数据 RPC 不需要在其配置中设置这个管理 HTTP token。`spec.authSecret` 中的 SQL 管理凭据与
`http_token` 相互独立。
+
## 配置服务探测超时
Doris Operator 为存算分离集群服务提供两种超时参数配置:存活探测超时和启动超时。
### 存活探测超时配置
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/query-acceleration/optimization-technology-principle/topn-optimization.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/query-acceleration/optimization-technology-principle/topn-optimization.md
index 6ef70b252c2..8589034b104 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/query-acceleration/optimization-technology-principle/topn-optimization.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-3.x/query-acceleration/optimization-technology-principle/topn-optimization.md
@@ -140,3 +140,15 @@ explain SQL 拿到 query plan 可以确认这个 sql 是否启用 TOPN 查询优
- UncompressedBytesRead: 137.99 MB
- VectorPredEvalTime: 0ns
```
+
+### 能否提高 1024 阈值以优化更大的 LIMIT? {#larger-topn-limits}
+
+可以。`topn_opt_limit_threshold` 默认为 1024,控制多条 TopN 优化路径。它不是 SQL `LIMIT`
的最大值,各条路径对边界值是否包含的判断也不完全相同。例如,评估 `LIMIT 5000` 的查询时,可以先提高当前会话的阈值:
+
+```sql
+SET topn_opt_limit_threshold = 10000;
+```
+
+对同一查询调整前后的 `EXPLAIN` 和 Profile 进行比较。候选集变大可能增加排序
CPU、峰值内存及两阶段物化需要回读的行数。在存算分离模式下,离散的第二阶段读取还可能增加远端
I/O。表类型、排序键、选择性及其他优化条件仍需满足;提高阈值不会强制启用所有 TopN 优化。
+
+在具有代表性的并发负载下,对比查询延迟、扫描行数、峰值内存,以及可用的第二阶段回读时间和远端读取字节数。实验结束后可用 `SET
topn_opt_limit_threshold = 1024;` 恢复默认值。确认实际负载受益后,再考虑提高全局值。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
index d67d2ba41b2..6f1b8eb4189 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
@@ -38,6 +38,27 @@ spec:
| `image` | 部署 BE 服务的镜像地址。请使用 [Apache Doris
官方镜像仓库](https://hub.docker.com/r/apache/doris) 提供的镜像 |
| `replicas` | 计算组内 BE 服务节点的数量 |
+### 节点系统初始化 {#be-system-initialization}
+
+`skipDefaultSystemInit` 默认为 `false`。Doris Operator 会添加以特权模式运行的 BE init
container,执行:
+
+```shell
+sysctl -w vm.max_map_count=2000000 && swapoff -a
+```
+
+这些命令修改底层 Linux 节点的系统设置,会影响共享该设置的其他工作负载。在桌面环境使用 kind 时,受影响的 Linux
内核可能属于容器运行时的虚拟机。初始化的作用范围不限于 BE 进程。
+
+如果节点管理员已完成所需的系统配置,可以在已有计算组条目中设置 `skipDefaultSystemInit:
true`。编辑清单时保留其他字段及其他计算组:
+
+```yaml
+spec:
+ computeGroups:
+ - uniqueId: cg1
+ skipDefaultSystemInit: true
+```
+
+对于 Doris 3.0.4 及以上版本,该开关还会注入
`SKIP_CHECK_ULIMIT=true`,跳过部分系统参数的启动检查。启用前应自行确认节点设置满足运行要求。
+
## 场景 2:部署多个计算组实现业务隔离
`DorisDisaggregatedCluster` 资源支持部署多套计算组,每套计算组之间相互独立。以下示例展示了部署名称为 `cg1` 和 `cg2`
两套计算组的配置:
@@ -434,3 +455,23 @@ Operator 26.0.1 在移除整个 `computeGroups` 条目时使用 BE 直接移除
每次变更后,结合自定义资源中的计算组阶段、Pod 就绪状态和 `SHOW BACKENDS` 验证结果。确认节点数量符合目标、保留节点的 `Alive =
true`、退役节点已被移除,并通过预期计算组执行查询和导入验证。
如果操作停滞,检查 `kubectl describe ddc`、Pod Events、PVC 绑定状态和 Operator 日志。Pod 为
`Pending` 时通常需要检查调度或存储;`Decommissioning` 需要检查 Doris 退役进度;`ScaleDownFailed` 需要检查
Operator 记录的 SQL 错误以及 FE
连通性和权限。解决报告的原因后再重试,不要直接强制删除资源。参见[状态与故障排查](../doris-operator/status-and-troubleshooting.md)。
+
+## 计划重启 BE {#planned-be-restarts}
+
+临时重启应保留 BE 的注册信息,并使用优雅停止。在 Doris 4.1.5 存算分离模式下,`DECOMMISSION BACKEND`
是持久化的缩容操作:重启 Pod 不会取消该状态,且不支持 `CANCEL DECOMMISSION BACKEND`。永久移除 BE
时,应使用上方的缩容流程。
+
+普通 `DROP BACKEND` 被拒绝是主动保护;显式删除语法 `DROPP BACKEND` 参见 [DROP
BACKEND](../../../sql-manual/sql-statements/cluster-management/instance-management/DROP-BACKEND.md)。永久删除节点时,优先使用
Operator 的缩容流程,使 Doris 元数据和 Kubernetes 资源保持一致。
+
+Doris Operator 会配置 BE 的 `preStop` hook。Doris 4.1.5 原始镜像中的 hook 调用 `stop_be.sh
--grace`;BE 通过心跳上报 shutdown 状态,并等待正在运行的查询退出。重启前:
+
+1. 确认计算组内其他 BE 健康且有足够容量。
+2. 检查生成的 Pod 的 `terminationGracePeriodSeconds`,使其覆盖查询等待、退出后的延迟及执行开销。Doris
4.1.5 默认
`grace_shutdown_wait_seconds=120`、`grace_shutdown_post_delay_seconds=30`。Operator
26.0.1 默认给 BE Pod 设置 200 秒宽限期,但在 `be.conf` 中显式设置 `grace_shutdown_wait_seconds`
也会改变生成的 Pod 值,应同时检查两处设置。
+3. 正常删除 Pod 以执行 `preStop`,不要强制删除:
+
+ ```shell
+ kubectl delete pod ${be_pod} -n ${namespace}
+ ```
+
+4. 等待替换 Pod 变为 Ready,并通过 `SHOW BACKENDS` 确认 BE 存活且 shutdown
状态已清除。验证查询和导入正常后,再重启另一个 BE。
+
+Operator 26.0.1 另有两阶段优雅滚动流程,需要兼容的 BE 镜像。Doris 4.1.5 原始镜像不满足该流程的兼容性检查,因此仅升级
Operator 无法让该镜像启用新流程;已有的 `preStop` hook
仍是其优雅停止机制。优雅停止可以减少影响,但无法保护节点突然丢失或被强制终止时正在运行的查询。
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
index b9a54adacbd..0fd7c35ce2d 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
@@ -46,6 +46,14 @@ spec:
将上述配置信息更新到[需要部署的 `DorisDisaggregatedCluster`
资源](./install-doris-cluster.md#配置-dorisdisaggregatedcluster-资源)中。
+### JVM 堆与 Pod 内存 {#fe-jvm-memory}
+
+Kubernetes 的 `requests.memory` 用于调度,`limits.memory` 限制容器总内存。Doris Operator
不会根据这两个值自动计算 FE 的 JVM 堆大小。堆、Metaspace、线程栈、直接内存及其他本地内存分配都必须计入容器内存预算。
+
+降低内存限制前,先从待部署镜像中复制 `fe.conf`,同时调整 JVM 参数中的 `-Xms` 和 `-Xmx`。使用 JDK 17 的镜像应修改
`JAVA_OPTS_FOR_JDK_17`,并保留其中的其他参数。例如,下方 ConfigMap 在上方 8 GiB 容器限制内使用 4 GiB
堆;生产环境应按元数据规模和负载确定堆大小。按“自定义启动配置”中的步骤,通过 `feSpec.configMaps` 挂载文件并重启 FE,使其生效。
+
+如果 Kubernetes 报告 `OOMKilled`,请对比实际 JVM 参数与 Pod
内存限制。提高容器限制后仍需为非堆内存留出空间;降低堆大小时,不能只减小 `-Xmx` 而保留更大的 `-Xms`。
+
## 配置 Follower 节点数量
FE 服务包含两种角色,分工如下:
@@ -93,7 +101,7 @@ data:
# Log dir
LOG_DIR = ${DORIS_HOME}/log
# For jdk 17, this JAVA_OPTS will be used as default JVM options
- JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false
-Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR
-Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED"
+ JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false
-Xmx4096m -Xms4096m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR
-Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED"
# INFO, WARN, ERROR, FATAL
sys_log_level = INFO
# NORMAL, BRIEF, ASYNC
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
index 4d42d4bced6..8954fee9c5c 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
@@ -135,7 +135,7 @@ data:
brpc_listen_port = 5000
brpc_num_threads = -1
brpc_idle_timeout_sec = 30
- http_token = greedisgood9999
+ http_token = <custom-http-token>
# // doris txn config
label_keep_max_second = 259200
@@ -177,6 +177,14 @@ spec:
将上述配置更新到需要部署的 [DorisDisaggregatedCluster
资源](./install-doris-cluster.md#3-配置-dorisdisaggregatedcluster-资源)中。
+### 自定义管理 HTTP token {#ms-http-token}
+
+`http_token` 用于校验 MetaService 管理 HTTP API 的请求,默认值为 `greedisgood9999`。请将示例中的
`<custom-http-token>` 替换为自己的值,并保留 `doris_cloud.conf` 中的其他配置。将文件放在 ConfigMap 的
`doris_cloud.conf` 键下,再按上方步骤通过 `spec.metaService.configMaps` 引用该 ConfigMap。
+
+所有 MetaService 副本应使用相同 token,并重启以加载启动配置。API 客户端通过 `token` 查询参数传入该值。Doris
Operator 也会从引用的 MS ConfigMap 中读取 `http_token`,用于管理 HTTP 请求;只修改 Pod 内的文件会导致
Operator 与 MS 配置不一致。
+
+FE、BE 的元数据 RPC 不需要在其配置中设置这个管理 HTTP token。`spec.authSecret` 中的 SQL 管理凭据与
`http_token` 相互独立。
+
## 5. 配置服务探测超时
Doris Operator 为存算分离集群服务提供两种超时参数配置:
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/query-acceleration/optimization-technology-principle/topn-optimization.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/query-acceleration/optimization-technology-principle/topn-optimization.md
index 8b366f7f7ba..edb6180334b 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/query-acceleration/optimization-technology-principle/topn-optimization.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/query-acceleration/optimization-technology-principle/topn-optimization.md
@@ -327,6 +327,18 @@ Profile 还提供分 BE
指标。`TopNLazyMaterializationSecondPhasePerBackend`
该策略只控制 Doris 内表的 TOPN 延迟物化第二阶段。第一阶段、其他算子和外表读取仍可能写入 File Cache。请使用
`TopNLazyMaterializationSecondPhaseWriteCacheBytes`
判断第二阶段是否发生写入,不要使用整个查询的写入量代替该指标。
+### 能否提高 1024 阈值以优化更大的 LIMIT? {#larger-topn-limits}
+
+可以。`topn_opt_limit_threshold` 默认为 1024,控制多条 TopN 优化路径。它不是 SQL `LIMIT`
的最大值,各条路径对边界值是否包含的判断也不完全相同。例如,评估 `LIMIT 5000` 的查询时,可以先提高当前会话的阈值:
+
+```sql
+SET topn_opt_limit_threshold = 10000;
+```
+
+对同一查询调整前后的 `EXPLAIN` 和 Profile 进行比较。候选集变大可能增加排序
CPU、峰值内存及两阶段物化需要回读的行数。在存算分离模式下,离散的第二阶段读取还可能增加远端
I/O。表类型、排序键、选择性及其他优化条件仍需满足;提高阈值不会强制启用所有 TopN 优化。
+
+在具有代表性的并发负载下,对比查询延迟、扫描行数、峰值内存,以及可用的第二阶段回读时间和远端读取字节数。实验结束后可用 `SET
topn_opt_limit_threshold = 1024;` 恢复默认值。确认实际负载受益后,再考虑提高全局值。
+
## 相关参数对照速查
| 想要的效果 | 设置方式 |
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/index/inverted-index/custom-analyzer.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/index/inverted-index/custom-analyzer.md
index 6fe941cec1b..6fd1d6b6b87 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/index/inverted-index/custom-analyzer.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/index/inverted-index/custom-analyzer.md
@@ -311,18 +311,16 @@ SHOW ALTER TABLE COLUMN WHERE TableName='articles';
### 构建索引
-添加索引后,需要为已有数据构建索引:
+在存算一体模式下,新增倒排索引后,应显式为历史数据构建索引:
```sql
--- 构建指定索引(非云模式)
BUILD INDEX idx_content_chinese ON articles;
+SHOW BUILD INDEX WHERE TableName = 'articles';
+```
--- 构建所有索引(云模式)
-BUILD INDEX ON articles;
+在 Doris 4.1.5 存算分离模式下,前一个示例中新增的分词索引会通过 Schema Change 构建历史数据。等待 `SHOW ALTER
TABLE COLUMN` 显示 `FINISHED` 即可,再为该索引执行 `BUILD INDEX` 会被判断为不需要构建而拒绝。
--- 查看构建进度
-SHOW BUILD INDEX WHERE TableName='articles';
-```
+通过 `enable_add_index_for_new_data=true` 添加的不分词倒排索引和 NGram BloomFilter
索引支持单独补建历史数据。此时云模式语法为 `BUILD INDEX ON
table_name`,不指定索引名。参见[按部署模式构建索引的示例](./overview.md#cloud-index-build)。上述 Doris
4.1.5 行为不描述较新内核中的 SNII 构建路径。
### 关键说明
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/index/inverted-index/overview.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/index/inverted-index/overview.md
index beaaaa6fef8..5b7bf9419f9 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/index/inverted-index/overview.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/table-design/index/inverted-index/overview.md
@@ -221,10 +221,37 @@ CREATE TABLE t (
-- 在现有表上创建
CREATE INDEX idx_content ON docs(content) USING INVERTED PROPERTIES ("parser"
= "chinese");
--- 为现有数据构建索引
+-- 存算一体模式:为现有数据构建索引
BUILD INDEX idx_content ON docs;
```
+#### 存算分离模式下的历史数据 {#cloud-index-build}
+
+在 Doris 4.1.5 中,新增分词倒排索引(如上方 `parser="chinese"` 的索引)会通过 Schema Change
任务构建历史数据。等待任务达到 `FINISHED` 即可,该索引不需要再单独执行 `BUILD INDEX`:
+
+```sql
+SHOW ALTER TABLE COLUMN WHERE TableName = 'docs';
+```
+
+对于不分词倒排索引(`parser="none"`)和 NGram BloomFilter 索引,默认的
`enable_add_index_for_new_data=false` 也会通过 Schema Change
构建历史数据。如果希望先为新增数据添加索引,再单独补建历史数据,可以将该变量设为 `true`。请在已选择可用计算组的会话中执行以下示例:
+
+```sql
+CREATE TABLE cloud_index_example (
+ id INT,
+ tag VARCHAR(64)
+) DUPLICATE KEY(id)
+DISTRIBUTED BY HASH(id) BUCKETS 1;
+
+INSERT INTO cloud_index_example VALUES (1, 'doris'), (2, 'database');
+SET enable_add_index_for_new_data = true;
+CREATE INDEX idx_tag ON cloud_index_example(tag)
+USING INVERTED PROPERTIES("parser" = "none");
+BUILD INDEX ON cloud_index_example;
+SHOW BUILD INDEX WHERE TableName = 'cloud_index_example';
+```
+
+构建任务异步执行。等待 `SHOW BUILD INDEX` 显示 `FINISHED` 后,历史数据才能使用该索引。云模式语法不指定索引名,`BUILD
INDEX idx_tag ON cloud_index_example` 会被拒绝。以上 Doris 4.1.5
说明针对该版本已有的索引格式,不描述较新内核中独立的 SNII 构建路径。
+
### 删除索引
```sql
@@ -238,6 +265,14 @@ SHOW CREATE TABLE docs;
SHOW INDEX FROM docs;
```
+## 常见问题 {#inverted-index-faq}
+
+### Doris 4.1.5 是否支持日文分词和拼写容错?
+
+Doris 4.1.5 不支持内置 Kuromoji 分词器。可以用具有代表性的日文文本评估 `icu` 或 `unicode` 分词器,并通过
`TOKENIZE()` 检查输出;它们的分词方式与 Kuromoji 形态素分析不同。选择较新的分词器前,应核对实际部署版本的文档。
+
+[搜索算子](./search-operators.md)支持前缀、正则及短语 slop
匹配,但这些能力不提供基于编辑距离的拼写纠错,不会自动将拼错的词扩展为词典中相近的词。
+
## 延伸阅读
### 核心文档
diff --git
a/versioned_docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
b/versioned_docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
index b019f18dfb2..a4531477403 100644
---
a/versioned_docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
+++
b/versioned_docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
@@ -19,6 +19,27 @@ spec:
```
Here, `${beImage}` is the image used to deploy the BE service; please use the
image provided by the [Apache Doris official
repository](https://hub.docker.com/r/apache/doris). `${uniqueId}` is the unique
identifier and name of the compute group, which must match the pattern
`[a-zA-Z][0-9a-zA-Z_]+`. The `replicas` field specifies the number of BE nodes
within the compute group.
+### Node system initialization {#be-system-initialization}
+
+By default, `skipDefaultSystemInit` is `false`. Doris Operator adds a
privileged BE init container that runs:
+
+```shell
+sysctl -w vm.max_map_count=2000000 && swapoff -a
+```
+
+These commands change the underlying Linux node's settings, including settings
shared by other workloads. With kind on a desktop, the affected Linux kernel
can belong to the container runtime's virtual machine. This initialization is
not limited to the BE process.
+
+When node administrators have already prepared the required settings, set
`skipDefaultSystemInit: true` on the existing compute-group entry. Keep the
other fields and compute groups in your manifest:
+
+```yaml
+spec:
+ computeGroups:
+ - uniqueId: cg1
+ skipDefaultSystemInit: true
+```
+
+For Doris 3.0.4 and later, this switch also injects `SKIP_CHECK_ULIMIT=true`,
which skips startup checks for some system parameters. Verify the node settings
yourself before enabling the switch.
+
## Configuring Multiple Compute Groups
The `DorisDisaggregatedCluster` resource supports the deployment of multiple
independent compute groups. The following example shows a configuration with
two compute groups, `cg1` and `cg2`:
```yaml
diff --git
a/versioned_docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
b/versioned_docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
index 17a07d236f9..fa1c2caca80 100644
---
a/versioned_docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
+++
b/versioned_docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
@@ -22,6 +22,14 @@ spec:
```
Update the above configuration in the [DorisDisaggregatedCluster
resource](./install-doris-cluster.md#step-3-deploy-the-compute-storage-decoupled-cluster)
that you intend to deploy.
+### JVM heap and Pod memory {#fe-jvm-memory}
+
+Kubernetes `requests.memory` controls scheduling, while `limits.memory` limits
the container's total memory. Doris Operator does not calculate the FE JVM heap
from either value. The heap, Metaspace, thread stacks, direct memory, and other
native allocations must fit within the container limit.
+
+Before lowering the memory limit, copy `fe.conf` from the image you deploy and
adjust both `-Xms` and `-Xmx` in its JVM options. For JDK 17 images, edit
`JAVA_OPTS_FOR_JDK_17` and preserve its other options. For example, the
ConfigMap below uses a 4 GiB heap within the 8 GiB container limit above; size
production heaps for the metadata and workload. Mount the file through
`feSpec.configMaps` as described in Custom startup configuration, then restart
the FE to apply it.
+
+If Kubernetes reports `OOMKilled`, compare the effective JVM options with the
Pod memory limit. Leave room for non-heap memory when increasing the limit;
when reducing the heap, do not lower only `-Xmx` while leaving a larger `-Xms`.
+
## Configuring the Number of Follower Nodes
In a Doris Frontend (FE) service, there are two types of roles: Follower and
Observer. Follower nodes are responsible for SQL parsing, metadata management,
and storage. Observer nodes primarily handle SQL parsing to offload query and
write traffic from Followers. Doris uses the bdbje storage system for metadata
management, which implements an algorithm similar to the Paxos protocol.
@@ -56,7 +64,7 @@ The Doris Operator mounts the FE startup configuration using
a Kubernetes Config
# Log dir
LOG_DIR = ${DORIS_HOME}/log
# For jdk 17, this JAVA_OPTS will be used as default JVM options
- JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false
-Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR
-Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED"
+ JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false
-Xmx4096m -Xms4096m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR
-Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED"
# INFO, WARN, ERROR, FATAL
sys_log_level = INFO
# NORMAL, BRIEF, ASYNC
diff --git
a/versioned_docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
b/versioned_docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
index 4b5050d7f5a..0abaf3bf968 100644
---
a/versioned_docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
+++
b/versioned_docs/version-3.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
@@ -70,7 +70,7 @@ Doris-Operator uses a ConfigMap to mount the startup
configuration file for comp
brpc_listen_port = 5000
brpc_num_threads = -1
brpc_idle_timeout_sec = 30
- http_token = greedisgood9999
+ http_token = <custom-http-token>
# // doris txn config
label_keep_max_second = 259200
@@ -103,6 +103,14 @@ Doris-Operator uses a ConfigMap to mount the startup
configuration file for comp
When customizing the MetaService startup configuration in a Kubernetes
deployment, do not include the fdb_cluster configuration; Doris Operator will
handle that automatically.
:::
+### Customize the management HTTP token {#ms-http-token}
+
+`http_token` authenticates MetaService management HTTP API requests. Its
default value is `greedisgood9999`. Replace `<custom-http-token>` in the
example with your own value and keep the rest of your `doris_cloud.conf`
settings. Store the file under the ConfigMap key `doris_cloud.conf` and
reference that ConfigMap through `spec.metaService.configMaps`, as shown above.
+
+Apply the same token to all MetaService replicas and restart them to load the
startup configuration. API clients must pass that value in the `token` query
parameter. Doris Operator also reads `http_token` from the referenced MS
ConfigMap for its management HTTP requests; changing only a file inside a Pod
leaves the Operator's configuration out of sync.
+
+FE and BE metadata RPCs do not require this management HTTP token in their
configuration. The SQL management credentials in `spec.authSecret` are separate
from `http_token`.
+
## Configuring Service Probe Timeouts
Doris Operator provides two timeout parameters for decoupled storage and
compute services: liveness probe timeout and startup timeout.
diff --git
a/versioned_docs/version-3.x/query-acceleration/optimization-technology-principle/topn-optimization.md
b/versioned_docs/version-3.x/query-acceleration/optimization-technology-principle/topn-optimization.md
index e1c92f84112..d011383094f 100644
---
a/versioned_docs/version-3.x/query-acceleration/optimization-technology-principle/topn-optimization.md
+++
b/versioned_docs/version-3.x/query-acceleration/optimization-technology-principle/topn-optimization.md
@@ -138,3 +138,15 @@ Before version 2.0.3, the `RuntimePredicate` metrics were
not separated out, and
- UncompressedBytesRead: 137.99 MB
- VectorPredEvalTime: 0ns
```
+
+### Can I raise the 1024 threshold for larger LIMIT queries?
{#larger-topn-limits}
+
+Yes. `topn_opt_limit_threshold` defaults to 1024 and controls several TopN
optimization paths. It is not the maximum supported SQL `LIMIT`, and different
paths do not all use the same inclusive boundary. To evaluate a query with
`LIMIT 5000`, first raise the threshold in the current session:
+
+```sql
+SET topn_opt_limit_threshold = 10000;
+```
+
+Compare the same query before and after the change with `EXPLAIN` and Profile.
A larger candidate set can increase sort CPU, peak memory, and the rows fetched
during two-phase materialization. In compute-storage decoupled mode, scattered
second-phase reads can also increase remote I/O. Table type, sort keys,
selectivity, and the other optimization conditions still apply; raising the
threshold does not force every TopN optimization to activate.
+
+Check the query latency, scan rows, peak memory, and, where available,
second-read time and remote-read bytes under representative concurrency.
Restore the default with `SET topn_opt_limit_threshold = 1024;` after the
experiment. Avoid increasing the global value until the workload benefits have
been measured.
diff --git
a/versioned_docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
b/versioned_docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
index 7802d7f147a..3166d5fe4a0 100644
---
a/versioned_docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
+++
b/versioned_docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-cg.md
@@ -38,6 +38,27 @@ spec:
| `image` | The image address for deploying the BE service. Use the images
provided by the [Apache Doris official image
registry](https://hub.docker.com/r/apache/doris) |
| `replicas` | The number of BE service nodes in the compute group |
+### Node system initialization {#be-system-initialization}
+
+By default, `skipDefaultSystemInit` is `false`. Doris Operator adds a
privileged BE init container that runs:
+
+```shell
+sysctl -w vm.max_map_count=2000000 && swapoff -a
+```
+
+These commands change the underlying Linux node's settings, including settings
shared by other workloads. With kind on a desktop, the affected Linux kernel
can belong to the container runtime's virtual machine. This initialization is
not limited to the BE process.
+
+When node administrators have already prepared the required settings, set
`skipDefaultSystemInit: true` on the existing compute-group entry. Keep the
other fields and compute groups in your manifest:
+
+```yaml
+spec:
+ computeGroups:
+ - uniqueId: cg1
+ skipDefaultSystemInit: true
+```
+
+For Doris 3.0.4 and later, this switch also injects `SKIP_CHECK_ULIMIT=true`,
which skips startup checks for some system parameters. Verify the node settings
yourself before enabling the switch.
+
## Scenario 2: Deploy Multiple Compute Groups for Business Isolation
The `DorisDisaggregatedCluster` resource supports deploying multiple compute
groups, and each compute group is independent of the others. The following
example shows the configuration for deploying two compute groups named `cg1`
and `cg2`:
@@ -434,3 +455,23 @@ In Operator 26.0.1, removing an entire `computeGroups`
entry uses the BE drop pa
After each change, check the compute group's phase in the custom resource, Pod
readiness, and `SHOW BACKENDS` together. Confirm the target node count, `Alive
= true` for retained nodes, removal of retired nodes, and successful queries
and loads through the intended compute group.
If the operation stalls, inspect `kubectl describe ddc`, Pod Events, PVC
binding, and Operator logs. `Pending` Pods often require checking scheduling or
storage; `Decommissioning` requires checking Doris decommission progress;
`ScaleDownFailed` requires checking the Operator's SQL error and FE
connectivity or permissions. Resolve the reported cause before retrying or
forcing resource deletion. See [Status and
Troubleshooting](../doris-operator/status-and-troubleshooting.md).
+
+## Planned BE restarts {#planned-be-restarts}
+
+For a temporary restart, retain the BE registration and use graceful shutdown.
In Doris 4.1.5 compute-storage decoupled mode, `DECOMMISSION BACKEND` is a
persistent scale-in operation: restarting the Pod does not cancel it, and
`CANCEL DECOMMISSION BACKEND` is unsupported. Use the scale-in workflow above
when permanently removing a BE.
+
+The rejection of plain `DROP BACKEND` is an intentional safeguard; see [DROP
BACKEND](../../../sql-manual/sql-statements/cluster-management/instance-management/DROP-BACKEND.md)
for the explicit `DROPP BACKEND` syntax. For permanent removal, prefer the
Operator scale-in workflow to keep Doris metadata and Kubernetes resources
consistent.
+
+Doris Operator installs a BE `preStop` hook. The stock Doris 4.1.5 image's
hook calls `stop_be.sh --grace`; the BE reports its shutdown state through
heartbeats and waits for running queries. Before restarting:
+
+1. Confirm that the other BEs in the compute group are healthy and have enough
capacity.
+2. Check the generated Pod's `terminationGracePeriodSeconds`. It must cover
the query wait, the post-shutdown delay, and execution overhead. Doris 4.1.5
defaults to `grace_shutdown_wait_seconds=120` and
`grace_shutdown_post_delay_seconds=30`. Operator 26.0.1 defaults the BE Pod
grace period to 200 seconds, but an explicit `grace_shutdown_wait_seconds` in
`be.conf` also changes the generated Pod value; check both settings together.
+3. Use normal Pod deletion, which runs `preStop`, rather than force deletion:
+
+ ```shell
+ kubectl delete pod ${be_pod} -n ${namespace}
+ ```
+
+4. Wait for the replacement Pod to become Ready and for `SHOW BACKENDS` to
report the BE alive with its shutdown state cleared. Verify queries and loads
before restarting another BE.
+
+Operator 26.0.1 also has a two-phase graceful rollout that requires a
compatible BE image. The stock Doris 4.1.5 image does not meet that flow's
compatibility check, so upgrading the Operator alone does not enable it for
this image. The existing `preStop` hook remains the graceful-stop mechanism.
Graceful shutdown reduces disruption; it cannot protect running queries after
an abrupt node loss or forced termination.
diff --git
a/versioned_docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
b/versioned_docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
index 67e8d078002..9bd9645837b 100644
---
a/versioned_docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
+++
b/versioned_docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-fe.md
@@ -46,6 +46,14 @@ spec:
Apply this configuration to the [`DorisDisaggregatedCluster` resource you want
to
deploy](./install-doris-cluster.md#configure-the-dorisdisaggregatedcluster-resource).
+### JVM heap and Pod memory {#fe-jvm-memory}
+
+Kubernetes `requests.memory` controls scheduling, while `limits.memory` limits
the container's total memory. Doris Operator does not calculate the FE JVM heap
from either value. The heap, Metaspace, thread stacks, direct memory, and other
native allocations must fit within the container limit.
+
+Before lowering the memory limit, copy `fe.conf` from the image you deploy and
adjust both `-Xms` and `-Xmx` in its JVM options. For JDK 17 images, edit
`JAVA_OPTS_FOR_JDK_17` and preserve its other options. For example, the
ConfigMap below uses a 4 GiB heap within the 8 GiB container limit above; size
production heaps for the metadata and workload. Mount the file through
`feSpec.configMaps` as described in Custom startup configuration, then restart
the FE to apply it.
+
+If Kubernetes reports `OOMKilled`, compare the effective JVM options with the
Pod memory limit. Leave room for non-heap memory when increasing the limit;
when reducing the heap, do not lower only `-Xmx` while leaving a larger `-Xms`.
+
## Configure the number of Follower nodes
The FE service has two roles, with the following responsibilities:
@@ -93,7 +101,7 @@ data:
# Log dir
LOG_DIR = ${DORIS_HOME}/log
# For jdk 17, this JAVA_OPTS will be used as default JVM options
- JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false
-Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR
-Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED"
+ JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false
-Xmx4096m -Xms4096m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR
-Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED"
# INFO, WARN, ERROR, FATAL
sys_log_level = INFO
# NORMAL, BRIEF, ASYNC
diff --git
a/versioned_docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
b/versioned_docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
index 6832d7e0372..4441412cf0e 100644
---
a/versioned_docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
+++
b/versioned_docs/version-4.x/install/deploy-on-kubernetes/separating-storage-compute/config-ms.md
@@ -135,7 +135,7 @@ data:
brpc_listen_port = 5000
brpc_num_threads = -1
brpc_idle_timeout_sec = 30
- http_token = greedisgood9999
+ http_token = <custom-http-token>
# // doris txn config
label_keep_max_second = 259200
@@ -177,6 +177,14 @@ Parameter description:
Apply the above configuration to the [DorisDisaggregatedCluster
resource](./install-doris-cluster.md#2-modify-key-configurations) you want to
deploy.
+### Customize the management HTTP token {#ms-http-token}
+
+`http_token` authenticates MetaService management HTTP API requests. Its
default value is `greedisgood9999`. Replace `<custom-http-token>` in the
example with your own value and keep the rest of your `doris_cloud.conf`
settings. Store the file under the ConfigMap key `doris_cloud.conf` and
reference that ConfigMap through `spec.metaService.configMaps`, as shown above.
+
+Apply the same token to all MetaService replicas and restart them to load the
startup configuration. API clients must pass that value in the `token` query
parameter. Doris Operator also reads `http_token` from the referenced MS
ConfigMap for its management HTTP requests; changing only a file inside a Pod
leaves the Operator's configuration out of sync.
+
+FE and BE metadata RPCs do not require this management HTTP token in their
configuration. The SQL management credentials in `spec.authSecret` are separate
from `http_token`.
+
## 5. Configure service probe timeouts
Doris Operator provides two timeout parameters for storage-compute decoupled
cluster services:
diff --git
a/versioned_docs/version-4.x/query-acceleration/optimization-technology-principle/topn-optimization.md
b/versioned_docs/version-4.x/query-acceleration/optimization-technology-principle/topn-optimization.md
index 8184711cd17..d6a980b7930 100644
---
a/versioned_docs/version-4.x/query-acceleration/optimization-technology-principle/topn-optimization.md
+++
b/versioned_docs/version-4.x/query-acceleration/optimization-technology-principle/topn-optimization.md
@@ -327,6 +327,18 @@ Confirm that TOPN optimization is not enabled on the MOR
table. For MOR tables,
The policy controls only phase 2 of TOPN lazy materialization for Doris
internal tables. Phase 1, other operators, and external-table reads may still
write to File Cache. Use `TopNLazyMaterializationSecondPhaseWriteCacheBytes` to
determine whether phase 2 performed cache writes instead of relying on the
query's total write volume.
+### Can I raise the 1024 threshold for larger LIMIT queries?
{#larger-topn-limits}
+
+Yes. `topn_opt_limit_threshold` defaults to 1024 and controls several TopN
optimization paths. It is not the maximum supported SQL `LIMIT`, and different
paths do not all use the same inclusive boundary. To evaluate a query with
`LIMIT 5000`, first raise the threshold in the current session:
+
+```sql
+SET topn_opt_limit_threshold = 10000;
+```
+
+Compare the same query before and after the change with `EXPLAIN` and Profile.
A larger candidate set can increase sort CPU, peak memory, and the rows fetched
during two-phase materialization. In compute-storage decoupled mode, scattered
second-phase reads can also increase remote I/O. Table type, sort keys,
selectivity, and the other optimization conditions still apply; raising the
threshold does not force every TopN optimization to activate.
+
+Check the query latency, scan rows, peak memory, and, where available,
second-read time and remote-read bytes under representative concurrency.
Restore the default with `SET topn_opt_limit_threshold = 1024;` after the
experiment. Avoid increasing the global value until the workload benefits have
been measured.
+
## Quick Reference for Related Parameters
| Desired Effect | Setting |
diff --git
a/versioned_docs/version-4.x/table-design/index/inverted-index/custom-analyzer.md
b/versioned_docs/version-4.x/table-design/index/inverted-index/custom-analyzer.md
index f5249de9d8f..5130695d3f0 100644
---
a/versioned_docs/version-4.x/table-design/index/inverted-index/custom-analyzer.md
+++
b/versioned_docs/version-4.x/table-design/index/inverted-index/custom-analyzer.md
@@ -310,18 +310,16 @@ SHOW ALTER TABLE COLUMN WHERE TableName='articles';
### Building an Index
-After adding an index, you must build the index for existing data:
+In compute-storage integrated mode, build an added inverted index for existing
data explicitly:
```sql
--- Build a specific index (non-cloud mode)
BUILD INDEX idx_content_chinese ON articles;
+SHOW BUILD INDEX WHERE TableName = 'articles';
+```
--- Build all indexes (cloud mode)
-BUILD INDEX ON articles;
+In Doris 4.1.5 compute-storage decoupled mode, adding the tokenized index in
the previous example builds historical data through schema change. Wait for
`SHOW ALTER TABLE COLUMN` to report `FINISHED`; an additional `BUILD INDEX` for
that index is unnecessary and is rejected as not needed.
--- Check the build progress
-SHOW BUILD INDEX WHERE TableName='articles';
-```
+Separate historical backfill is supported for non-tokenized inverted indexes
and NGram BloomFilter indexes added with `enable_add_index_for_new_data=true`.
In this case, the cloud syntax is `BUILD INDEX ON table_name`, without an index
name. See the [deployment-specific build
example](./overview.md#cloud-index-build). This Doris 4.1.5 behavior does not
describe the SNII build path in newer kernels.
### Key Notes
diff --git
a/versioned_docs/version-4.x/table-design/index/inverted-index/overview.md
b/versioned_docs/version-4.x/table-design/index/inverted-index/overview.md
index bbe8a0fd85f..c9c238ded32 100644
--- a/versioned_docs/version-4.x/table-design/index/inverted-index/overview.md
+++ b/versioned_docs/version-4.x/table-design/index/inverted-index/overview.md
@@ -221,10 +221,37 @@ CREATE TABLE t (
-- Create on an existing table
CREATE INDEX idx_content ON docs(content) USING INVERTED PROPERTIES ("parser"
= "chinese");
--- Build the index for existing data
+-- Compute-storage integrated mode: build the index for existing data
BUILD INDEX idx_content ON docs;
```
+#### Existing data in compute-storage decoupled mode {#cloud-index-build}
+
+In Doris 4.1.5, adding a tokenized inverted index, such as the
`parser="chinese"` index above, builds historical data through a schema-change
job. Wait for the job to reach `FINISHED`; a separate `BUILD INDEX` is not
needed for that index:
+
+```sql
+SHOW ALTER TABLE COLUMN WHERE TableName = 'docs';
+```
+
+For non-tokenized inverted indexes (`parser="none"`) and NGram BloomFilter
indexes, the default `enable_add_index_for_new_data=false` also builds
historical data through schema change. Set it to `true` when you want to add
the index for new data first and backfill historical data separately. Run the
following example in a session with an available compute group:
+
+```sql
+CREATE TABLE cloud_index_example (
+ id INT,
+ tag VARCHAR(64)
+) DUPLICATE KEY(id)
+DISTRIBUTED BY HASH(id) BUCKETS 1;
+
+INSERT INTO cloud_index_example VALUES (1, 'doris'), (2, 'database');
+SET enable_add_index_for_new_data = true;
+CREATE INDEX idx_tag ON cloud_index_example(tag)
+USING INVERTED PROPERTIES("parser" = "none");
+BUILD INDEX ON cloud_index_example;
+SHOW BUILD INDEX WHERE TableName = 'cloud_index_example';
+```
+
+The build is asynchronous. Wait for `SHOW BUILD INDEX` to report `FINISHED`
before expecting historical rows to use the index. The cloud syntax omits the
index name: `BUILD INDEX idx_tag ON cloud_index_example` is rejected. These
Doris 4.1.5 instructions describe its existing index formats, not the separate
SNII build path in newer kernels.
+
### Drop an index
```sql
@@ -238,6 +265,14 @@ SHOW CREATE TABLE docs;
SHOW INDEX FROM docs;
```
+## Frequently asked questions {#inverted-index-faq}
+
+### Does Doris 4.1.5 support Japanese tokenization and typo tolerance?
+
+Doris 4.1.5 does not support the built-in Kuromoji tokenizer. Evaluate the
`icu` or `unicode` parser with representative Japanese text and inspect its
output with `TOKENIZE()`; their segmentation is different from Kuromoji
morphological analysis. Check the documentation for the exact deployed release
before selecting a newer parser.
+
+Prefix, regular-expression, and phrase-slop matching are supported by the
[search operators](./search-operators.md). They do not provide edit-distance
spelling correction: a misspelled term is not automatically expanded to nearby
dictionary terms.
+
## Further reading
### Core documentation
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]