This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 86b74d856b11 [doc](lakehouse) Document Parquet page-index write
controls (#4581)
86b74d856b11 is described below
commit 86b74d856b11f04d9fc5888e6da3d3ef3c83c168
Author: Gabriel <[email protected]>
AuthorDate: Wed Nov 4 13:06:50 2026 +0800
[doc](lakehouse) Document Parquet page-index write controls (#4581)
Describe the new Parquet output session controls in the English and
Chinese current documentation: `enable_parquet_write_page_index`
(default `true`) and `parquet_write_data_page_size` (default 1 MiB).
Include an OUTFILE example, disabling/restoring the settings,
Hive/Iceberg/EXPORT scope, the Hadoop-framed LZ4 page target cap, and
tuning tradeoffs.
Implementation: https://github.com/apache/doris/pull/69971
Version scope: current/Dev only. These controls are newly introduced on
master and are not available in released 4.x or 3.x, so their versioned
pages are intentionally unchanged.
Validation: both changed pages compile as Docusaurus-compatible MDX;
`git diff --check` passes. Changed-link lint reports two existing
`parquet_metadata_size_limit` anchor findings already present before
this change; the new OUTFILE links resolve. I18n lint flags the
intentionally current-only scope described above. A full website build
was not run.
---
docs/lakehouse/file-formats/parquet.md | 47 ++++++++++++++++++++++
.../current/lakehouse/file-formats/parquet.md | 47 ++++++++++++++++++++++
2 files changed, 94 insertions(+)
diff --git a/docs/lakehouse/file-formats/parquet.md
b/docs/lakehouse/file-formats/parquet.md
index e9182d634ffb..24be88ccceda 100644
--- a/docs/lakehouse/file-formats/parquet.md
+++ b/docs/lakehouse/file-formats/parquet.md
@@ -43,6 +43,53 @@ For the complete UUID output mappings in OUTFILE and EXPORT,
see [Column Type Ma
* lzo
* brotli
+## Writing Page Indexes (Dev Version)
+
+Doris can optionally write Parquet `ColumnIndex` and `OffsetIndex` metadata
when exporting with OUTFILE or EXPORT, or inserting into Hive and Iceberg
tables stored as Parquet. A reader that supports these indexes can locate
candidate pages by value and keep rows aligned across columns. Existing files
are unchanged.
+
+Set the following session variables before the write statement. They apply to
Parquet output only and do not enable page-index filtering on the read side.
+
+| Variable | Type | Default | Behavior |
+| --- | --- | --- | --- |
+| `enable_parquet_write_page_index` | Boolean | `true` | Writes page indexes
for new Parquet files. `ColumnIndex` depends on statistics being available for
the column; `OffsetIndex` records data-page locations and row positions. |
+| `parquet_write_data_page_size` | Integer, bytes | `1048576` (1 MiB) | Target
uncompressed data-page size, in the range `1` to `2147483647`. Applies whether
index writing is enabled or disabled. It does not set the row-group size or
dictionary-page size. |
+
+Page size is a target: a value or a write batch can exceed it. For
Hadoop-framed LZ4, used by Hive/Iceberg `lz4` output, Doris caps this target at
`131072` bytes (128 KiB) to retain compatibility with Hadoop readers. Raw LZ4
output is not subject to this codec-specific cap.
+
+For example, after preparing a writable S3 bucket and credentials as described
in [OUTFILE](../../data-operate/export/outfile.md), export a query result with
indexes and a 256 KiB page target. Replace the storage placeholders with your
own configuration:
+
+```sql
+SET enable_parquet_write_page_index = true;
+SET parquet_write_data_page_size = 262144;
+
+SELECT 1 AS id, 'example' AS label
+INTO OUTFILE 's3://example-bucket/export/result_'
+FORMAT AS PARQUET
+PROPERTIES (
+ "s3.endpoint" = "https://s3.example.com";,
+ "s3.region" = "us-east-1",
+ "s3.access_key" = "<access_key>",
+ "s3.secret_key" = "<secret_key>"
+);
+```
+
+The resulting files include page-index metadata for eligible columns. For Hive
or Iceberg, set the same variables in the session before `INSERT INTO` or
`INSERT OVERWRITE`; the destination table must already be configured to write
Parquet. For EXPORT, set them before submitting the job.
+
+Index writing defaults to `true`, preserving the existing dev writer behavior.
To omit page indexes from subsequent output, set the option before writing:
+
+```sql
+SET enable_parquet_write_page_index = false;
+```
+
+To restore the defaults in the current session:
+
+```sql
+SET enable_parquet_write_page_index = true;
+SET parquet_write_data_page_size = 1048576;
+```
+
+Indexes add metadata and write-side work. Smaller pages can improve selective
reads, especially when values are clustered, but also create more index entries
and page headers. Benefits depend on the consuming engine and data
distribution; enabling indexes does not guarantee faster queries. Readers can
still read the files without using the optional indexes. These settings do not
change the Parquet format version or data-page version.
+
## Parameters
### Timestamp and Date Compatibility (Since 4.1.5)
{#timestamp-and-date-compatibility}
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/file-formats/parquet.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/file-formats/parquet.md
index 7dd841a36770..105f7b5b4de7 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/file-formats/parquet.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/lakehouse/file-formats/parquet.md
@@ -53,6 +53,53 @@ OUTFILE 和 EXPORT 导出 UUID 的完整类型映射见[导出文件列类型映
* brotli
+## 写入页索引（Dev 版本）
+
+Doris 支持在 OUTFILE、EXPORT 导出，以及向使用 Parquet 格式的 Hive、Iceberg 表写入数据时，可选输出 Parquet
`ColumnIndex` 和 `OffsetIndex`
元数据。支持这些索引的读取引擎可以按列值定位候选数据页，并保持各列读取的行范围一致。已有文件不会被修改。
+
+在写入语句执行前设置以下会话变量。它们仅作用于 Parquet 输出，不会开启读取侧的页索引过滤。
+
+| 变量 | 类型 | 默认值 | 行为 |
+| --- | --- | --- | --- |
+| `enable_parquet_write_page_index` | Boolean | `true` | 为新生成的 Parquet
文件写入页索引。`ColumnIndex` 依赖该列是否具有可用的统计信息；`OffsetIndex` 记录数据页的位置和行位置。 |
+| `parquet_write_data_page_size` | Integer，字节 | `1048576`（1 MiB） |
未压缩数据页的目标大小，范围为 `1` 至 `2147483647`。开启或关闭索引写入时均生效，不控制 Row Group 大小或字典页大小。 |
+
+页大小是目标值，单个值或一次写入批次可能使数据页超过该值。对于 Hive/Iceberg `lz4` 输出使用的 Hadoop 帧格式 LZ4，Doris
将该目标值限制在 `131072` 字节（128 KiB）以内，以保持与 Hadoop 读取端的兼容性。Raw LZ4 输出不受此压缩格式专属限制。
+
+例如，按 [OUTFILE](../../data-operate/export/outfile.md) 文档准备可写的 S3 Bucket
和访问凭据后，可以将查询结果导出为带索引的 Parquet 文件，并将目标页大小设为 256 KiB。请将存储占位符替换为实际配置：
+
+```sql
+SET enable_parquet_write_page_index = true;
+SET parquet_write_data_page_size = 262144;
+
+SELECT 1 AS id, 'example' AS label
+INTO OUTFILE 's3://example-bucket/export/result_'
+FORMAT AS PARQUET
+PROPERTIES (
+ "s3.endpoint" = "https://s3.example.com";,
+ "s3.region" = "us-east-1",
+ "s3.access_key" = "<access_key>",
+ "s3.secret_key" = "<secret_key>"
+);
+```
+
+生成的文件会为满足条件的列包含页索引元数据。对于 Hive 或 Iceberg，在执行 `INSERT INTO` 或 `INSERT OVERWRITE`
前设置相同的会话变量即可；目标表须已配置为写入 Parquet。对于 EXPORT，须在提交任务前设置。
+
+索引写入默认 `true`，保持 dev 版本现有的写入行为。若后续输出文件不需要页索引，可在写入前关闭：
+
+```sql
+SET enable_parquet_write_page_index = false;
+```
+
+可在当前会话中恢复默认值：
+
+```sql
+SET enable_parquet_write_page_index = true;
+SET parquet_write_data_page_size = 1048576;
+```
+
+页索引会增加元数据和写入开销。较小的数据页可能有利于高选择性查询，尤其是数据按过滤列聚集时，但也会生成更多索引条目和页头。收益取决于读取引擎和数据分布，开启索引不保证查询加速；读取端不使用这些可选索引时，仍可正常读取文件。这两个变量不改变
Parquet 格式版本或数据页版本。
+
## 相关参数
### 时间戳与日期兼容性（4.1.5 起）
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
