This is an automated email from the ASF dual-hosted git repository.
hello-stephen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 20de5cfbc8c [fix](paimon) Align BLOB Thrift fields with branch-4.1
(#70073)
20de5cfbc8c is described below
commit 20de5cfbc8c72430747d7ba3a09d1425d1774b44
Author: HappenLee <[email protected]>
AuthorDate: Wed Nov 18 14:12:34 2026 +0800
[fix](paimon) Align BLOB Thrift fields with branch-4.1 (#70073)
### What problem does this PR solve?
Related PR: #69658
Problem Summary: `TQueryOptions` field 237 represents
`paimon_blob_max_size`
on branch-4.1 and the analytic spill limit on master. A non-default BLOB
limit is therefore decoded as the spill limit when these definitions
meet.
Master also reserves the branch-4.1 BLOB scan metadata IDs without
defining
the corresponding fields.
Align the BLOB protocol while preserving the existing Parquet and vector
search assignments:
- Keep optional `i64 paimon_blob_max_size` at field **237**, with its 10
MiB
default. Move `spill_analytic_sink_mem_limit_bytes` to unused field
**240**,
with its 64 MiB default and unchanged accessors.
- Preserve `enable_parquet_write_page_index` at **238** and
`parquet_write_data_page_size` at **239**, matching branch-4.2.
- Add `TPaimonBlobFileDesc` with optional `path`, `file_size`,
`first_row_id`,
`row_count` and `field_id` at IDs **1-5**, matching branch-4.1.
- Define `TPaimonFileDesc.blob_files` at **21**, `anchor_first_row_id`
at **22**
and `anchor_row_count` at **23**. Preserve master's `original_file_path`
at **20** and `vector_payload` at **24**, including its enum values.
The anchor fields describe the complete Parquet anchor's logical row
range;
byte splits share the same origin rather than using their first returned
row.
The assignments were checked against current branch-3.1, branch-4.0,
branch-4.1, branch-4.2 and master definitions. This aligns protocol
definitions;
BLOB materialization and FILE execution support remain separate changes.
### Release note
Align BLOB query and scan metadata with branch-4.1, preserving the
existing
Parquet control IDs used by branch-4.2. Older master builds using field
237
for analytic spill are not wire compatible with this correction: deploy
matching FE/BE builds when replacing those builds. Earlier revisions of
this
PR assigned spill to 238; those unmerged builds also require matching
rebuilt
FE/BE artifacts after this update. This is not a guarantee of general
cross-version query-plan compatibility.
### Check List (For Author)
- Test: Java/C++/Python Thrift code generation. Binary and Compact
protocol
checks using actual generated original-PR, current-master, branch-4.1,
branch-4.2 and rebased types (26 check groups): preserve non-default
BLOB
limits with branch-4.1, preserve non-default Parquet controls with
branch-4.2, retain defaults and the independent spill limit at 240,
verify the old-master/former-PR boundaries, round-trip every BLOB and
anchor field, retain fields 20/24 and opaque vector bytes with master
types, and handle absent optional fields. `git diff --check`.
Full FE/BE builds and SQL regression were not run for this IDL-only
change.
- Behavior changed: Yes. Field 237 represents the BLOB limit and spill
uses
240. Parquet IDs 238/239, BLOB scan IDs 21-23, existing IDs 20/24 and
all
defaults are preserved.
- Does this need documentation: No. The rollout boundary is documented
above.
---
gensrc/thrift/PaloInternalService.thrift | 7 +++++--
gensrc/thrift/PlanNodes.thrift | 21 +++++++++++++++++----
2 files changed, 22 insertions(+), 6 deletions(-)
diff --git a/gensrc/thrift/PaloInternalService.thrift
b/gensrc/thrift/PaloInternalService.thrift
index 56427cfcb73..15e675e9945 100644
--- a/gensrc/thrift/PaloInternalService.thrift
+++ b/gensrc/thrift/PaloInternalService.thrift
@@ -535,11 +535,14 @@ struct TQueryOptions {
// FE can explicitly and idempotently acknowledge external-file commit
reports.
236: optional bool supports_external_file_report_ack = false;
- // Memory limit for one non-streaming analytic partition before it
proactively spills.
- 237: optional i64 spill_analytic_sink_mem_limit_bytes = 67108864;
+ // Keep field 237 aligned with branch-4.1 Paimon BLOB query options.
+ 237: optional i64 paimon_blob_max_size = 10485760;
+
// Optional Parquet output indexes; absent fields retain the existing writer
defaults.
238: optional bool enable_parquet_write_page_index = true;
239: optional i32 parquet_write_data_page_size = 1048576;
+ // Memory limit for one non-streaming analytic partition before it
proactively spills.
+ 240: optional i64 spill_analytic_sink_mem_limit_bytes = 67108864;
// For cloud, to control if the content would be written into file cache
// In write path, to control if the content would be written into file cache.
// In read path, read from file cache or remote storage when execute query.
diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift
index 82f89454a19..2c3df7796bb 100644
--- a/gensrc/thrift/PlanNodes.thrift
+++ b/gensrc/thrift/PlanNodes.thrift
@@ -413,6 +413,16 @@ struct TPaimonVectorPayload {
2: optional binary bytes;
}
+// A managed BLOB column file, aligned with a Parquet anchor by absolute
Paimon row id.
+struct TPaimonBlobFileDesc {
+ 1: optional string path;
+ 2: optional i64 file_size;
+ // This .blob file's own logical row range: [first_row_id, first_row_id +
row_count).
+ 3: optional i64 first_row_id;
+ 4: optional i64 row_count;
+ 5: optional i32 field_id;
+}
+
struct TPaimonFileDesc {
1: optional string paimon_split
2: optional string paimon_column_names
@@ -439,10 +449,13 @@ struct TPaimonFileDesc {
// to materialize the public file-location metadata column. This
master-only field is appended after
// the branch-4.1 fields instead of reusing field 18.
20: optional string original_file_path;
- // Reserved for branch-4.1 fields:
- // 21: optional list<TPaimonBlobFileDesc> blob_files
- // 22: optional i64 anchor_first_row_id
- // 23: optional i64 anchor_row_count
+ // Keep BLOB scan metadata aligned with branch-4.1.
+ 21: optional list<TPaimonBlobFileDesc> blob_files
+ // Logical row ID of row position 0 in the complete Parquet anchor. Byte
splits of
+ // the same anchor share this origin; this is not the first row returned
by a split.
+ 22: optional i64 anchor_first_row_id
+ // Complete anchor range: [anchor_first_row_id, anchor_first_row_id +
anchor_row_count).
+ 23: optional i64 anchor_row_count
// Reserved for the per-split Paimon primary-key vector search payload in
#69493.
24: optional TPaimonVectorPayload vector_payload;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]