This is an automated email from the ASF dual-hosted git repository.

hello-stephen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new 20de5cfbc8c [fix](paimon) Align BLOB Thrift fields with branch-4.1 
(#70073)
20de5cfbc8c is described below

commit 20de5cfbc8c72430747d7ba3a09d1425d1774b44
Author: HappenLee <[email protected]>
AuthorDate: Wed Nov 18 14:12:34 2026 +0800

    [fix](paimon) Align BLOB Thrift fields with branch-4.1 (#70073)
    
    ### What problem does this PR solve?
    
    Related PR: #69658
    
    Problem Summary: `TQueryOptions` field 237 represents
    `paimon_blob_max_size`
    on branch-4.1 and the analytic spill limit on master. A non-default BLOB
    limit is therefore decoded as the spill limit when these definitions
    meet.
    Master also reserves the branch-4.1 BLOB scan metadata IDs without
    defining
    the corresponding fields.
    
    Align the BLOB protocol while preserving the existing Parquet and vector
    search assignments:
    
    - Keep optional `i64 paimon_blob_max_size` at field **237**, with its 10
    MiB
    default. Move `spill_analytic_sink_mem_limit_bytes` to unused field
    **240**,
      with its 64 MiB default and unchanged accessors.
    - Preserve `enable_parquet_write_page_index` at **238** and
      `parquet_write_data_page_size` at **239**, matching branch-4.2.
    - Add `TPaimonBlobFileDesc` with optional `path`, `file_size`,
    `first_row_id`,
      `row_count` and `field_id` at IDs **1-5**, matching branch-4.1.
    - Define `TPaimonFileDesc.blob_files` at **21**, `anchor_first_row_id`
    at **22**
    and `anchor_row_count` at **23**. Preserve master's `original_file_path`
      at **20** and `vector_payload` at **24**, including its enum values.
    
    The anchor fields describe the complete Parquet anchor's logical row
    range;
    byte splits share the same origin rather than using their first returned
    row.
    The assignments were checked against current branch-3.1, branch-4.0,
    branch-4.1, branch-4.2 and master definitions. This aligns protocol
    definitions;
    BLOB materialization and FILE execution support remain separate changes.
    
    ### Release note
    
    Align BLOB query and scan metadata with branch-4.1, preserving the
    existing
    Parquet control IDs used by branch-4.2. Older master builds using field
    237
    for analytic spill are not wire compatible with this correction: deploy
    matching FE/BE builds when replacing those builds. Earlier revisions of
    this
    PR assigned spill to 238; those unmerged builds also require matching
    rebuilt
    FE/BE artifacts after this update. This is not a guarantee of general
    cross-version query-plan compatibility.
    
    ### Check List (For Author)
    
    - Test: Java/C++/Python Thrift code generation. Binary and Compact
    protocol
      checks using actual generated original-PR, current-master, branch-4.1,
    branch-4.2 and rebased types (26 check groups): preserve non-default
    BLOB
      limits with branch-4.1, preserve non-default Parquet controls with
      branch-4.2, retain defaults and the independent spill limit at 240,
      verify the old-master/former-PR boundaries, round-trip every BLOB and
      anchor field, retain fields 20/24 and opaque vector bytes with master
      types, and handle absent optional fields. `git diff --check`.
    Full FE/BE builds and SQL regression were not run for this IDL-only
    change.
    - Behavior changed: Yes. Field 237 represents the BLOB limit and spill
    uses
    240. Parquet IDs 238/239, BLOB scan IDs 21-23, existing IDs 20/24 and
    all
      defaults are preserved.
    - Does this need documentation: No. The rollout boundary is documented
    above.
---
 gensrc/thrift/PaloInternalService.thrift |  7 +++++--
 gensrc/thrift/PlanNodes.thrift           | 21 +++++++++++++++++----
 2 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/gensrc/thrift/PaloInternalService.thrift 
b/gensrc/thrift/PaloInternalService.thrift
index 56427cfcb73..15e675e9945 100644
--- a/gensrc/thrift/PaloInternalService.thrift
+++ b/gensrc/thrift/PaloInternalService.thrift
@@ -535,11 +535,14 @@ struct TQueryOptions {
 
   // FE can explicitly and idempotently acknowledge external-file commit 
reports.
   236: optional bool supports_external_file_report_ack = false;
-  // Memory limit for one non-streaming analytic partition before it 
proactively spills.
-  237: optional i64 spill_analytic_sink_mem_limit_bytes = 67108864;
+  // Keep field 237 aligned with branch-4.1 Paimon BLOB query options.
+  237: optional i64 paimon_blob_max_size = 10485760;
+
   // Optional Parquet output indexes; absent fields retain the existing writer 
defaults.
   238: optional bool enable_parquet_write_page_index = true;
   239: optional i32 parquet_write_data_page_size = 1048576;
+  // Memory limit for one non-streaming analytic partition before it 
proactively spills.
+  240: optional i64 spill_analytic_sink_mem_limit_bytes = 67108864;
   // For cloud, to control if the content would be written into file cache
   // In write path, to control if the content would be written into file cache.
   // In read path, read from file cache or remote storage when execute query.
diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift
index 82f89454a19..2c3df7796bb 100644
--- a/gensrc/thrift/PlanNodes.thrift
+++ b/gensrc/thrift/PlanNodes.thrift
@@ -413,6 +413,16 @@ struct TPaimonVectorPayload {
     2: optional binary bytes;
 }
 
+// A managed BLOB column file, aligned with a Parquet anchor by absolute 
Paimon row id.
+struct TPaimonBlobFileDesc {
+    1: optional string path;
+    2: optional i64 file_size;
+    // This .blob file's own logical row range: [first_row_id, first_row_id + 
row_count).
+    3: optional i64 first_row_id;
+    4: optional i64 row_count;
+    5: optional i32 field_id;
+}
+
 struct TPaimonFileDesc {
     1: optional string paimon_split
     2: optional string paimon_column_names
@@ -439,10 +449,13 @@ struct TPaimonFileDesc {
     // to materialize the public file-location metadata column. This 
master-only field is appended after
     // the branch-4.1 fields instead of reusing field 18.
     20: optional string original_file_path;
-    // Reserved for branch-4.1 fields:
-    // 21: optional list<TPaimonBlobFileDesc> blob_files
-    // 22: optional i64 anchor_first_row_id
-    // 23: optional i64 anchor_row_count
+    // Keep BLOB scan metadata aligned with branch-4.1.
+    21: optional list<TPaimonBlobFileDesc> blob_files
+    // Logical row ID of row position 0 in the complete Parquet anchor. Byte 
splits of
+    // the same anchor share this origin; this is not the first row returned 
by a split.
+    22: optional i64 anchor_first_row_id
+    // Complete anchor range: [anchor_first_row_id, anchor_first_row_id + 
anchor_row_count).
+    23: optional i64 anchor_row_count
     // Reserved for the per-split Paimon primary-key vector search payload in 
#69493.
     24: optional TPaimonVectorPayload vector_payload;
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to