This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b5996fbf108 [chore](paimon) Upgrade paimon-rust and reserve ANN Thrift
fields (#70075)
b5996fbf108 is described below
commit b5996fbf10831d331c37c38867ad79718b5b5910
Author: Socrates <[email protected]>
AuthorDate: Wed Nov 4 10:12:43 2026 +0800
[chore](paimon) Upgrade paimon-rust and reserve ANN Thrift fields (#70075)
### What problem does this PR solve?
Related PR: #69493
Problem Summary: Master still builds paimon-rust v0.4.0-rc1, while the
Paimon vector-search work in #69493 needs v0.5.0-rc1 and additional wire
fields. Upgrade the native dependency and reserve its Thrift layout on
master so later changes cannot reuse the same identifiers for different
meanings.
The dependency change updates the source URL, archive name, source
directory and MD5 checksum together. The schema change adds the
following optional fields and their Paimon payload/options types:
| Structure | Field | ID |
| --- | --- | --- |
| `TPaimonFileDesc` | `vector_payload` | 24 |
| `TExternalSearchRequest` | `paimon_options` | 6 |
| `TFileScanRangeParams` | `external_search_request` | 42 |
`TPaimonVectorPayloadType` keeps `BUCKET_SPLIT_BYTES=0` and
`MATERIALIZE_OPAQUE_BYTES=1`. Both payload members and all Paimon
search-option members are optional. Master does not emit these fields
yet; this PR reserves the protocol without enabling SQL/TVF ANN search
or upgrading the Paimon Java SDK.
Compatibility notes:
- `TPaimonFileDesc.20` remains `original_file_path`; comments reserve
branch-4.1's `blob_files=21`, `anchor_first_row_id=22` and
`anchor_row_count=23`. The vector payload uses 24 rather than the
conflicting 21 from the original #69493 diff.
- Comments reserve the existing branch-4.1/4.2 shared-search fields
`search_query=2`, `search_filter=3`, `vector_search_options=4` and
`prefilter=5`. Their query types can be introduced independently on
master.
- `TFileScanRangeParams.41` remains `fluss_properties`.
- The affected definitions were reviewed against current master and
branches 3.1, 4.0, 4.1 and 4.2. Existing field IDs and enum values
retain their meanings.
### Release note
Upgrade the native Paimon Rust reader dependency to v0.5.0-rc1.
### Check List (For Author)
- Test: No local build, Thrift generation or unit tests on master, as
requested; CI validation pending. During the preceding branch-4.2 work,
the archive checksum was verified and the v0.5.0-rc1 dependency was
built with Rust 1.94.0. This is not a completed master BE build or BE
unit-test run.
- Behavior changed: Yes; the bundled native Paimon dependency is
upgraded. The new Thrift fields are reserved and remain unset by master
producers.
- Does this need documentation: No
---
gensrc/thrift/PlanNodes.thrift | 43 +++++++++++++++++++++++++++++++++++++++++-
thirdparty/vars.sh | 8 ++++----
2 files changed, 46 insertions(+), 5 deletions(-)
diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift
index cd02c8127ec..fefb20b140f 100644
--- a/gensrc/thrift/PlanNodes.thrift
+++ b/gensrc/thrift/PlanNodes.thrift
@@ -381,6 +381,20 @@ enum TPaimonReaderType {
PAIMON_RUST = 3,
}
+// Reserved for Paimon primary-key vector search. Keep wire values aligned
with branch-4.2.
+enum TPaimonVectorPayloadType {
+ // Serialized BucketVectorSearchSplit for candidate generation.
+ BUCKET_SPLIT_BYTES = 0,
+ // Opaque payload for delayed materialization.
+ MATERIALIZE_OPAQUE_BYTES = 1,
+}
+
+// Reserved for #69493. The payload contains raw bytes, not a base64-encoded
split.
+struct TPaimonVectorPayload {
+ 1: optional TPaimonVectorPayloadType payload_type;
+ 2: optional binary bytes;
+}
+
struct TPaimonFileDesc {
1: optional string paimon_split
2: optional string paimon_column_names
@@ -407,6 +421,12 @@ struct TPaimonFileDesc {
// to materialize the public file-location metadata column. This
master-only field is appended after
// the branch-4.1 fields instead of reusing field 18.
20: optional string original_file_path;
+ // Reserved for branch-4.1 fields:
+ // 21: optional list<TPaimonBlobFileDesc> blob_files
+ // 22: optional i64 anchor_first_row_id
+ // 23: optional i64 anchor_row_count
+ // Reserved for the per-split Paimon primary-key vector search payload in
#69493.
+ 24: optional TPaimonVectorPayload vector_payload;
}
struct TTrinoConnectorFileDesc {
@@ -529,10 +549,28 @@ enum TTextSerdeType {
HIVE_TEXT_SERDE = 1,
}
+// Reserved for Paimon vector search in #69493. These options are not emitted
on master yet.
+struct TPaimonVectorSearchOptions {
+ 1: optional map<string, string> options
+ // TVF _distance uses squared L2 or negative inner product. Unset
preserves ordinary SQL.
+ 2: optional bool tvf_distance = false
+ // Only the TVF's explicit prefilter affects candidate generation; outer
WHERE stays residual.
+ 3: optional list<Exprs.TExpr> prefilter
+}
+
// A provider-independent logical search request. Physical target information
remains in the
-// provider FileDesc (for example, dataset_uri/version/fragment_ids in
TLanceFileDesc).
+// provider FileDesc (for example, dataset_uri/version/fragment_ids in
TLanceFileDesc, or
+// vector_payload in TPaimonFileDesc).
struct TExternalSearchRequest {
1: optional i32 schema_version = 1
+ // Reserved for branch-4.1 / branch-4.2 fields. Keep these ids until the
shared search
+ // query types are introduced on master:
+ // 2: optional TExternalSearchQuery search_query
+ // 3: optional TSearchFilter search_filter
+ // 4: optional TVectorSearchOptions vector_search_options
+ // 5: optional bool prefilter
+ // Reserved for Paimon-specific vector search tuning in #69493.
+ 6: optional TPaimonVectorSearchOptions paimon_options
}
struct TLanceScanParams {
@@ -636,6 +674,9 @@ struct TFileScanRangeParams {
// Fluss scan-level properties (bootstrap servers, table identity,
client/table options,
// projected columns). Set at ScanNode level to avoid redundant
serialization in each split.
41: optional map<string, string> fluss_properties
+ // Reserved for the scan-level Paimon vector search request in #69493.
+ // The per-split physical payload stays in TPaimonFileDesc.vector_payload.
+ 42: optional TExternalSearchRequest external_search_request
}
struct TFileRangeDesc {
diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh
index 6a08d306cef..76c00f8c250 100644
--- a/thirdparty/vars.sh
+++ b/thirdparty/vars.sh
@@ -626,10 +626,10 @@
LANCE_C_SOURCE="lance-c-a51ab47a9c814a746cb1714c99fad93dd3a188b8"
LANCE_C_MD5SUM="d1282cff3d63acfc0450db1f00d6ab6f"
# paimon-rust
-PAIMON_RUST_DOWNLOAD="https://github.com/apache/paimon-rust/archive/refs/tags/v0.4.0-rc1.tar.gz";
-PAIMON_RUST_NAME="paimon-rust-v0.4.0-rc1.tar.gz"
-PAIMON_RUST_SOURCE="paimon-rust-0.4.0-rc1"
-PAIMON_RUST_MD5SUM="23cd0634692108debf77a26ef4dc20a7"
+PAIMON_RUST_DOWNLOAD="https://github.com/apache/paimon-rust/archive/refs/tags/v0.5.0-rc1.tar.gz";
+PAIMON_RUST_NAME="paimon-rust-v0.5.0-rc1.tar.gz"
+PAIMON_RUST_SOURCE="paimon-rust-0.5.0-rc1"
+PAIMON_RUST_MD5SUM="fabf721bb26c6a2c165a00b65f074ac4"
# all thirdparties which need to be downloaded is set in array TP_ARCHIVES
export TP_ARCHIVES=(
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
