pzhdfy opened a new pull request, #68988:
URL: https://github.com/apache/doris/pull/68988
## What problem does this PR solve?
Follow-up to #66227, which introduced the paimon-rust reader. This PR
upgrades the Paimon SDK to `2.1-community-release` and ports the primary-key
vector (ANN) search path onto the community stack: `ORDER BY
l2_distance_approximate|inner_product_approximate(col, vector) LIMIT n` is
rewritten into a pushdown onto the paimon scan and served by the paimon-rust
vector reader on BE.
**Problem Summary:**
**1. Upgrade Paimon to 2.1**
- Port FE off the paimon privilege package removed in 2.1 (`metastore`
privilege API changes).
- The 2.1 `S3FileIO` reads S3 credentials **only** from
`CatalogContext.options()` (verified at bytecode level: `S3FileIO#configure ->
loadHadoopConfigFromContext` never touches the Hadoop `Configuration`, unlike
1.4). Doris keeps a catalog's storage credentials in the Hadoop conf only, so
every s3-warehouse paimon catalog initialized the 2.1 SDK with no credentials
and failed with `Unable to load credentials`.
`mirrorHadoopS3OptionsToCatalogOptions` now mirrors the `fs.s3a.*` entries onto
the catalog options for the FileSystem/HMS/JDBC metastores, keeping user extras
(path style, connection limits) flowing as well.
- 2.1 aggregates overflow **fail loud** where 1.4 silently wrapped (e.g.
`byte overflow: 127 + 1 = 128` on TINYINT/SMALLINT/INT/BIGINT sum/product). The
`test_paimon_rust_reader_compatibility` cases are adapted to assert the error
and keep value coverage through safe operands.
**2. PK-vector (ANN) search**
- New rewrite rule `PushDownVectorTopNIntoPaimonScan` (registered in
`Rewriter`) matches a vector distance top-N over a paimon file scan and pushes
the query vector, the indexed vector column, the metric and the limit down to
the scan. A pre-flight check refuses incremental scans (`t@incr`), so an
incremental query can never silently resolve to a snapshot read.
- `AnnTopNInfo` carries the pushed-down info from `LogicalFileScan` to
`PhysicalFileScan` and onto `PaimonScanNode` (`PhysicalPlanTranslator` sets it
only inside the `PAIMON_EXTERNAL_TABLE` branch); the `PlanNodes.thrift`
additions are additive fields.
- BE: `paimon_rust_table_reader` grows the vector-search path, reading
through the rust PK-vector builder. `read.batch-size` is a no-op on this path
(the rust vector readers do not consume it) — noted in the reader.
- The pushdown reuses the existing `enable_paimon_rust_reader` and
`enable_file_scanner_v2` switches; with either off, the query brute-forces
through the normal scan.
**3. Unify the rust eligibility gates**
- `PaimonRustEligibility` evaluates the 12 table-level/storage/BE checks
once and is shared by the batch and vector paths. The vector gate is now
computed independently of the batch gate (it previously inherited
`fallbackRead` / `scan.mode` restrictions from the batch path) and both
branches share a common gate chain with a batch-only split-suitability suffix.
- Compaction-merge splits that are raw-convertible skip the merge-reader
gates.
- JNI fallback reasons are logged so a silent downgrade to the JNI reader is
diagnosable.
**Tests**
- FE UT: new `PushDownVectorTopNIntoPaimonScanTest` (positive /
incremental-refused / gate-negative coverage, 710 lines), `PaimonScanNodeTest`
gate coverage, planner `PaimonPredicateConverter` tests, `PaimonUtilTest`.
- New end-to-end regression `test_paimon_pk_vector_ann`: the suite builds
its own fixture through the Spark ThriftServer — an ivf-rq PK-vector index with
`nlist = nprobe = 4` on 512 deterministic 16-dim vectors, i.e. **exhaustive
recall, so ANN must equal brute force by construction**. It compares against
the brute-force answer computed with the rust reader disabled, asserts the ANN
leg really executed on the paimon-rust vector reader (profile check against a
silent brute-force pass), and covers both `l2` and `inner_product` metrics,
residual filters and non-index projections.
- Fixed two regression fixtures broken by the 2.1 Spark stack: the
creation-time-millis helper was shifted by the session UTC offset (every
`scan.creation-time-millis` leg silently resolved to the latest snapshot;
Doris-side resolution verified correct by hand-reproducing the queries), and
`assertSparkDorisResultEquals` now decodes the `0x`-hex tokens Doris renders
for VARBINARY nested in arrays/maps/structs so identical nested binary values
compare equal across engines.
- Swept all 37 locally runnable `external_table_p0/paimon` regression suites
against the upgraded SDK: no Doris-side regressions.
## Release note
The Paimon catalog now runs on the Paimon 2.1 SDK. Queries of the form
`ORDER BY l2_distance_approximate|inner_product_approximate(<vector col>,
<query vector>) LIMIT n` over paimon tables with a PK-vector index are pushed
down into the paimon-rust vector reader (requires `enable_paimon_rust_reader`
and `enable_file_scanner_v2`). Paimon 2.1 aggregates now fail loudly on integer
overflow instead of wrapping.
## Check List (For Author)
- Test
- [x] Unit test
- [x] Regression test
- [ ] Manual test (add the script if necessary)
- [ ] No need to test
- [ ] Behavior changed: yes (aggregate-overflow fail-loud semantics come
from the SDK; the ANN path is new and switch-gated)
- [x] Does this need documentation?
## Check List (For Reviewer who merge this PR)
- [ ] Does the PR have a corresponding release note?
- [ ] Is the release note filled in correctly?
- [ ] Are the test cases added or updated correctly?
- [ ] Is the document updated (if necessary)?
- [ ] Does this PR need to be picked to other branches? (branch-4.1 only:
**yes**)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
