morningman opened a new pull request, #70681:
URL: https://github.com/apache/doris/pull/70681
### Background
Lance datasets store schemas, versions and fragment lists in manifests. FE
reads those manifests to plan a query, and BE reads the fragments through
lance-c. Doris 4.1/4.2 provide Lance catalogs in fe-core; master uses connector
plugins and already has the BE reader (#69581) and legacy image migration
(#69575).
### What problem does this PR solve?
Issue Number: #70680
Related PR: #65730, #66581, #66597, #66805 (source read-path changes);
#70338 (secret masking), #70351 (TIMEV2 precision), #69581 (BE reader), #69575
(legacy images)
PR Type: Feature
Problem Summary:
Master has no plugin serving the `lance` catalog type, so users cannot query
Lance catalogs after upgrading from 4.1/4.2. This ports the first part of that
read path: filesystem catalogs, static S3 credentials, Arrow type mapping,
basic scans and main-chain time travel. The plugin also needs JNI packaging and
resource management; other catalog types keep their existing behavior.
### Details
- **Change map:** 54 files, +6246/-2: production Java +3221, tests +2599,
build/resources +426/-2. Production has four parts: property binding and
namespace lifetime; schema and version resolution; S3 storage-option
translation shared by FE/BE; fragment scan planning. Tests cover those
contracts, native datasets and a regression suite. Packaging registers the
module, plugin distribution, JNI replacement and MinIO fixture.
- **Scope:** Local/file and S3 filesystem catalogs; `SHOW DATABASES/TABLES`,
`DESC`, `SELECT`, `FOR VERSION AS OF n`, and `FOR TIME AS OF 'yyyy-MM-dd
HH:mm:ss[.SSS]'`. REST catalogs fail explicitly until the next PR, including
first access to replayed REST catalogs. Tags, branches, OSS warehouses,
pushdown and writes are outside this PR.
- **Design:** Follow the existing connector plugins, rather than add Lance
catalog classes to fe-core. The plugin opts into MVCC and memoizes metadata per
statement, so schema, fragments and BE ranges use one immutable dataset
version. Time travel compares manifest commit times at their original precision
in the session time zone. Each fragment becomes one scan range. An in-flight
call count delays native resource disposal after catalog retirement. Proposed
in #70680, opened with this PR; no community design discussion has taken place
yet.
- **Usage:**
```sql
CREATE CATALOG lance_s3 PROPERTIES (
"type" = "lance",
"warehouse" = "s3://warehouse/lance",
"s3.endpoint" = "http://127.0.0.1:9000",
"s3.access_key" = "admin",
"s3.secret_key" = "password",
"s3.region" = "us-east-1",
"use_path_style" = "true"
);
SELECT count(*) FROM lance_s3.`default`.all_types FOR VERSION AS OF 1;
SELECT count(*) FROM lance_s3.`default`.all_types
FOR TIME AS OF '2026-07-30 10:11:50';
```
- **Default behavior:** Enabled by creating a Lance catalog. `lance.conf`
contains the optional `lance_catalog_arrow_memory_limit_bytes`, default 256 MiB
per catalog; changes require an FE restart.
- **Compatibility:** Preserve the 4.1/4.2 property map and names. Legacy
catalogs use the existing image migration. The Arrow memory setting moves from
fe.conf to lance.conf; copy any customized value. `SHOW CREATE TABLE` uses
`ENGINE=lance`, as other plugins do. No Thrift or persisted-metadata format
changes. BE needs #69581; projecting Arrow Null, Duration, JSON or bfloat16
fences out smooth-upgrade source BEs. Newer catalog features remain unsupported
during this staged port. Both cloud and shared-nothing use this planning path;
a cloud deployment was not tested locally.
- **Persistence and failure handling:** Catalog metadata uses the existing
plugin catalog journal/image path; dataset contents are read-only. The
namespace opens lazily, avoiding storage I/O and JNI loading during replay.
Missing versions and unavailable native libraries fail explicitly. Statement
cleanup releases its metadata memo; dropping or altering a catalog retires its
client and closes resources after active calls finish.
- **Security and privileges:** CREATE_PRIV permits catalog creation;
ALTER_PRIV is required to change its properties. FE and BE make outbound HTTP
requests to the configured S3 endpoint, an SSRF surface. The provider declares
S3-compatible endpoint aliases for the existing CREATE/ALTER gate when
`security_checker_class_name` is configured. REST credential keys are declared
for masking even though REST reads are deferred. Build-time JNI replacement
downloads a fixed, checksum-pinned asset.
- **Follow-up plan:** REST/OSS and namespace-vended credentials;
predicate/LIMIT/COUNT pushdown, fragment grouping and scalar indexes; then
session/table caches and profile counters. Search TVFs, index operations and
tag/branch/managed versioning follow separately under #70680.
### Testing
- Local macOS arm64/JDK 17: 71 Lance tests in 13 classes, including all six
native dataset tests with no skips. SPI: 162 tests. Relevant FE scan, MVCC,
image replay, type conversion, plugin loading and endpoint-gate tests: 507
tests in 43 classes. All passed with the Maven build cache disabled and
Checkstyle enabled.
- Two provider mutations: removing the primary S3 endpoint or the API-key
masking declaration fails the corresponding assertion; the unmodified control
passes.
- `build.sh --fe` completed successfully, including Checkstyle and both
Lance/MongoDB plugin distributions at API 32.0. Shell syntax checks passed for
the build, FE test and JNI helper scripts.
- `test_lance_catalog_all_types` passed in compare mode against standalone
MinIO using the new FE and this worktree's BE `33d48fcf6da`; Lance reader
sources match master. Three shared result blocks match the 4.2 snapshot. A
non-root user is denied CREATE/ALTER before the corresponding grants, then
creates, alters and reads the pinned dataset after explicit grants. The BE was
not rebuilt at current master.
- Historical packaging check: the pinned Linux x86_64 JNI entry was replaced
and verified on macOS, leaving the Darwin entry unchanged. The helper is
unchanged; a glibc 2.17 Linux host has not been tested.
### Notes for Reviewers
- The BE Lance reader does not consume `history_schema_info`. Omitting it
retains FileScannerV2's exact-name matching, which the reader tests exercise.
- Native FE tests skip explicitly on hosts unable to load Lance or Arrow C
Data JNI. The External pipeline enables the MinIO fixture and runs the new
regression suite.
### Release note
Add filesystem-backed Lance catalogs on master, with static S3 credentials,
basic reads and main-chain version/time travel. Move the catalog Arrow memory
setting to the Lance plugin's lance.conf.
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: Yes (update master Lance catalog
documentation for plugin installation, lance.conf migration and the staged
scope; documentation is maintained separately)
- Pre-submission status: Partially verified: the local regression uses
standalone MinIO and a compatible BE built in this worktree; a complete
current-master BE build, the full External environment, a cloud cluster and a
glibc 2.17 Linux host were not verified
### AI Disclosure
- Tool: Claude Code (Claude Opus 5.5); Codex
- Generated: code, tests, this description
- Human review: not stated
- Prompts: The author asked to port the branch-4.2 Lance read path as a
connector plugin, split it into four PRs, include main-chain time travel in the
first, and resume from the handoff after its prerequisite PRs merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]