morningman opened a new pull request, #70681:
URL: https://github.com/apache/doris/pull/70681

   ### Background
   
   Lance datasets store schemas, versions and fragment lists in manifests. FE 
reads those manifests to plan a query, and BE reads the fragments through 
lance-c. Doris 4.1/4.2 provide Lance catalogs in fe-core; master uses connector 
plugins and already has the BE reader (#69581) and legacy image migration 
(#69575).
   
   ### What problem does this PR solve?
   
   Issue Number: #70680
   Related PR: #65730, #66581, #66597, #66805 (source read-path changes); 
#70338 (secret masking), #70351 (TIMEV2 precision), #69581 (BE reader), #69575 
(legacy images)
   PR Type: Feature
   
   Problem Summary:
   
   Master has no plugin serving the `lance` catalog type, so users cannot query 
Lance catalogs after upgrading from 4.1/4.2. This ports the first part of that 
read path: filesystem catalogs, static S3 credentials, Arrow type mapping, 
basic scans and main-chain time travel. The plugin also needs JNI packaging and 
resource management; other catalog types keep their existing behavior.
   
   ### Details
   
   - **Change map:** 54 files, +6246/-2: production Java +3221, tests +2599, 
build/resources +426/-2. Production has four parts: property binding and 
namespace lifetime; schema and version resolution; S3 storage-option 
translation shared by FE/BE; fragment scan planning. Tests cover those 
contracts, native datasets and a regression suite. Packaging registers the 
module, plugin distribution, JNI replacement and MinIO fixture.
   - **Scope:** Local/file and S3 filesystem catalogs; `SHOW DATABASES/TABLES`, 
`DESC`, `SELECT`, `FOR VERSION AS OF n`, and `FOR TIME AS OF 'yyyy-MM-dd 
HH:mm:ss[.SSS]'`. REST catalogs fail explicitly until the next PR, including 
first access to replayed REST catalogs. Tags, branches, OSS warehouses, 
pushdown and writes are outside this PR.
   - **Design:** Follow the existing connector plugins, rather than add Lance 
catalog classes to fe-core. The plugin opts into MVCC and memoizes metadata per 
statement, so schema, fragments and BE ranges use one immutable dataset 
version. Time travel compares manifest commit times at their original precision 
in the session time zone. Each fragment becomes one scan range. An in-flight 
call count delays native resource disposal after catalog retirement. Proposed 
in #70680, opened with this PR; no community design discussion has taken place 
yet.
   - **Usage:**
   
     ```sql
     CREATE CATALOG lance_s3 PROPERTIES (
         "type" = "lance",
         "warehouse" = "s3://warehouse/lance",
         "s3.endpoint" = "http://127.0.0.1:9000";,
         "s3.access_key" = "admin",
         "s3.secret_key" = "password",
         "s3.region" = "us-east-1",
         "use_path_style" = "true"
     );
     SELECT count(*) FROM lance_s3.`default`.all_types FOR VERSION AS OF 1;
     SELECT count(*) FROM lance_s3.`default`.all_types
         FOR TIME AS OF '2026-07-30 10:11:50';
     ```
   
   - **Default behavior:** Enabled by creating a Lance catalog. `lance.conf` 
contains the optional `lance_catalog_arrow_memory_limit_bytes`, default 256 MiB 
per catalog; changes require an FE restart.
   - **Compatibility:** Preserve the 4.1/4.2 property map and names. Legacy 
catalogs use the existing image migration. The Arrow memory setting moves from 
fe.conf to lance.conf; copy any customized value. `SHOW CREATE TABLE` uses 
`ENGINE=lance`, as other plugins do. No Thrift or persisted-metadata format 
changes. BE needs #69581; projecting Arrow Null, Duration, JSON or bfloat16 
fences out smooth-upgrade source BEs. Newer catalog features remain unsupported 
during this staged port. Both cloud and shared-nothing use this planning path; 
a cloud deployment was not tested locally.
   - **Persistence and failure handling:** Catalog metadata uses the existing 
plugin catalog journal/image path; dataset contents are read-only. The 
namespace opens lazily, avoiding storage I/O and JNI loading during replay. 
Missing versions and unavailable native libraries fail explicitly. Statement 
cleanup releases its metadata memo; dropping or altering a catalog retires its 
client and closes resources after active calls finish.
   - **Security and privileges:** CREATE_PRIV permits catalog creation; 
ALTER_PRIV is required to change its properties. FE and BE make outbound HTTP 
requests to the configured S3 endpoint, an SSRF surface. The provider declares 
S3-compatible endpoint aliases for the existing CREATE/ALTER gate when 
`security_checker_class_name` is configured. REST credential keys are declared 
for masking even though REST reads are deferred. Build-time JNI replacement 
downloads a fixed, checksum-pinned asset.
   - **Follow-up plan:** REST/OSS and namespace-vended credentials; 
predicate/LIMIT/COUNT pushdown, fragment grouping and scalar indexes; then 
session/table caches and profile counters. Search TVFs, index operations and 
tag/branch/managed versioning follow separately under #70680.
   
   ### Testing
   
   - Local macOS arm64/JDK 17: 71 Lance tests in 13 classes, including all six 
native dataset tests with no skips. SPI: 162 tests. Relevant FE scan, MVCC, 
image replay, type conversion, plugin loading and endpoint-gate tests: 507 
tests in 43 classes. All passed with the Maven build cache disabled and 
Checkstyle enabled.
   - Two provider mutations: removing the primary S3 endpoint or the API-key 
masking declaration fails the corresponding assertion; the unmodified control 
passes.
   - `build.sh --fe` completed successfully, including Checkstyle and both 
Lance/MongoDB plugin distributions at API 32.0. Shell syntax checks passed for 
the build, FE test and JNI helper scripts.
   - `test_lance_catalog_all_types` passed in compare mode against standalone 
MinIO using the new FE and this worktree's BE `33d48fcf6da`; Lance reader 
sources match master. Three shared result blocks match the 4.2 snapshot. A 
non-root user is denied CREATE/ALTER before the corresponding grants, then 
creates, alters and reads the pinned dataset after explicit grants. The BE was 
not rebuilt at current master.
   - Historical packaging check: the pinned Linux x86_64 JNI entry was replaced 
and verified on macOS, leaving the Darwin entry unchanged. The helper is 
unchanged; a glibc 2.17 Linux host has not been tested.
   
   ### Notes for Reviewers
   
   - The BE Lance reader does not consume `history_schema_info`. Omitting it 
retains FileScannerV2's exact-name matching, which the reader tests exercise.
   - Native FE tests skip explicitly on hosts unable to load Lance or Arrow C 
Data JNI. The External pipeline enables the MinIO fixture and runs the new 
regression suite.
   
   ### Release note
   
   Add filesystem-backed Lance catalogs on master, with static S3 credentials, 
basic reads and main-chain version/time travel. Move the catalog Arrow memory 
setting to the Lance plugin's lance.conf.
   
   ### Check List (For Author)
   
   - Behavior changed: No
   - Does this need documentation: Yes (update master Lance catalog 
documentation for plugin installation, lance.conf migration and the staged 
scope; documentation is maintained separately)
   - Pre-submission status: Partially verified: the local regression uses 
standalone MinIO and a compatible BE built in this worktree; a complete 
current-master BE build, the full External environment, a cloud cluster and a 
glibc 2.17 Linux host were not verified
   
   ### AI Disclosure
   
   - Tool: Claude Code (Claude Opus 5.5); Codex
   - Generated: code, tests, this description
   - Human review: not stated
   - Prompts: The author asked to port the branch-4.2 Lance read path as a 
connector plugin, split it into four PRs, include main-chain time travel in the 
first, and resume from the handoff after its prerequisite PRs merged.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to