morningman opened a new issue, #70680:
URL: https://github.com/apache/doris/issues/70680
### Background
Doris 4.1 added `lance` catalogs: a Lance namespace (a directory warehouse
or a REST namespace server) is shown as databases and tables, FE plans the scan
from the dataset's manifest, and BE reads the data through lance-c. That code
lives in fe-core on branch-4.1 and branch-4.2. On master, external catalogs are
connector plugins loaded at runtime (`fe/fe-connector`), the BE Lance reader is
already in (#69581), and an image written by 4.1 replays Lance catalogs as
plugin-driven ones (#69575), but no `lance` connector exists, so such catalogs
cannot be read.
### Current behavior
On master, `CREATE CATALOG ... ("type" = "lance")` fails because no
connector provides the type, and a Lance catalog carried over from a 4.1 image
cannot be queried.
### Expected behavior
`lance` catalogs work on master with the same properties, names, types and
SQL as on 4.1/4.2, served by a `fe-connector-lance` plugin; fe-core gains no
Lance class.
### Proposed design
A connector plugin that follows the existing ones (ADBC, Fluss):
- `LanceConnectorProvider` (`type = lance`) binds the 4.x property map
unchanged and declares S3-compatible endpoint aliases for the CREATE/ALTER
endpoint gate and REST credentials for masking through the connector security
surface (#70338).
- Metadata goes through the Lance Java SDK (lance-core 12.0.0, a JNI
binding); the native library is loaded on first use, never at catalog
construction or replay.
- Every query pins one dataset version (`SUPPORTS_MVCC_SNAPSHOT`); `FOR
VERSION AS OF n` and `FOR TIME AS OF '...'` resolve on the main chain, as in
4.1.
- Scans are one range per fragment of the pinned version, carried in the
existing `TLanceFileDesc` / `TLanceScanParams`; no thrift change.
- A column of the types added in 4.2 (Arrow Null, Duration, JSON, bfloat16)
fences out smooth-upgrade source BEs through
`REQUIRED_CURRENT_BACKEND_SEMANTICS`.
- The per-FE Arrow memory limit moves from fe.conf to the plugin's
`lance.conf`, same key.
Alternatives: keeping Lance in fe-core was rejected because master no longer
adds catalog code there; waiting for the 4.x code to settle was rejected in
favor of porting the 4.2 shape now and catching up in the next batch.
The port lands in four PRs:
1. The plugin, filesystem catalogs, static S3 credentials, types, basic
scans, main-chain time travel, packaging.
2. REST catalogs (SSRF surface `lance.rest.uri`), OSS storage,
namespace-vended credentials.
3. Pushdown: Substrait predicates, LIMIT, COUNT(*), fragment grouping,
scalar-index segments.
4. Session and table-access caches, profile counters.
Search TVFs, index DDL and inspection, the file TVF `format=lance`, and
tag/branch/managed versioning follow separately.
### Impact
Users of Lance catalogs who upgrade from 4.1/4.2 to a release built from
master. FE only; the BE reader is already on master. Shared-nothing and cloud
modes alike.
### Reproduction or evidence
`regression-test/suites/external_table_p0/lance/` on branch-4.2;
`fe/fe-core/src/main/java/org/apache/doris/datasource/lance/` on branch-4.2
(snapshot `cabcf45c1af`).
### Scope and non-goals
In scope: the read path of `lance` catalogs listed above. Not in scope:
writes and DDL against Lance, which 4.2 is still developing.
### Acceptance criteria
The branch-4.2 read-path FE tests, rewritten against the connector SPI, and
the read-path regression suites (`test_lance_catalog_all_types`,
`test_lance_rest_catalog`, the pushdown suites, `test_lance_optimize_count`,
`test_lance_fragment_grouping`) pass on master.
### Risks and compatibility
4.1 images persist the raw property map, so the plugin must accept it
unchanged. lance-core's linux-x86-64 JNI library needs GLIBC 2.28; the build
swaps in a glibc 2.17 build, as 4.2 does. Reading a schema also loads
arrow-c-data's JNI shim, which needs CXXABI_1.3.9 (4.2 has the same
requirement).
### Related
#69581 (BE Lance reader on master), #69575 (replaying 4.1 Lance catalogs as
plugin-driven ones), #70338 (connector security surface for secret keys), and
the source PRs on branch-4.1/4.2: #65730, #66581, #66597, #66805, #67157,
#68078, #68305, #67862, #68052.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]