morningman opened a new issue, #70680:
URL: https://github.com/apache/doris/issues/70680

   ### Background
   
   Doris 4.1 added `lance` catalogs: a Lance namespace (a directory warehouse 
or a REST namespace server) is shown as databases and tables, FE plans the scan 
from the dataset's manifest, and BE reads the data through lance-c. That code 
lives in fe-core on branch-4.1 and branch-4.2. On master, external catalogs are 
connector plugins loaded at runtime (`fe/fe-connector`), the BE Lance reader is 
already in (#69581), and an image written by 4.1 replays Lance catalogs as 
plugin-driven ones (#69575), but no `lance` connector exists, so such catalogs 
cannot be read.
   
   ### Current behavior
   
   On master, `CREATE CATALOG ... ("type" = "lance")` fails because no 
connector provides the type, and a Lance catalog carried over from a 4.1 image 
cannot be queried.
   
   ### Expected behavior
   
   `lance` catalogs work on master with the same properties, names, types and 
SQL as on 4.1/4.2, served by a `fe-connector-lance` plugin; fe-core gains no 
Lance class.
   
   ### Proposed design
   
   A connector plugin that follows the existing ones (ADBC, Fluss):
   - `LanceConnectorProvider` (`type = lance`) binds the 4.x property map 
unchanged and declares S3-compatible endpoint aliases for the CREATE/ALTER 
endpoint gate and REST credentials for masking through the connector security 
surface (#70338).
   - Metadata goes through the Lance Java SDK (lance-core 12.0.0, a JNI 
binding); the native library is loaded on first use, never at catalog 
construction or replay.
   - Every query pins one dataset version (`SUPPORTS_MVCC_SNAPSHOT`); `FOR 
VERSION AS OF n` and `FOR TIME AS OF '...'` resolve on the main chain, as in 
4.1.
   - Scans are one range per fragment of the pinned version, carried in the 
existing `TLanceFileDesc` / `TLanceScanParams`; no thrift change.
   - A column of the types added in 4.2 (Arrow Null, Duration, JSON, bfloat16) 
fences out smooth-upgrade source BEs through 
`REQUIRED_CURRENT_BACKEND_SEMANTICS`.
   - The per-FE Arrow memory limit moves from fe.conf to the plugin's 
`lance.conf`, same key.
   
   Alternatives: keeping Lance in fe-core was rejected because master no longer 
adds catalog code there; waiting for the 4.x code to settle was rejected in 
favor of porting the 4.2 shape now and catching up in the next batch.
   
   The port lands in four PRs:
   1. The plugin, filesystem catalogs, static S3 credentials, types, basic 
scans, main-chain time travel, packaging.
   2. REST catalogs (SSRF surface `lance.rest.uri`), OSS storage, 
namespace-vended credentials.
   3. Pushdown: Substrait predicates, LIMIT, COUNT(*), fragment grouping, 
scalar-index segments.
   4. Session and table-access caches, profile counters.
   
   Search TVFs, index DDL and inspection, the file TVF `format=lance`, and 
tag/branch/managed versioning follow separately.
   
   ### Impact
   
   Users of Lance catalogs who upgrade from 4.1/4.2 to a release built from 
master. FE only; the BE reader is already on master. Shared-nothing and cloud 
modes alike.
   
   ### Reproduction or evidence
   
   `regression-test/suites/external_table_p0/lance/` on branch-4.2; 
`fe/fe-core/src/main/java/org/apache/doris/datasource/lance/` on branch-4.2 
(snapshot `cabcf45c1af`).
   
   ### Scope and non-goals
   
   In scope: the read path of `lance` catalogs listed above. Not in scope: 
writes and DDL against Lance, which 4.2 is still developing.
   
   ### Acceptance criteria
   
   The branch-4.2 read-path FE tests, rewritten against the connector SPI, and 
the read-path regression suites (`test_lance_catalog_all_types`, 
`test_lance_rest_catalog`, the pushdown suites, `test_lance_optimize_count`, 
`test_lance_fragment_grouping`) pass on master.
   
   ### Risks and compatibility
   
   4.1 images persist the raw property map, so the plugin must accept it 
unchanged. lance-core's linux-x86-64 JNI library needs GLIBC 2.28; the build 
swaps in a glibc 2.17 build, as 4.2 does. Reading a schema also loads 
arrow-c-data's JNI shim, which needs CXXABI_1.3.9 (4.2 has the same 
requirement).
   
   ### Related
   
   #69581 (BE Lance reader on master), #69575 (replaying 4.1 Lance catalogs as 
plugin-driven ones), #70338 (connector security surface for secret keys), and 
the source PRs on branch-4.1/4.2: #65730, #66581, #66597, #66805, #67157, 
#68078, #68305, #67862, #68052.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to