zy-kkk commented on PR #68453: URL: https://github.com/apache/doris/pull/68453#issuecomment-5882509470
c57da5d5c5d changes how tables with `managed_versioning = true` are read. Up to 8e0eda26dae the FE opened such a table through the SDK's namespace client, while the BE opened it by URI and version number through lance-c. Most threads since round 9 were about keeping those two readers aligned: the native store identity, vended credentials, the SDK's second `DescribeTable`, the manifest it opened, a relocation during the open. Each fix left a narrower window. Namespace mode rested on one premise, that a reader has to finalize a staged manifest itself. That holds for Lance's default external manifest store, but its namespace store overrides it: Lance's own namespaces finalize the manifest inside `CreateTableVersion` and list versions from the canonical manifests (`lance-namespace-impls` `dir.rs`, `create_table_version` and `list_table_versions_from_storage`), and the format spec's External Manifest Store section makes the canonical path the commit point. So the FE now reads a managed table the way the BE does, by the location and options of its own `DescribeTable` and a version number: - The namespace still decides which versions exist and which is the newest. Doris asks it directly (`ListTableVersions`, `DescribeTableVersion`, with the branch when there is one), and a managed table's access is never cached, so every read describes the table once and hands exactly that location and those options to the BE. - The manifest the namespace records for the selected version must be the version's canonical path on its chain, or a staged manifest beside it. Anything else fails the read with both paths and a hint to retry; that also covers a table moved between the describe and the version lookup. - A version recorded only at a staged manifest whose canonical manifest does not exist cannot be read. The query fails and says why, with no fallback to an older version and no write on read, as Doris's Iceberg and Paimon connectors fail on a snapshot their metadata names but they cannot load. - On a managed chain, a version number the namespace no longer records between versions it records now cuts the `FOR TIME AS OF` history, like a version removed by cleanup. It used to be skipped, which returned an older version for a time the missing version covered. - The version requests run outside the catalog's namespace lock; the requests it covered before keep it. `LanceSdkNamespace`, `LanceSdkNamespaceTest`, `LanceSdkOpenedAccessTest` and `LanceManagedS3StoreTest` are removed, and so is the storage-option normalization only the SDK open needed. The earlier threads about the wrapper (store identity, credentials, OpenDAL, Azure endpoint through the SDK open, the SDK's second describe) no longer apply to the code; I've answered the two open ones below. The PR description and apache/doris-website#4172 describe the new behavior. Verification: every Lance test class in fe-core with the `@Disabled`s lifted (47 classes, 625 cases) and the `external_table_p0/lance` suites against a 4.1 FE built from this revision. `test_lance_rest_time_travel` gains three managed fixture tables: one whose newest version (on `main` and on a branch) is recorded at a staged manifest that was never finalized, one recorded at a staged manifest that was, and one recorded at a manifest Doris does not read. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
