This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 84eec48788f [feature](lance) Add lance_index_entries inspection TVF
(#66671)
84eec48788f is described below
commit 84eec48788f82dd5e66b70a5ac2b4616c669b3a5
Author: kid <[email protected]>
AuthorDate: Sun Sep 6 06:13:49 2026 +0800
[feature](lance) Add lance_index_entries inspection TVF (#66671)
### What problem does this PR solve?
Issue Number: #66497
Related PR: #66637 (merged)
Problem Summary:
This is PR2 of the Lance index lifecycle work. PR1 (#66637) has been
merged, and this branch is now rebased onto the latest `branch-4.1` with
only the five PR2/review-fix commits retained.
This PR implements delivery slice 2 of the v5.1 design ([final 4.2
contract](https://github.com/apache/doris/issues/66497#issuecomment-5301314544),
scope confirmed in [this
review](https://github.com/apache/doris/issues/66497#issuecomment-5301637401)):
minimal, read-only Directory physical-index inspection. It was rewritten
from the earlier inspection PR because the design moved the duplicate
logical TVF, rich physical diagnostics, and all REST inspection out of
the 4.2 boundary.
What this PR adds:
- `lance_index_entries("table" = "catalog.db.table")` returns one row
per physical index entry with exactly `CatalogName, DatabaseName,
TableName, IndexName, IndexUuid, DatasetVersion`. This TVF exists only
because index UUID and dataset version do not fit the established
13-column `SHOW INDEX` schema; logical name, columns, type, and
properties remain in `SHOW INDEX`.
- Rows are read from `Dataset.getIndexes()` on one latest Directory
`Dataset` snapshot through PR1's bounded FE read executor (shared finite
pool, single deadline, no claimed JNI cancellation on timeout). This
path never calls `describeIndices()`, `countRows()`, or
`getIndexStatistics()`.
- The read is all-or-error and fail-closed: the raw provider entry count
is capped at 16,384 before filtering; every raw entry is validated
before reserved system entries (`__lance_frag_reuse`, `__lance_mem_wal`)
are excluded. Blank or oversized names, missing UUIDs, non-positive
dataset versions, and duplicate UUIDs (including system/user collisions)
fail the whole read. Output is deterministically ordered by `(IndexName,
IndexUuid)` and never silently truncated.
- Table `SHOW` privilege is checked in the analyzer before any catalog
lookup or initialization, and re-checked on the FE master from the
relayed user identity before serving the BE metadata RPC.
Backtick-quoted identifiers are relayed without changing whitespace that
is part of the identifier.
- Lance REST catalogs are rejected from configuration with a fixed
unsupported error before catalog initialization or any remote namespace
request, in both the analyzer and master paths.
Explicitly not in this PR (v5.1 Section 1.3 deferrals): the logical
`lance_indexes()` TVF, server-side exact logical-index counts,
`countRows()` in inspection, row/fragment coverage, provider/consistency
state, and every REST inspection or mutation surface. `SHOW INDEX`
behavior from PR1 is unchanged; this PR is additive (22 files) apart
from validation hardening and test/golden cleanup.
### Release note
Add the read-only `lance_index_entries` table-valued function exposing
Lance physical index UUID and dataset version for Directory catalog
tables.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test
- Post-review FE focused run passed 60/60 with checkstyle enabled:
`LanceIndexMetadataLoaderTest` (32), `LanceFilesystemCatalogTest` (11),
`LanceIndexEntriesTableValuedFunctionTest` (13), and
`LanceIndexEntriesTableValuedFunctionAuthTest` (4).
- Current `meta_scanner.cpp` and `meta_scanner_test.cpp` compile under
the ASAN unit-test configuration; `MetaScannerTest.*` passes 2/2. Clang
format and cumulative diff whitespace checks pass.
- `test_lance_index_entries` and PR1's `test_lance_show_index` passed in
verify mode against a fresh local FE+BE cluster backed by a rebuilt
Docker MinIO fixture before the review-only hardening/rebase. The
six-column SQL contract and query expectations are unchanged; GitHub CI
is requested again for the rebased head.
- The full Docker regression suite (all external suites) has not been
run.
- Behavior changed:
- [ ] No.
- [x] Yes. Adds the read-only `lance_index_entries` TVF. Existing `SHOW
INDEX` behavior is unchanged.
- Does this need documentation?
- [ ] No.
- [x] Yes. The staged SQL contract and limitations are recorded here;
website documentation is tracked with the broader lifecycle work in
#66497 because this source branch does not contain the website
documentation tree.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm #66637 is merged and this PR has been rebased to contain
only PR2 commits.
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---------
Co-authored-by: u70b3 <[email protected]>
---
be/src/exec/scan/meta_scanner.cpp | 27 ++
be/src/exec/scan/meta_scanner.h | 5 +
be/test/exec/scan/meta_scanner_test.cpp | 143 +++++++++
.../doris/catalog/BuiltinTableValuedFunctions.java | 2 +
.../datasource/lance/LanceExternalCatalog.java | 38 +++
.../doris/datasource/lance/LanceExternalTable.java | 5 +
.../datasource/lance/LanceIndexMetadataLoader.java | 65 +++-
.../datasource/lance/LancePhysicalIndexEntry.java | 54 ++++
.../functions/table/LanceIndexEntries.java | 49 +++
.../LanceIndexEntriesTableValuedFunction.java | 213 +++++++++++++
.../doris/tablefunction/MetadataGenerator.java | 104 +++++++
.../tablefunction/MetadataTableValuedFunction.java | 2 +
.../doris/tablefunction/TableValuedFunctionIf.java | 2 +
.../lance/LanceFilesystemCatalogTest.java | 48 +++
.../lance/LanceIndexMetadataLoaderTest.java | 152 +++++++++
...nceIndexEntriesTableValuedFunctionAuthTest.java | 203 ++++++++++++
.../LanceIndexEntriesTableValuedFunctionTest.java | 345 +++++++++++++++++++++
gensrc/thrift/FrontendService.thrift | 1 +
gensrc/thrift/PlanNodes.thrift | 8 +
gensrc/thrift/Types.thrift | 3 +
.../lance/test_lance_index_entries.out | 14 +
.../lance/test_lance_index_entries.groovy | 121 ++++++++
22 files changed, 1603 insertions(+), 1 deletion(-)
diff --git a/be/src/exec/scan/meta_scanner.cpp
b/be/src/exec/scan/meta_scanner.cpp
index 5440922f05d..863764a87a9 100644
--- a/be/src/exec/scan/meta_scanner.cpp
+++ b/be/src/exec/scan/meta_scanner.cpp
@@ -265,6 +265,10 @@ Status MetaScanner::_fetch_metadata(const TMetaScanRange&
meta_scan_range) {
case TMetadataType::PARTITION_VALUES:
RETURN_IF_ERROR(_build_partition_values_metadata_request(meta_scan_range,
&request));
break;
+ case TMetadataType::LANCE_INDEX_ENTRIES:
+
RETURN_IF_ERROR(_build_lance_index_entries_metadata_request(meta_scan_range,
_user_identity,
+ &request));
+ break;
default:
_meta_eos = true;
return Status::OK();
@@ -507,6 +511,29 @@ Status
MetaScanner::_build_partition_values_metadata_request(
return Status::OK();
}
+Status MetaScanner::_build_lance_index_entries_metadata_request(
+ const TMetaScanRange& meta_scan_range, const TUserIdentity&
user_identity,
+ TFetchSchemaTableDataRequest* request) {
+ VLOG_CRITICAL <<
"MetaScanner::_build_lance_index_entries_metadata_request";
+ if (!meta_scan_range.__isset.lance_index_params) {
+ return Status::InternalError(
+ "Can not find TLanceIndexMetadataParams from
meta_scan_range.");
+ }
+
+ // create request
+ request->__set_cluster_name("");
+ request->__set_schema_table_name(TSchemaTableName::METADATA_TABLE);
+
+ // create TMetadataTableRequestParams
+ TMetadataTableRequestParams metadata_table_params;
+
metadata_table_params.__set_metadata_type(TMetadataType::LANCE_INDEX_ENTRIES);
+
metadata_table_params.__set_lance_index_metadata_params(meta_scan_range.lance_index_params);
+ metadata_table_params.__set_current_user_ident(user_identity);
+
+ request->__set_metada_table_params(metadata_table_params);
+ return Status::OK();
+}
+
Status MetaScanner::close(RuntimeState* state) {
VLOG_CRITICAL << "MetaScanner::close";
if (_reader) {
diff --git a/be/src/exec/scan/meta_scanner.h b/be/src/exec/scan/meta_scanner.h
index 362c85d20ac..8c41d25abfe 100644
--- a/be/src/exec/scan/meta_scanner.h
+++ b/be/src/exec/scan/meta_scanner.h
@@ -86,6 +86,11 @@ private:
TFetchSchemaTableDataRequest*
request);
Status _build_partition_values_metadata_request(const TMetaScanRange&
meta_scan_range,
TFetchSchemaTableDataRequest* request);
+ // Pure request assembly, kept static so unit tests can exercise it without
+ // constructing a full Scanner/RuntimeState graph.
+ static Status _build_lance_index_entries_metadata_request(
+ const TMetaScanRange& meta_scan_range, const TUserIdentity&
user_identity,
+ TFetchSchemaTableDataRequest* request);
bool _meta_eos;
TupleId _tuple_id;
TUserIdentity _user_identity;
diff --git a/be/test/exec/scan/meta_scanner_test.cpp
b/be/test/exec/scan/meta_scanner_test.cpp
new file mode 100644
index 00000000000..14356d1a457
--- /dev/null
+++ b/be/test/exec/scan/meta_scanner_test.cpp
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/scan/meta_scanner.h"
+
+#include <gen_cpp/FrontendService_types.h>
+#include <gen_cpp/PlanNodes_types.h>
+#include <gen_cpp/Types_types.h>
+#include <gtest/gtest.h>
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "common/object_pool.h"
+#include "common/status.h"
+#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_number.h"
+#include "exec/operator/mock_scan_operator.h"
+#include "runtime/cluster_info.h"
+#include "runtime/descriptors.h"
+#include "runtime/exec_env.h"
+#include "runtime/runtime_profile.h"
+#include "testutil/mock/mock_descriptors.h"
+#include "testutil/mock/mock_runtime_state.h"
+
+namespace doris {
+
+TEST(MetaScannerTest, BuildLanceIndexEntriesMetadataRequest) {
+ TLanceIndexMetadataParams lance_params;
+ lance_params.__set_catalog("lance_ctl");
+ lance_params.__set_database("db1");
+ lance_params.__set_table("tbl1");
+
+ TMetaScanRange meta_scan_range;
+ meta_scan_range.__set_metadata_type(TMetadataType::LANCE_INDEX_ENTRIES);
+ meta_scan_range.__set_lance_index_params(lance_params);
+
+ TUserIdentity user_identity;
+ user_identity.__set_username("lance_user");
+ user_identity.__set_host("%");
+
+ TFetchSchemaTableDataRequest request;
+ Status status = MetaScanner::_build_lance_index_entries_metadata_request(
+ meta_scan_range, user_identity, &request);
+ EXPECT_TRUE(status.ok()) << status.to_string();
+
+ EXPECT_EQ(request.cluster_name, "");
+ EXPECT_EQ(request.schema_table_name, TSchemaTableName::METADATA_TABLE);
+ ASSERT_TRUE(request.__isset.metada_table_params);
+ const TMetadataTableRequestParams& table_params =
request.metada_table_params;
+ EXPECT_EQ(table_params.metadata_type, TMetadataType::LANCE_INDEX_ENTRIES);
+ ASSERT_TRUE(table_params.__isset.lance_index_metadata_params);
+ EXPECT_EQ(table_params.lance_index_metadata_params.catalog, "lance_ctl");
+ EXPECT_EQ(table_params.lance_index_metadata_params.database, "db1");
+ EXPECT_EQ(table_params.lance_index_metadata_params.table, "tbl1");
+ ASSERT_TRUE(table_params.__isset.current_user_ident);
+ EXPECT_EQ(table_params.current_user_ident.username, "lance_user");
+ EXPECT_EQ(table_params.current_user_ident.host, "%");
+}
+
+TEST(MetaScannerTest, BuildLanceIndexEntriesMetadataRequestMissingParams) {
+ TMetaScanRange meta_scan_range;
+ meta_scan_range.__set_metadata_type(TMetadataType::LANCE_INDEX_ENTRIES);
+
+ TFetchSchemaTableDataRequest request;
+ Status status = MetaScanner::_build_lance_index_entries_metadata_request(
+ meta_scan_range, TUserIdentity(), &request);
+ EXPECT_FALSE(status.ok());
+ EXPECT_NE(status.to_string().find("TLanceIndexMetadataParams"),
std::string::npos)
+ << status.to_string();
+}
+
+// Exercises the TMetadataType::LANCE_INDEX_ENTRIES dispatch inside
_fetch_metadata,
+// which the static-assembler tests above cannot reach. The request is
assembled
+// successfully, then the FE-master RPC fails fast because the UT has no master
+// address configured; the dispatch lines still execute before that failure.
+// Private-member access relies on the build-wide -fno-access-control flag used
+// for doris_be_test, the same mechanism as scanner_late_arrival_rf_test.cpp.
+TEST(MetaScannerTest, FetchMetadataLanceIndexEntriesDispatch) {
+ ObjectPool pool;
+ auto data_type = std::make_shared<DataTypeInt32>();
+ auto row_descriptor = MockRowDescriptor({data_type}, &pool);
+
+ MockRuntimeState state;
+ auto op = std::make_shared<MockScanOperatorX>();
+ op->_row_descriptor = row_descriptor;
+ op->_output_row_descriptor =
+ std::make_unique<MockRowDescriptor>(std::vector<DataTypePtr>
{data_type}, &pool);
+ op->_output_tuple_desc =
op->_output_row_descriptor->tuple_descriptors()[0];
+ auto local_state = std::make_shared<MockScanLocalState>(&state, op.get());
+
+ RuntimeProfile profile("meta_scanner");
+ // _scan_range is a reference member bound into this params object, so the
+ // params must outlive the scanner.
+ TScanRangeParams scan_range_params;
+ TUserIdentity user_identity;
+ user_identity.__set_username("lance_user");
+ user_identity.__set_host("%");
+ MetaScanner scanner(&state, local_state.get(), /*tuple_id=*/0,
scan_range_params,
+ /*limit=*/-1, &profile, user_identity);
+
+ // Zero slots: the filter-columns loop after the dispatch is a no-op.
+ TupleDescriptor tuple_desc;
+ scanner._tuple_desc = &tuple_desc;
+
+ // A default ClusterInfo carries an empty master address, so
+ // ThriftRpcHelper::rpc returns SERVICE_UNAVAILABLE immediately instead of
+ // attempting any network IO. Restore the previous value on the way out.
+ ClusterInfo cluster_info;
+ ExecEnv* exec_env = ExecEnv::GetInstance();
+ ClusterInfo* previous_cluster_info = exec_env->cluster_info();
+ exec_env->set_cluster_info(&cluster_info);
+
+ TLanceIndexMetadataParams lance_params;
+ lance_params.__set_catalog("lance_ctl");
+ lance_params.__set_database("db1");
+ lance_params.__set_table("tbl1");
+ TMetaScanRange meta_scan_range;
+ meta_scan_range.__set_metadata_type(TMetadataType::LANCE_INDEX_ENTRIES);
+ meta_scan_range.__set_lance_index_params(lance_params);
+
+ Status status = scanner._fetch_metadata(meta_scan_range);
+ EXPECT_FALSE(status.ok()) << status.to_string();
+
+ exec_env->set_cluster_info(previous_cluster_info);
+}
+
+} // namespace doris
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinTableValuedFunctions.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinTableValuedFunctions.java
index 06b7242d161..cd17ac9cb38 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinTableValuedFunctions.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinTableValuedFunctions.java
@@ -29,6 +29,7 @@ import
org.apache.doris.nereids.trees.expressions.functions.table.Http;
import org.apache.doris.nereids.trees.expressions.functions.table.HttpStream;
import org.apache.doris.nereids.trees.expressions.functions.table.HudiMeta;
import org.apache.doris.nereids.trees.expressions.functions.table.Jobs;
+import
org.apache.doris.nereids.trees.expressions.functions.table.LanceIndexEntries;
import org.apache.doris.nereids.trees.expressions.functions.table.Local;
import org.apache.doris.nereids.trees.expressions.functions.table.MvInfos;
import org.apache.doris.nereids.trees.expressions.functions.table.Numbers;
@@ -67,6 +68,7 @@ public class BuiltinTableValuedFunctions implements
FunctionHelper {
tableValued(MvInfos.class, "mv_infos"),
tableValued(Partitions.class, "partitions"),
tableValued(Jobs.class, "jobs"),
+ tableValued(LanceIndexEntries.class, "lance_index_entries"),
tableValued(Tasks.class, "tasks"),
tableValued(Query.class, "query"),
tableValued(PartitionValues.class, "partition_values"),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java
index c2fa3d1a6ee..cef2b064b98 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java
@@ -370,6 +370,44 @@ public class LanceExternalCatalog extends ExternalCatalog {
}
}
+ public List<LancePhysicalIndexEntry> loadTableIndexEntries(
+ String dbName, String tableName) throws AnalysisException {
+ if (isRestCatalogConfigured()) {
+ throw new AnalysisException(
+ "Lance index inspection is not supported for Lance REST
catalogs");
+ }
+ try {
+ makeSureInitialized();
+ } catch (Exception e) {
+ throw indexMetadataLoadFailure(dbName, tableName, e, null,
namespaceStorageOptions);
+ }
+
+ ResolvedTableAccess tableAccess = null;
+ try {
+ // Keep Directory namespace resolution on the caller while it owns
the catalog's
+ // shared namespace and allocator. Moving that shared owner into a
timed task would
+ // let catalog close release it after the caller returns but
before the task ends.
+ // The deadline below covers the Dataset/JNI index metadata read
itself.
+ tableAccess = resolveTableAccess(dbName, tableName);
+ String datasetUri = tableAccess.datasetUri;
+ Map<String, String> storageOptions = tableAccess.storageOptions;
+ return LanceMetadataReadExecutor.execute(() -> {
+ // The caller may return on deadline while JNI is still
running. A task-owned
+ // allocator prevents catalog close from releasing native
resources prematurely.
+ try (BufferAllocator readAllocator = new
RootAllocator(ALLOCATOR_LIMIT)) {
+ return LanceIndexMetadataLoader.loadPhysicalEntries(
+ datasetUri, storageOptions, readAllocator);
+ }
+ });
+ } catch (Exception e) {
+ String datasetUri = tableAccess == null ? null :
tableAccess.datasetUri;
+ Map<String, String> runtimeStorageOptions = tableAccess == null
+ ? namespaceStorageOptions : tableAccess.storageOptions;
+ throw indexMetadataLoadFailure(
+ dbName, tableName, e, datasetUri, runtimeStorageOptions);
+ }
+ }
+
@VisibleForTesting
RuntimeException indexMetadataLoadFailure(String dbName, String tableName,
Throwable throwable, String datasetUri, Map<String, String>
runtimeStorageOptions) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalTable.java
index acba68b63fc..668959f8295 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalTable.java
@@ -76,6 +76,11 @@ public class LanceExternalTable extends ExternalTable
implements MvccTable {
db.getRemoteName(), remoteName);
}
+ public List<LancePhysicalIndexEntry> loadIndexEntries() throws
AnalysisException {
+ return ((LanceExternalCatalog) catalog).loadTableIndexEntries(
+ db.getRemoteName(), remoteName);
+ }
+
private LanceTableMetadata loadMetadata(Optional<TableSnapshot>
tableSnapshot) {
return ((LanceExternalCatalog) catalog).loadTableMetadata(
db.getRemoteName(), remoteName, tableSnapshot);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexMetadataLoader.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexMetadataLoader.java
index 431b7826bec..2eed9710fce 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexMetadataLoader.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexMetadataLoader.java
@@ -27,6 +27,7 @@ import com.google.gson.stream.JsonToken;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.commons.lang3.StringUtils;
import org.lance.Dataset;
+import org.lance.index.Index;
import org.lance.index.IndexCriteria;
import org.lance.index.IndexDescription;
import org.lance.schema.LanceField;
@@ -45,8 +46,9 @@ import java.util.Map;
import java.util.OptionalLong;
import java.util.Set;
import java.util.TreeMap;
+import java.util.UUID;
-/** Loads and normalizes logical index metadata from one latest Lance dataset
snapshot. */
+/** Loads and normalizes logical and physical index metadata from one latest
Lance dataset snapshot. */
public final class LanceIndexMetadataLoader {
private static final int MAX_LOGICAL_INDEXES = 256;
private static final int MAX_COLUMNS_PER_INDEX = 64;
@@ -88,6 +90,67 @@ public final class LanceIndexMetadataLoader {
}
}
+ /**
+ * Loads physical index entries from one latest dataset snapshot. This
path only calls
+ * {@link Dataset#getIndexes()}; it never describes indexes or reads row
statistics.
+ */
+ public static List<LancePhysicalIndexEntry> loadPhysicalEntries(String
datasetUri,
+ Map<String, String> javaStorageOptions, BufferAllocator allocator)
throws Exception {
+ try (Dataset dataset =
Dataset.open().allocator(allocator).uri(datasetUri)
+ .readOptions(LanceReadOptions.build(javaStorageOptions,
OptionalLong.empty())).build()) {
+ return collectPhysicalEntries(dataset);
+ }
+ }
+
+ /** Converts the snapshot's raw index list into sorted immutable physical
entries. */
+ static List<LancePhysicalIndexEntry> collectPhysicalEntries(Dataset
dataset) {
+ List<Index> indexes = dataset.getIndexes();
+ if (indexes == null) {
+ throw new IllegalArgumentException("Lance physical index entries
must not be null");
+ }
+ // Bound the raw provider response before any filtering so a flood of
system
+ // entries still fails closed instead of consuming unbounded memory.
+ if (indexes.size() > MAX_PHYSICAL_INDEX_ENTRIES) {
+ throw new IllegalArgumentException(
+ "Lance physical index entry count exceeds limit "
+ + MAX_PHYSICAL_INDEX_ENTRIES);
+ }
+ List<LancePhysicalIndexEntry> entries = new
ArrayList<>(indexes.size());
+ Set<String> seenUuids = new HashSet<>();
+ for (Index index : indexes) {
+ if (index == null) {
+ throw new IllegalArgumentException(
+ "Lance physical index entry must not be null");
+ }
+ String name = requireExternalString(
+ index.name(), "Lance physical index entry name");
+ UUID uuid = index.uuid();
+ if (uuid == null) {
+ throw new IllegalArgumentException(
+ "Lance physical index entry uuid must not be null");
+ }
+ long datasetVersion = index.datasetVersion();
+ if (datasetVersion <= 0) {
+ throw new IllegalArgumentException(
+ "Lance physical index entry dataset version must be
positive");
+ }
+ String uuidString = uuid.toString();
+ if (!seenUuids.add(uuidString)) {
+ throw new IllegalArgumentException(
+ "Duplicate Lance physical index entry uuid '" +
uuidString + "'");
+ }
+ // Validate every raw entry before filtering so malformed system
metadata or a UUID
+ // collision between a system and user entry cannot be hidden from
the all-or-error read.
+ if (SYSTEM_INDEX_NAMES.contains(name)) {
+ continue;
+ }
+ entries.add(new LancePhysicalIndexEntry(name, uuidString,
datasetVersion));
+ }
+ entries.sort(Comparator.comparing(LancePhysicalIndexEntry::getName)
+ .thenComparing(LancePhysicalIndexEntry::getUuid));
+ return Collections.unmodifiableList(entries);
+ }
+
/**
* Describes only user-created indexes. The Lance JNI bulk describe path
also tries to
* materialize details for internal indexes, whose details are not
supported by the SDK.
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LancePhysicalIndexEntry.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LancePhysicalIndexEntry.java
new file mode 100644
index 00000000000..52a96014e23
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LancePhysicalIndexEntry.java
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.lance;
+
+import org.apache.commons.lang3.StringUtils;
+
+/** Immutable physical index entry read from one Lance dataset snapshot. */
+public final class LancePhysicalIndexEntry {
+ private final String name;
+ private final String uuid;
+ private final long datasetVersion;
+
+ public LancePhysicalIndexEntry(String name, String uuid, long
datasetVersion) {
+ if (StringUtils.isBlank(name)) {
+ throw new IllegalArgumentException("name must not be null or
blank");
+ }
+ if (StringUtils.isBlank(uuid)) {
+ throw new IllegalArgumentException("uuid must not be null or
blank");
+ }
+ if (datasetVersion <= 0) {
+ throw new IllegalArgumentException("dataset version must be
positive");
+ }
+ this.name = name;
+ this.uuid = uuid;
+ this.datasetVersion = datasetVersion;
+ }
+
+ public String getName() {
+ return name;
+ }
+
+ public String getUuid() {
+ return uuid;
+ }
+
+ public long getDatasetVersion() {
+ return datasetVersion;
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/LanceIndexEntries.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/LanceIndexEntries.java
new file mode 100644
index 00000000000..d73661c160c
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/LanceIndexEntries.java
@@ -0,0 +1,49 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.table;
+
+import org.apache.doris.catalog.FunctionSignature;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.trees.expressions.Properties;
+import org.apache.doris.nereids.types.coercion.AnyDataType;
+import org.apache.doris.tablefunction.LanceIndexEntriesTableValuedFunction;
+import org.apache.doris.tablefunction.TableValuedFunctionIf;
+
+import java.util.Map;
+
+/** lance_index_entries */
+public class LanceIndexEntries extends TableValuedFunction {
+ public LanceIndexEntries(Properties properties) {
+ super(LanceIndexEntriesTableValuedFunction.NAME, properties);
+ }
+
+ @Override
+ public FunctionSignature customSignature() {
+ return FunctionSignature.of(AnyDataType.INSTANCE_WITHOUT_INDEX,
getArgumentsTypes());
+ }
+
+ @Override
+ protected TableValuedFunctionIf toCatalogFunction() {
+ try {
+ Map<String, String> arguments = getTVFProperties().getMap();
+ return new LanceIndexEntriesTableValuedFunction(arguments);
+ } catch (Throwable t) {
+ throw new AnalysisException("Can not build lance_index_entries():
" + t.getMessage(), t);
+ }
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/LanceIndexEntriesTableValuedFunction.java
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/LanceIndexEntriesTableValuedFunction.java
new file mode 100644
index 00000000000..fdf6a0fff25
--- /dev/null
+++
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/LanceIndexEntriesTableValuedFunction.java
@@ -0,0 +1,213 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.tablefunction;
+
+import org.apache.doris.analysis.TableName;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.common.ErrorCode;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.datasource.lance.LanceExternalCatalog;
+import org.apache.doris.datasource.lance.LanceExternalTable;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.analyzer.UnboundSlot;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.exceptions.ParseException;
+import org.apache.doris.nereids.parser.NereidsParser;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.thrift.TLanceIndexMetadataParams;
+import org.apache.doris.thrift.TMetaScanRange;
+import org.apache.doris.thrift.TMetadataType;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+
+/**
+ * Read-only inspection TVF listing the physical index entries of one Lance
Directory catalog
+ * table. Rows are produced on the FE master through the bounded catalog read
path; this class
+ * only validates arguments, authorization and the wire contract.
+ */
+public class LanceIndexEntriesTableValuedFunction extends
MetadataTableValuedFunction {
+ public static final String NAME = "lance_index_entries";
+ static final String REST_CATALOG_REJECT_MESSAGE =
+ "lance_index_entries is not supported for Lance REST catalogs";
+ private static final String TABLE = "table";
+ private static final Set<String> PROPERTIES = ImmutableSet.of(TABLE);
+ private static final String FULLY_QUALIFIED_TABLE_NAME_ERROR =
+ "'table' must be a fully qualified catalog.database.table name";
+ private static final ImmutableList<Column> SCHEMA = ImmutableList.of(
+ new Column("CatalogName", ScalarType.createStringType()),
+ new Column("DatabaseName", ScalarType.createStringType()),
+ new Column("TableName", ScalarType.createStringType()),
+ new Column("IndexName", ScalarType.createStringType()),
+ new Column("IndexUuid", ScalarType.createStringType()),
+ new Column("DatasetVersion", PrimitiveType.BIGINT, false));
+ private static final ImmutableMap<String, Integer> COLUMN_TO_INDEX =
buildColumnIndex();
+
+ private final TableName sourceTableName;
+
+ public LanceIndexEntriesTableValuedFunction(Map<String, String>
properties) throws AnalysisException {
+ sourceTableName =
parseTableName(normalizeProperties(properties).get(TABLE));
+
+ // This check intentionally precedes catalog lookup/initialization and
every provider call.
+ checkShowPrivilege(ConnectContext.get(), sourceTableName);
+ resolveLanceTable(sourceTableName);
+ }
+
+ public final String getCatalogName() {
+ return sourceTableName.getCtl();
+ }
+
+ public final String getDatabaseName() {
+ return sourceTableName.getDb();
+ }
+
+ public final String getSourceTableName() {
+ return sourceTableName.getTbl();
+ }
+
+ @Override
+ public final TMetadataType getMetadataType() {
+ return TMetadataType.LANCE_INDEX_ENTRIES;
+ }
+
+ @Override
+ public final TMetaScanRange getMetaScanRange(List<String> requiredFields) {
+ TLanceIndexMetadataParams params = new TLanceIndexMetadataParams()
+ .setCatalog(getCatalogName())
+ .setDatabase(getDatabaseName())
+ .setTable(getSourceTableName());
+ return new TMetaScanRange()
+ .setMetadataType(TMetadataType.LANCE_INDEX_ENTRIES)
+ .setLanceIndexParams(params);
+ }
+
+ public static Integer getColumnIndexFromColumnName(String columnName) {
+ return COLUMN_TO_INDEX.get(columnName.toLowerCase(Locale.ROOT));
+ }
+
+ static List<Column> getSchemaForTest() {
+ return SCHEMA;
+ }
+
+ @Override
+ public String getTableName() {
+ return "LanceIndexEntriesTableValuedFunction";
+ }
+
+ @Override
+ public List<Column> getTableColumns() {
+ return SCHEMA;
+ }
+
+ @VisibleForTesting
+ static TableName parseTableName(String value) throws AnalysisException {
+ Expression expression;
+ try {
+ expression = new NereidsParser().parseExpression(value);
+ } catch (ParseException e) {
+ throw new AnalysisException(FULLY_QUALIFIED_TABLE_NAME_ERROR, e);
+ }
+ if (!(expression instanceof UnboundSlot)) {
+ throw new AnalysisException(FULLY_QUALIFIED_TABLE_NAME_ERROR);
+ }
+ List<String> names = ((UnboundSlot) expression).getNameParts();
+ if (names.size() != 3) {
+ throw new AnalysisException(FULLY_QUALIFIED_TABLE_NAME_ERROR);
+ }
+ return new TableName(names.get(0), names.get(1), names.get(2));
+ }
+
+ @VisibleForTesting
+ static Map<String, String> normalizeProperties(Map<String, String>
properties)
+ throws AnalysisException {
+ Map<String, String> normalized = new
TreeMap<>(String.CASE_INSENSITIVE_ORDER);
+ for (Map.Entry<String, String> entry : properties.entrySet()) {
+ String key = entry.getKey().toLowerCase(Locale.ROOT);
+ if (!PROPERTIES.contains(key)) {
+ throw new AnalysisException("'" + entry.getKey()
+ + "' is an invalid property for " + NAME);
+ }
+ if (normalized.containsKey(key)) {
+ throw new AnalysisException("Duplicate " + NAME + " property
'" + key + "'");
+ }
+ normalized.put(key, entry.getValue());
+ }
+ String table = normalized.get(TABLE);
+ if (table == null || table.trim().isEmpty()) {
+ throw new AnalysisException("Missing required " + NAME + "
property '" + TABLE + "'");
+ }
+ normalized.put(TABLE, table.trim());
+ return normalized;
+ }
+
+ static LanceExternalTable resolveLanceTable(TableName tableName) throws
AnalysisException {
+ CatalogIf<?> catalog =
Env.getCurrentEnv().getCatalogMgr().getCatalog(tableName.getCtl());
+ if (!(catalog instanceof LanceExternalCatalog)) {
+ throw new AnalysisException("Catalog '" + tableName.getCtl() + "'
is not a Lance catalog");
+ }
+ // REST catalogs are rejected from configuration only, before any
database or table
+ // resolution that could trigger remote namespace requests or catalog
initialization.
+ if (((LanceExternalCatalog) catalog).isRestCatalogConfigured()) {
+ throw new AnalysisException(REST_CATALOG_REJECT_MESSAGE);
+ }
+ TableIf table;
+ try {
+ table = catalog.getDbOrAnalysisException(tableName.getDb())
+ .getTableOrAnalysisException(tableName.getTbl());
+ } catch (org.apache.doris.common.AnalysisException e) {
+ throw new AnalysisException(e.getMessage(), e);
+ }
+ if (!(table instanceof LanceExternalTable)) {
+ throw new AnalysisException("Table '" + tableName + "' is not a
Lance table");
+ }
+ return (LanceExternalTable) table;
+ }
+
+ private static void checkShowPrivilege(ConnectContext context, TableName
tableName)
+ throws AnalysisException {
+ if (context == null || !Env.getCurrentEnv().getAccessManager()
+ .checkTblPriv(context, tableName, PrivPredicate.SHOW)) {
+ String user = context == null ? "unknown" :
context.getQualifiedUser();
+ String remoteIp = context == null ? "unknown" :
context.getRemoteIP();
+ throw new
AnalysisException(ErrorCode.ERR_TABLEACCESS_DENIED_ERROR.formatErrorMsg(
+ "SHOW", user, remoteIp,
+ tableName.getDb() + ": " + tableName.getTbl()));
+ }
+ }
+
+ private static ImmutableMap<String, Integer> buildColumnIndex() {
+ ImmutableMap.Builder<String, Integer> builder = ImmutableMap.builder();
+ for (int i = 0; i < SCHEMA.size(); i++) {
+ builder.put(SCHEMA.get(i).getName().toLowerCase(Locale.ROOT), i);
+ }
+ return builder.build();
+ }
+}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
index bf51fe27616..619f8748962 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java
@@ -66,6 +66,9 @@ import org.apache.doris.datasource.TablePartitionValues;
import org.apache.doris.datasource.hive.HMSExternalCatalog;
import org.apache.doris.datasource.hive.HMSExternalTable;
import org.apache.doris.datasource.hive.HiveExternalMetaCache;
+import org.apache.doris.datasource.lance.LanceExternalCatalog;
+import org.apache.doris.datasource.lance.LanceExternalTable;
+import org.apache.doris.datasource.lance.LancePhysicalIndexEntry;
import org.apache.doris.datasource.maxcompute.MaxComputeExternalCatalog;
import org.apache.doris.datasource.metacache.MetaCacheEntryStats;
import org.apache.doris.datasource.mvcc.MvccUtil;
@@ -102,6 +105,7 @@ import org.apache.doris.thrift.TFrontendsMetadataParams;
import org.apache.doris.thrift.THudiMetadataParams;
import org.apache.doris.thrift.THudiQueryType;
import org.apache.doris.thrift.TJobsMetadataParams;
+import org.apache.doris.thrift.TLanceIndexMetadataParams;
import org.apache.doris.thrift.TMaterializedViewsMetadataParams;
import org.apache.doris.thrift.TMetadataTableRequestParams;
import org.apache.doris.thrift.TMetadataType;
@@ -131,6 +135,7 @@ import org.apache.logging.log4j.Logger;
import org.apache.thrift.TException;
import org.jetbrains.annotations.NotNull;
+import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
@@ -170,6 +175,10 @@ public class MetadataGenerator {
private static final ImmutableMap<String, Integer>
ROLE_MAPPINGS_COLUMN_TO_INDEX;
+ // Bound for the relayed Lance table identity fields; matches the loader's
external
+ // string limit so an oversized identity fails before any catalog
resolution.
+ private static final int MAX_LANCE_IDENTIFIER_BYTES = 1024;
+
static {
ImmutableMap.Builder<String, Integer> activeQueriesbuilder = new
ImmutableMap.Builder();
List<Column> activeQueriesColList =
SchemaTable.TABLE_MAP.get("active_queries").getFullSchema();
@@ -308,6 +317,9 @@ public class MetadataGenerator {
case PARTITION_VALUES:
result = partitionValuesMetadataResult(params);
break;
+ case LANCE_INDEX_ENTRIES:
+ result = lanceIndexEntriesMetadataResult(params);
+ break;
default:
return errorResult("Metadata table params is not set.");
}
@@ -401,6 +413,98 @@ public class MetadataGenerator {
return result;
}
+ static TFetchSchemaTableDataResult lanceIndexEntriesMetadataResult(
+ TMetadataTableRequestParams params) {
+ if (!params.isSetLanceIndexMetadataParams()) {
+ return errorResult("Lance index metadata params is not set.");
+ }
+ if (!params.isSetCurrentUserIdent()) {
+ return errorResult("Current user identity is not set for Lance
index metadata.");
+ }
+ TLanceIndexMetadataParams lanceParams =
params.getLanceIndexMetadataParams();
+ if (!lanceParams.isSetCatalog() || !lanceParams.isSetDatabase() ||
!lanceParams.isSetTable()
+ || !isBoundedLanceIdentifier(lanceParams.getCatalog())
+ || !isBoundedLanceIdentifier(lanceParams.getDatabase())
+ || !isBoundedLanceIdentifier(lanceParams.getTable())) {
+ return errorResult("Invalid Lance index metadata table identity.");
+ }
+
+ // These names were already parsed by the analyzer. Preserve their
exact spelling because
+ // whitespace inside a backtick-quoted identifier is part of the
identifier.
+ String catalogName = lanceParams.getCatalog();
+ String databaseName = lanceParams.getDatabase();
+ String tableName = lanceParams.getTable();
+ UserIdentity userIdentity =
UserIdentity.fromThrift(params.getCurrentUserIdent());
+
+ // The master repeats authorization before resolving or initializing
an external
+ // catalog: this RPC arrives through the BE relay and carries no
privilege context.
+ if (!Env.getCurrentEnv().getAccessManager().checkTblPriv(userIdentity,
+ catalogName, databaseName, tableName, PrivPredicate.SHOW)) {
+ return
errorResult(ErrorCode.ERR_TABLEACCESS_DENIED_ERROR.formatErrorMsg(
+ "SHOW", userIdentity.getQualifiedUser(),
userIdentity.getHost(),
+ databaseName + ": " + tableName));
+ }
+
+ try {
+ CatalogIf<?> catalog =
Env.getCurrentEnv().getCatalogMgr().getCatalog(catalogName);
+ if (!(catalog instanceof LanceExternalCatalog)) {
+ return errorResult("Catalog '" + catalogName + "' is not a
Lance catalog.");
+ }
+ // REST catalogs are rejected from configuration only, before any
catalog
+ // initialization or remote namespace request.
+ if (((LanceExternalCatalog) catalog).isRestCatalogConfigured()) {
+ return
errorResult(LanceIndexEntriesTableValuedFunction.REST_CATALOG_REJECT_MESSAGE);
+ }
+ DatabaseIf<?> database =
catalog.getDbOrAnalysisException(databaseName);
+ TableIf table = database.getTableOrAnalysisException(tableName);
+ if (!(table instanceof LanceExternalTable)) {
+ return errorResult("Table '" + tableName + "' is not a Lance
table.");
+ }
+
+ List<LancePhysicalIndexEntry> entries = ((LanceExternalTable)
table).loadIndexEntries();
+ String resolvedCatalogName = catalog.getName();
+ String resolvedDatabaseName = database.getFullName();
+ String resolvedTableName = table.getName();
+ List<TRow> rows = new ArrayList<>(entries.size());
+ for (LancePhysicalIndexEntry entry : entries) {
+ TRow row = new TRow();
+ row.addToColumnValue(new
TCell().setStringVal(resolvedCatalogName));
+ row.addToColumnValue(new
TCell().setStringVal(resolvedDatabaseName));
+ row.addToColumnValue(new
TCell().setStringVal(resolvedTableName));
+ row.addToColumnValue(new
TCell().setStringVal(entry.getName()));
+ row.addToColumnValue(new
TCell().setStringVal(entry.getUuid()));
+ row.addToColumnValue(new
TCell().setLongVal(entry.getDatasetVersion()));
+ rows.add(row);
+ }
+ TFetchSchemaTableDataResult result = new
TFetchSchemaTableDataResult();
+ result.setStatus(new TStatus(TStatusCode.OK));
+ result.setDataBatch(rows);
+ return result;
+ } catch (Exception e) {
+ // Load failures arrive here already sanitized by the catalog
wrapper (provider
+ // credentials and dataset URIs stripped); anything else collapses
to a generic
+ // message so a relayed RPC cannot leak internals. Logging this
wrapper adds no
+ // new exposure and keeps sanitized load failures diagnosable.
+ LOG.warn("Failed to load Lance index entries for {}.{}.{}",
+ catalogName, databaseName, tableName, e);
+ return errorResult(lanceIndexEntriesErrorMessage(e));
+ }
+ }
+
+ private static boolean isBoundedLanceIdentifier(String value) {
+ return value != null && !value.trim().isEmpty()
+ && value.getBytes(StandardCharsets.UTF_8).length <=
MAX_LANCE_IDENTIFIER_BYTES;
+ }
+
+ private static String lanceIndexEntriesErrorMessage(Exception exception) {
+ String message = exception.getMessage();
+ if (message != null && (exception instanceof AnalysisException
+ || message.startsWith("Failed to load Lance index metadata for
"))) {
+ return message;
+ }
+ return "Failed to load Lance index entries.";
+ }
+
private static TFetchSchemaTableDataResult
hudiMetadataResult(TMetadataTableRequestParams params) {
if (!params.isSetHudiMetadataParams()) {
return errorResult("Hudi metadata params is not set.");
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataTableValuedFunction.java
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataTableValuedFunction.java
index 39fde6a5615..43d37816b76 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataTableValuedFunction.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataTableValuedFunction.java
@@ -54,6 +54,8 @@ public abstract class MetadataTableValuedFunction extends
TableValuedFunctionIf
return
JobsTableValuedFunction.getColumnIndexFromColumnName(columnName, params);
case TASKS:
return
TasksTableValuedFunction.getColumnIndexFromColumnName(columnName, params);
+ case LANCE_INDEX_ENTRIES:
+ return
LanceIndexEntriesTableValuedFunction.getColumnIndexFromColumnName(columnName);
default:
throw new AnalysisException("Unknown Metadata
TableValuedFunction type");
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java
index 7a7569583d5..823e12ee4f4 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java
@@ -75,6 +75,8 @@ public abstract class TableValuedFunctionIf {
return new JobsTableValuedFunction(params);
case TasksTableValuedFunction.NAME:
return new TasksTableValuedFunction(params);
+ case LanceIndexEntriesTableValuedFunction.NAME:
+ return new LanceIndexEntriesTableValuedFunction(params);
case ParquetMetadataTableValuedFunction.NAME:
return new ParquetMetadataTableValuedFunction(params);
case ParquetMetadataTableValuedFunction.NAME_FILE_METADATA: {
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceFilesystemCatalogTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceFilesystemCatalogTest.java
index c10de9944d3..df008abe564 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceFilesystemCatalogTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceFilesystemCatalogTest.java
@@ -17,6 +17,8 @@
package org.apache.doris.datasource.lance;
+import org.apache.doris.common.AnalysisException;
+
import org.junit.Assert;
import org.junit.Test;
@@ -44,6 +46,23 @@ import java.util.concurrent.atomic.AtomicReference;
public class LanceFilesystemCatalogTest {
+ @Test
+ public void testLoadTableIndexEntriesRejectsRestCatalogBeforeInit() {
+ Map<String, String> properties = new HashMap<>();
+ properties.put("type", "lance");
+ properties.put(LanceExternalCatalog.LANCE_CATALOG_TYPE,
LanceExternalCatalog.LANCE_REST);
+ properties.put(LanceExternalCatalog.REST_URI, "http://127.0.0.1:1/";);
+ LanceExternalCatalog catalog = new LanceExternalCatalog(
+ 5, "lance_rest_entries", null, properties, "");
+
+ Assert.assertFalse(catalog.isInitialized());
+ AnalysisException exception =
Assert.assertThrows(AnalysisException.class,
+ () -> catalog.loadTableIndexEntries("db", "table"));
+ Assert.assertEquals("Lance index inspection is not supported for Lance
REST catalogs",
+ exception.getDetailMessage());
+ Assert.assertFalse(catalog.isInitialized());
+ }
+
@Test
public void testNamespaceNameRoundTrip() throws Exception {
Assert.assertEquals(Collections.emptyList(),
LanceNamespaceName.dorisDatabaseNameToNamespace(
@@ -91,6 +110,35 @@ public class LanceFilesystemCatalogTest {
LanceNamespaceName.dorisDatabaseNameToNamespace(rootCollision,
".", "default"));
}
+ @Test
+ public void testLoadTableIndexEntriesWrapsFailureWithSanitizedMessage() {
+ String accessKey = "sentinel-access-key";
+ String secretKey = "sentinel-secret-key";
+ Map<String, String> properties = new HashMap<>();
+ properties.put("type", "lance");
+ properties.put(LanceExternalCatalog.LANCE_CATALOG_TYPE,
+ LanceExternalCatalog.LANCE_FILESYSTEM);
+ properties.put(LanceExternalCatalog.WAREHOUSE,
"/nonexistent-lance-warehouse-dir");
+ properties.put("AWS_ACCESS_KEY", accessKey);
+ properties.put("AWS_SECRET_KEY", secretKey);
+ LanceExternalCatalog catalog = new LanceExternalCatalog(
+ 6, "lance_filesystem_entries", null, properties, "");
+
+ RuntimeException exception =
Assert.assertThrows(RuntimeException.class,
+ () -> catalog.loadTableIndexEntries("db", "table"));
+
+ Assert.assertTrue(exception.getMessage().contains(
+ "Failed to load Lance index metadata for db.table: "));
+ Assert.assertNotNull(exception.getCause());
+ StringWriter stackTrace = new StringWriter();
+ exception.printStackTrace(new PrintWriter(stackTrace));
+ for (String sentinel : Arrays.asList(accessKey, secretKey)) {
+ Assert.assertFalse(exception.getMessage().contains(sentinel));
+
Assert.assertFalse(exception.getCause().getMessage().contains(sentinel));
+ Assert.assertFalse(stackTrace.toString().contains(sentinel));
+ }
+ }
+
@Test
public void testIndexMetadataErrorSanitization() {
String bearerToken = "sentinel-bearer-token";
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexMetadataLoaderTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexMetadataLoaderTest.java
index 68a5a79acff..c1b8f5090aa 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexMetadataLoaderTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexMetadataLoaderTest.java
@@ -20,6 +20,7 @@ package org.apache.doris.datasource.lance;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.lance.Dataset;
+import org.lance.index.Index;
import org.lance.index.IndexCriteria;
import org.lance.index.IndexDescription;
import org.lance.schema.LanceField;
@@ -33,6 +34,7 @@ import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.UUID;
import java.util.concurrent.atomic.AtomicInteger;
public class LanceIndexMetadataLoaderTest {
@@ -679,6 +681,156 @@ public class LanceIndexMetadataLoaderTest {
Assertions.assertNull(exception.getCause());
}
+ @Test
+ public void testCollectPhysicalEntriesSortsByNameAndUuid() {
+ Dataset dataset = Mockito.mock(Dataset.class);
+ Index zeta = index("z_idx", "33333333-3333-3333-3333-333333333333", 9);
+ Index alphaHigh = index("a_idx",
"22222222-2222-2222-2222-222222222222", 7);
+ Index alphaLow = index("a_idx",
"11111111-1111-1111-1111-111111111111", 5);
+ Mockito.when(dataset.getIndexes()).thenReturn(Arrays.asList(zeta,
alphaHigh, alphaLow));
+
+ List<LancePhysicalIndexEntry> entries =
+ LanceIndexMetadataLoader.collectPhysicalEntries(dataset);
+
+ Assertions.assertEquals(3, entries.size());
+ Assertions.assertEquals("a_idx", entries.get(0).getName());
+ Assertions.assertEquals("11111111-1111-1111-1111-111111111111",
entries.get(0).getUuid());
+ Assertions.assertEquals(5, entries.get(0).getDatasetVersion());
+ Assertions.assertEquals("a_idx", entries.get(1).getName());
+ Assertions.assertEquals("22222222-2222-2222-2222-222222222222",
entries.get(1).getUuid());
+ Assertions.assertEquals(7, entries.get(1).getDatasetVersion());
+ Assertions.assertEquals("z_idx", entries.get(2).getName());
+ Assertions.assertEquals("33333333-3333-3333-3333-333333333333",
entries.get(2).getUuid());
+ Assertions.assertEquals(9, entries.get(2).getDatasetVersion());
+ Assertions.assertThrows(UnsupportedOperationException.class,
+ () -> entries.add(new LancePhysicalIndexEntry(
+ "x_idx", "44444444-4444-4444-4444-444444444444", 1)));
+ }
+
+ @Test
+ public void testCollectPhysicalEntriesFiltersSystemEntries() {
+ Dataset dataset = Mockito.mock(Dataset.class);
+ Mockito.when(dataset.getIndexes()).thenReturn(Arrays.asList(
+ index("__lance_frag_reuse",
"11111111-1111-1111-1111-111111111111", 9),
+ index("user_idx", "22222222-2222-2222-2222-222222222222", 7),
+ index("__lance_mem_wal",
"33333333-3333-3333-3333-333333333333", 9)));
+
+ List<LancePhysicalIndexEntry> entries =
+ LanceIndexMetadataLoader.collectPhysicalEntries(dataset);
+
+ Assertions.assertEquals(1, entries.size());
+ Assertions.assertEquals("user_idx", entries.get(0).getName());
+ Assertions.assertEquals("22222222-2222-2222-2222-222222222222",
entries.get(0).getUuid());
+ Assertions.assertEquals(7, entries.get(0).getDatasetVersion());
+ }
+
+ @Test
+ public void testCollectPhysicalEntriesKeepsShortMemWalName() {
+ Dataset dataset = Mockito.mock(Dataset.class);
+
Mockito.when(dataset.getIndexes()).thenReturn(Collections.singletonList(
+ index("__mem_wal", "11111111-1111-1111-1111-111111111111",
5)));
+
+ List<LancePhysicalIndexEntry> entries =
+ LanceIndexMetadataLoader.collectPhysicalEntries(dataset);
+
+ Assertions.assertEquals(1, entries.size());
+ Assertions.assertEquals("__mem_wal", entries.get(0).getName());
+ }
+
+ @Test
+ public void testCollectPhysicalEntriesBoundsRawListBeforeFiltering() {
+ Dataset atLimitDataset = Mockito.mock(Dataset.class);
+ List<Index> atLimit = new ArrayList<>(16384);
+ for (int i = 0; i < 16384; ++i) {
+ atLimit.add(index("__lance_frag_reuse",
+ new UUID(0, i).toString(), 1));
+ }
+ Mockito.when(atLimitDataset.getIndexes()).thenReturn(atLimit);
+ Assertions.assertTrue(
+
LanceIndexMetadataLoader.collectPhysicalEntries(atLimitDataset).isEmpty());
+
+ // 16384 user entries plus one system entry still exceed the raw cap,
even though
+ // filtering would leave exactly 16384 and no pair of them shares a
uuid.
+ Dataset overLimitDataset = Mockito.mock(Dataset.class);
+ List<Index> overLimit = new ArrayList<>(16385);
+ for (int i = 0; i < 16384; ++i) {
+ overLimit.add(index("bulk_idx",
+ new UUID(0, i).toString(), 1));
+ }
+ overLimit.add(index("__lance_mem_wal",
"22222222-2222-2222-2222-222222222222", 1));
+ Mockito.when(overLimitDataset.getIndexes()).thenReturn(overLimit);
+
+ IllegalArgumentException exception = Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
LanceIndexMetadataLoader.collectPhysicalEntries(overLimitDataset));
+ Assertions.assertTrue(exception.getMessage().contains("16384"));
+ }
+
+ @Test
+ public void testCollectPhysicalEntriesRejectsInvalidEntries() {
+ assertPhysicalEntryFailure(
+ Collections.singletonList(null), "must not be null");
+ assertPhysicalEntryFailure(Collections.singletonList(
+ index(null, "11111111-1111-1111-1111-111111111111", 1)),
"name");
+ assertPhysicalEntryFailure(Collections.singletonList(
+ index("", "11111111-1111-1111-1111-111111111111", 1)), "name");
+ assertPhysicalEntryFailure(Collections.singletonList(
+ index(" ", "11111111-1111-1111-1111-111111111111", 1)),
"blank");
+ assertPhysicalEntryFailure(Collections.singletonList(
+ index(repeat("x", 1025),
"11111111-1111-1111-1111-111111111111", 1)), "1024");
+ assertPhysicalEntryFailure(Collections.singletonList(
+ index("idx", null, 1)), "uuid");
+ assertPhysicalEntryFailure(Collections.singletonList(
+ index("idx", "11111111-1111-1111-1111-111111111111", 0)),
"positive");
+ assertPhysicalEntryFailure(Collections.singletonList(
+ index("idx", "11111111-1111-1111-1111-111111111111", -1)),
"positive");
+ assertPhysicalEntryFailure(Arrays.asList(
+ index("first_idx", "11111111-1111-1111-1111-111111111111", 1),
+ index("second_idx", "11111111-1111-1111-1111-111111111111",
2)), "Duplicate");
+ assertPhysicalEntryFailure(Collections.singletonList(
+ index("__lance_frag_reuse", null, 1)), "uuid");
+ assertPhysicalEntryFailure(Arrays.asList(
+ index("__lance_frag_reuse",
"11111111-1111-1111-1111-111111111111", 1),
+ index("user_idx", "11111111-1111-1111-1111-111111111111", 1)),
"Duplicate");
+ }
+
+ private static void assertPhysicalEntryFailure(List<Index> indexes, String
expectedMessage) {
+ Dataset dataset = Mockito.mock(Dataset.class);
+ Mockito.when(dataset.getIndexes()).thenReturn(indexes);
+ IllegalArgumentException exception = Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () ->
LanceIndexMetadataLoader.collectPhysicalEntries(dataset));
+ Assertions.assertTrue(exception.getMessage().contains(expectedMessage),
+ "message <" + exception.getMessage() + "> should contain <" +
expectedMessage + ">");
+ }
+
+ @Test
+ public void testCollectPhysicalEntriesNeverReadsBeyondGetIndexes() {
+ Dataset dataset = Mockito.mock(Dataset.class);
+ Mockito.when(dataset.getIndexes()).thenReturn(Arrays.asList(
+ index("a_idx", "11111111-1111-1111-1111-111111111111", 5),
+ index("__lance_frag_reuse",
"22222222-2222-2222-2222-222222222222", 5)));
+
+ Assertions.assertEquals(1,
+
LanceIndexMetadataLoader.collectPhysicalEntries(dataset).size());
+
+ Mockito.verify(dataset).getIndexes();
+ Mockito.verify(dataset,
Mockito.never()).describeIndices(Mockito.any(IndexCriteria.class));
+ Mockito.verify(dataset, Mockito.never()).describeIndices();
+ Mockito.verify(dataset, Mockito.never()).countRows();
+ Mockito.verify(dataset,
Mockito.never()).countRows(Mockito.anyString());
+ Mockito.verify(dataset,
Mockito.never()).getIndexStatistics(Mockito.anyString());
+ Mockito.verify(dataset, Mockito.never()).getLanceSchema();
+ }
+
+ private static Index index(String name, String uuid, long datasetVersion) {
+ return Index.builder()
+ .uuid(uuid == null ? null : UUID.fromString(uuid))
+ .name(name)
+ .datasetVersion(datasetVersion)
+ .build();
+ }
+
private static void assertBoundFailure(IndexDescription description,
Map<Integer, String> fields, String expectedType, String
expectedLimit) {
IllegalArgumentException exception = Assertions.assertThrows(
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/LanceIndexEntriesTableValuedFunctionAuthTest.java
b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/LanceIndexEntriesTableValuedFunctionAuthTest.java
new file mode 100644
index 00000000000..37cf00cfe4e
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/LanceIndexEntriesTableValuedFunctionAuthTest.java
@@ -0,0 +1,203 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.tablefunction;
+
+import org.apache.doris.analysis.TableName;
+import org.apache.doris.analysis.UserIdentity;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.datasource.CatalogMgr;
+import org.apache.doris.datasource.ExternalDatabase;
+import org.apache.doris.datasource.lance.LanceExternalCatalog;
+import org.apache.doris.datasource.lance.LanceExternalTable;
+import org.apache.doris.datasource.lance.LancePhysicalIndexEntry;
+import org.apache.doris.mysql.privilege.AccessControllerManager;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.thrift.TFetchSchemaTableDataRequest;
+import org.apache.doris.thrift.TFetchSchemaTableDataResult;
+import org.apache.doris.thrift.TLanceIndexMetadataParams;
+import org.apache.doris.thrift.TMetadataTableRequestParams;
+import org.apache.doris.thrift.TMetadataType;
+import org.apache.doris.thrift.TRow;
+import org.apache.doris.thrift.TStatusCode;
+import org.apache.doris.thrift.TUserIdentity;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import mockit.Expectations;
+import mockit.Mocked;
+import mockit.Verifications;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class LanceIndexEntriesTableValuedFunctionAuthTest {
+ @Mocked
+ private Env env;
+ @Mocked
+ private AccessControllerManager accessManager;
+ @Mocked
+ private ConnectContext context;
+ @Mocked
+ private CatalogMgr catalogMgr;
+ @Mocked
+ private LanceExternalCatalog catalog;
+ @Mocked
+ private ExternalDatabase database;
+ @Mocked
+ private LanceExternalTable table;
+
+ @Test
+ public void testShowDeniedBeforeCatalogLookup() {
+ new Expectations() {
+ {
+ ConnectContext.get();
+ result = context;
+ Env.getCurrentEnv();
+ result = env;
+ env.getAccessManager();
+ result = accessManager;
+ accessManager.checkTblPriv(context,
+ withInstanceOf(TableName.class), PrivPredicate.SHOW);
+ result = false;
+ context.getQualifiedUser();
+ result = "denied_user";
+ context.getRemoteIP();
+ result = "127.0.0.1";
+ }
+ };
+
+ AnalysisException exception =
Assert.assertThrows(AnalysisException.class,
+ () -> new LanceIndexEntriesTableValuedFunction(
+ ImmutableMap.of("table", "ctl.db.tbl")));
+ Assert.assertTrue(exception.getMessage().contains("denied"));
+
+ new Verifications() {
+ {
+ env.getCatalogMgr();
+ times = 0;
+ }
+ };
+ }
+
+ @Test
+ public void testMasterRejectsMissingLanceParams() throws Exception {
+ TMetadataTableRequestParams params = new TMetadataTableRequestParams()
+ .setMetadataType(TMetadataType.LANCE_INDEX_ENTRIES);
+ TFetchSchemaTableDataRequest request = new
TFetchSchemaTableDataRequest()
+ .setMetadaTableParams(params);
+
+ TFetchSchemaTableDataResult result =
MetadataGenerator.getMetadataTable(request);
+
+ Assert.assertEquals(TStatusCode.INTERNAL_ERROR,
result.getStatus().getStatusCode());
+ Assert.assertTrue(result.getStatus().getErrorMsgs().toString()
+ .contains("Lance index metadata params is not set."));
+ Assert.assertFalse(result.isSetDataBatch());
+ }
+
+ @Test
+ public void testMasterRepeatsShowCheckBeforeCatalogLookup() {
+ new Expectations() {
+ {
+ Env.getCurrentEnv();
+ result = env;
+ env.getAccessManager();
+ result = accessManager;
+ accessManager.checkTblPriv(withInstanceOf(UserIdentity.class),
+ "ctl", "db", "tbl", PrivPredicate.SHOW);
+ result = false;
+ }
+ };
+
+ TFetchSchemaTableDataResult result =
MetadataGenerator.lanceIndexEntriesMetadataResult(
+ masterParams("ctl", "db", "tbl"));
+
+ Assert.assertEquals(TStatusCode.INTERNAL_ERROR,
result.getStatus().getStatusCode());
+
Assert.assertTrue(result.getStatus().getErrorMsgs().toString().contains("denied"));
+ Assert.assertFalse(result.isSetDataBatch());
+ new Verifications() {
+ {
+ env.getCatalogMgr();
+ times = 0;
+ }
+ };
+ }
+
+ @Test
+ public void testMasterPreservesRelayedNamesAndBuildsSixCellRows() throws
Exception {
+ new Expectations() {
+ {
+ Env.getCurrentEnv();
+ result = env;
+ times = 2;
+ env.getAccessManager();
+ result = accessManager;
+ accessManager.checkTblPriv(withInstanceOf(UserIdentity.class),
+ " ctl ", " db ", " tbl ", PrivPredicate.SHOW);
+ result = true;
+ env.getCatalogMgr();
+ result = catalogMgr;
+ catalogMgr.getCatalog(" ctl ");
+ result = catalog;
+ catalog.isRestCatalogConfigured();
+ result = false;
+ catalog.getDbOrAnalysisException(" db ");
+ result = database;
+ database.getTableOrAnalysisException(" tbl ");
+ result = table;
+ catalog.getName();
+ result = "resolved_catalog";
+ database.getFullName();
+ result = "resolved_db";
+ table.getName();
+ result = "resolved_table";
+ table.loadIndexEntries();
+ result = ImmutableList.of(new LancePhysicalIndexEntry("idx",
"uuid-1", 7));
+ }
+ };
+
+ TFetchSchemaTableDataResult result =
MetadataGenerator.lanceIndexEntriesMetadataResult(
+ masterParams(" ctl ", " db ", " tbl "));
+
+ Assert.assertEquals(TStatusCode.OK,
result.getStatus().getStatusCode());
+ Assert.assertEquals(1, result.getDataBatchSize());
+ TRow row = result.getDataBatch().get(0);
+ Assert.assertEquals(6, row.getColumnValueSize());
+ Assert.assertEquals("resolved_catalog",
row.getColumnValue().get(0).getStringVal());
+ Assert.assertEquals("resolved_db",
row.getColumnValue().get(1).getStringVal());
+ Assert.assertEquals("resolved_table",
row.getColumnValue().get(2).getStringVal());
+ Assert.assertEquals("idx", row.getColumnValue().get(3).getStringVal());
+ Assert.assertEquals("uuid-1",
row.getColumnValue().get(4).getStringVal());
+ Assert.assertEquals(7, row.getColumnValue().get(5).getLongVal());
+ }
+
+ private static TMetadataTableRequestParams masterParams(
+ String catalogName, String databaseName, String tableName) {
+ TLanceIndexMetadataParams lanceParams = new TLanceIndexMetadataParams()
+ .setCatalog(catalogName)
+ .setDatabase(databaseName)
+ .setTable(tableName);
+ TUserIdentity user = new TUserIdentity()
+ .setUsername("denied_user")
+ .setHost("127.0.0.1");
+ return new TMetadataTableRequestParams()
+ .setMetadataType(TMetadataType.LANCE_INDEX_ENTRIES)
+ .setLanceIndexMetadataParams(lanceParams)
+ .setCurrentUserIdent(user);
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/LanceIndexEntriesTableValuedFunctionTest.java
b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/LanceIndexEntriesTableValuedFunctionTest.java
new file mode 100644
index 00000000000..2f68c8063f5
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/LanceIndexEntriesTableValuedFunctionTest.java
@@ -0,0 +1,345 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.tablefunction;
+
+import org.apache.doris.analysis.TableName;
+import org.apache.doris.catalog.BuiltinTableValuedFunctions;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.datasource.CatalogMgr;
+import org.apache.doris.datasource.ExternalDatabase;
+import org.apache.doris.datasource.lance.LanceExternalCatalog;
+import org.apache.doris.datasource.lance.LanceExternalTable;
+import org.apache.doris.mysql.privilege.AccessControllerManager;
+import org.apache.doris.mysql.privilege.PrivPredicate;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.thrift.TLanceIndexMetadataParams;
+import org.apache.doris.thrift.TMetaScanRange;
+import org.apache.doris.thrift.TMetadataType;
+
+import com.google.common.collect.ImmutableMap;
+import mockit.Expectations;
+import mockit.Mocked;
+import mockit.Verifications;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+public class LanceIndexEntriesTableValuedFunctionTest {
+ private static final List<String> PINNED_COLUMN_NAMES = Arrays.asList(
+ "CatalogName", "DatabaseName", "TableName", "IndexName",
"IndexUuid", "DatasetVersion");
+
+ @Mocked
+ private Env env;
+ @Mocked
+ private AccessControllerManager accessManager;
+ @Mocked
+ private ConnectContext context;
+ @Mocked
+ private CatalogMgr catalogMgr;
+ @Mocked
+ private LanceExternalCatalog lanceCatalog;
+ @Mocked
+ private CatalogIf nonLanceCatalog;
+ @Mocked
+ private ExternalDatabase database;
+ @Mocked
+ private LanceExternalTable lanceTable;
+ @Mocked
+ private TableIf nonLanceTable;
+
+ @Test
+ public void testSchemaHasExactlySixPinnedColumnsInOrder() {
+ List<Column> schema =
LanceIndexEntriesTableValuedFunction.getSchemaForTest();
+
+ Assert.assertEquals(PINNED_COLUMN_NAMES,
schema.stream().map(Column::getName)
+ .collect(java.util.stream.Collectors.toList()));
+ for (int i = 0; i < 5; i++) {
+ Assert.assertEquals(PrimitiveType.STRING,
schema.get(i).getDataType());
+ Assert.assertFalse(schema.get(i).isAllowNull());
+ }
+ Assert.assertEquals(PrimitiveType.BIGINT, schema.get(5).getDataType());
+ Assert.assertFalse(schema.get(5).isAllowNull());
+ }
+
+ @Test
+ public void testColumnIndexRoundTripCaseInsensitive() {
+ for (int i = 0; i < PINNED_COLUMN_NAMES.size(); i++) {
+ String name = PINNED_COLUMN_NAMES.get(i);
+ Assert.assertEquals(Integer.valueOf(i),
+
LanceIndexEntriesTableValuedFunction.getColumnIndexFromColumnName(name));
+ Assert.assertEquals(Integer.valueOf(i),
+
LanceIndexEntriesTableValuedFunction.getColumnIndexFromColumnName(
+ name.toLowerCase(java.util.Locale.ROOT)));
+ Assert.assertEquals(Integer.valueOf(i),
+
LanceIndexEntriesTableValuedFunction.getColumnIndexFromColumnName(
+ name.toUpperCase(java.util.Locale.ROOT)));
+ }
+ Assert.assertNull(
+
LanceIndexEntriesTableValuedFunction.getColumnIndexFromColumnName("NoSuchColumn"));
+ }
+
+ @Test
+ public void testMissingTablePropertyRejected() {
+ Assert.assertThrows(AnalysisException.class,
+ () -> LanceIndexEntriesTableValuedFunction.normalizeProperties(
+ new LinkedHashMap<>()));
+
+ Map<String, String> blank = new LinkedHashMap<>();
+ blank.put("table", " ");
+ Assert.assertThrows(AnalysisException.class,
+ () ->
LanceIndexEntriesTableValuedFunction.normalizeProperties(blank));
+ }
+
+ @Test
+ public void testDuplicateTablePropertyCaseVariantRejected() {
+ Map<String, String> properties = new LinkedHashMap<>();
+ properties.put("TABLE", null);
+ properties.put("table", "ctl.db.tbl");
+
+ Assert.assertThrows(AnalysisException.class,
+ () ->
LanceIndexEntriesTableValuedFunction.normalizeProperties(properties));
+ }
+
+ @Test
+ public void testUnknownPropertyRejected() {
+ Map<String, String> properties = new LinkedHashMap<>();
+ properties.put("table", "ctl.db.tbl");
+ properties.put("deadline", "1");
+
+ Assert.assertThrows(AnalysisException.class,
+ () ->
LanceIndexEntriesTableValuedFunction.normalizeProperties(properties));
+ }
+
+ @Test
+ public void testTwoPartNameRejected() {
+ Assert.assertThrows(AnalysisException.class,
+ () ->
LanceIndexEntriesTableValuedFunction.parseTableName("db.tbl"));
+ }
+
+ @Test
+ public void testFourPartNameRejected() {
+ Assert.assertThrows(AnalysisException.class,
+ () ->
LanceIndexEntriesTableValuedFunction.parseTableName("lance_catalog.doris.analytics.items"));
+ }
+
+ @Test
+ public void testBacktickQuotedDottedNamesParse() throws Exception {
+ TableName name = LanceIndexEntriesTableValuedFunction.parseTableName(
+ "lance_catalog.`doris.analytics`.`my.items`");
+
+ Assert.assertEquals("lance_catalog", name.getCtl());
+ Assert.assertEquals("doris.analytics", name.getDb());
+ Assert.assertEquals("my.items", name.getTbl());
+ }
+
+ @Test
+ public void testNonLanceCatalogRejected() {
+ new Expectations() {
+ {
+ ConnectContext.get();
+ result = context;
+ Env.getCurrentEnv();
+ result = env;
+ times = 2;
+ env.getAccessManager();
+ result = accessManager;
+ accessManager.checkTblPriv(context,
+ withInstanceOf(TableName.class), PrivPredicate.SHOW);
+ result = true;
+ env.getCatalogMgr();
+ result = catalogMgr;
+ catalogMgr.getCatalog("ctl");
+ result = nonLanceCatalog;
+ }
+ };
+
+ AnalysisException exception =
Assert.assertThrows(AnalysisException.class,
+ () -> new LanceIndexEntriesTableValuedFunction(
+ ImmutableMap.of("table", "ctl.db.tbl")));
+ Assert.assertTrue(exception.getMessage().contains("is not a Lance
catalog"));
+ }
+
+ @Test
+ public void testNonLanceTableInLanceCatalogRejected() throws Exception {
+ new Expectations() {
+ {
+ ConnectContext.get();
+ result = context;
+ Env.getCurrentEnv();
+ result = env;
+ times = 2;
+ env.getAccessManager();
+ result = accessManager;
+ accessManager.checkTblPriv(context,
+ withInstanceOf(TableName.class), PrivPredicate.SHOW);
+ result = true;
+ env.getCatalogMgr();
+ result = catalogMgr;
+ catalogMgr.getCatalog("ctl");
+ result = lanceCatalog;
+ lanceCatalog.isRestCatalogConfigured();
+ result = false;
+ lanceCatalog.getDbOrAnalysisException("db");
+ result = database;
+ database.getTableOrAnalysisException("tbl");
+ result = nonLanceTable;
+ }
+ };
+
+ AnalysisException exception =
Assert.assertThrows(AnalysisException.class,
+ () -> new LanceIndexEntriesTableValuedFunction(
+ ImmutableMap.of("table", "ctl.db.tbl")));
+ Assert.assertTrue(exception.getMessage().contains("is not a Lance
table"));
+ }
+
+ @Test
+ public void testRestCatalogRejectedBeforeDatabaseResolution() throws
Exception {
+ new Expectations() {
+ {
+ ConnectContext.get();
+ result = context;
+ Env.getCurrentEnv();
+ result = env;
+ times = 2;
+ env.getAccessManager();
+ result = accessManager;
+ accessManager.checkTblPriv(context,
+ withInstanceOf(TableName.class), PrivPredicate.SHOW);
+ result = true;
+ env.getCatalogMgr();
+ result = catalogMgr;
+ catalogMgr.getCatalog("ctl");
+ result = lanceCatalog;
+ lanceCatalog.isRestCatalogConfigured();
+ result = true;
+ }
+ };
+
+ AnalysisException exception =
Assert.assertThrows(AnalysisException.class,
+ () -> new LanceIndexEntriesTableValuedFunction(
+ ImmutableMap.of("table", "ctl.db.tbl")));
+ Assert.assertEquals("lance_index_entries is not supported for Lance
REST catalogs",
+ exception.getMessage());
+
+ new Verifications() {
+ {
+ lanceCatalog.getDbOrAnalysisException(anyString);
+ times = 0;
+ }
+ };
+ }
+
+ @Test
+ public void testGetMetaScanRangeCarriesTableIdentity() throws Exception {
+ new Expectations() {
+ {
+ ConnectContext.get();
+ result = context;
+ Env.getCurrentEnv();
+ result = env;
+ times = 2;
+ env.getAccessManager();
+ result = accessManager;
+ accessManager.checkTblPriv(context,
+ withInstanceOf(TableName.class), PrivPredicate.SHOW);
+ result = true;
+ env.getCatalogMgr();
+ result = catalogMgr;
+ catalogMgr.getCatalog(" ctl ");
+ result = lanceCatalog;
+ lanceCatalog.isRestCatalogConfigured();
+ result = false;
+ lanceCatalog.getDbOrAnalysisException(" db ");
+ result = database;
+ database.getTableOrAnalysisException(" tbl ");
+ result = lanceTable;
+ }
+ };
+
+ LanceIndexEntriesTableValuedFunction tvf = new
LanceIndexEntriesTableValuedFunction(
+ ImmutableMap.of("TABLE", " ` ctl `.` db `.` tbl ` "));
+ Assert.assertEquals(" ctl ", tvf.getCatalogName());
+ Assert.assertEquals(" db ", tvf.getDatabaseName());
+ Assert.assertEquals(" tbl ", tvf.getSourceTableName());
+ Assert.assertEquals(TMetadataType.LANCE_INDEX_ENTRIES,
tvf.getMetadataType());
+
+ TMetaScanRange scanRange =
tvf.getMetaScanRange(Collections.emptyList());
+ Assert.assertEquals(TMetadataType.LANCE_INDEX_ENTRIES,
scanRange.getMetadataType());
+ Assert.assertTrue(scanRange.isSetLanceIndexParams());
+ TLanceIndexMetadataParams params = scanRange.getLanceIndexParams();
+ Assert.assertEquals(" ctl ", params.getCatalog());
+ Assert.assertEquals(" db ", params.getDatabase());
+ Assert.assertEquals(" tbl ", params.getTable());
+
+ new Verifications() {
+ {
+ lanceTable.loadIndexEntries();
+ times = 0;
+ }
+ };
+ }
+
+ @Test
+ public void testRegistrationWiring() throws Exception {
+
Assert.assertTrue(BuiltinTableValuedFunctions.INSTANCE.tableValuedFunctions.stream()
+ .anyMatch(func ->
func.names.contains(LanceIndexEntriesTableValuedFunction.NAME)));
+
+ Assert.assertEquals(Integer.valueOf(4),
+ MetadataTableValuedFunction.getColumnIndexFromColumnName(
+ TMetadataType.LANCE_INDEX_ENTRIES, "indexuuid", null));
+
+ new Expectations() {
+ {
+ ConnectContext.get();
+ result = context;
+ Env.getCurrentEnv();
+ result = env;
+ times = 2;
+ env.getAccessManager();
+ result = accessManager;
+ accessManager.checkTblPriv(context,
+ withInstanceOf(TableName.class), PrivPredicate.SHOW);
+ result = true;
+ env.getCatalogMgr();
+ result = catalogMgr;
+ catalogMgr.getCatalog("ctl");
+ result = lanceCatalog;
+ lanceCatalog.isRestCatalogConfigured();
+ result = false;
+ lanceCatalog.getDbOrAnalysisException("db");
+ result = database;
+ database.getTableOrAnalysisException("tbl");
+ result = lanceTable;
+ }
+ };
+
+ TableValuedFunctionIf tvf = TableValuedFunctionIf.getTableFunction(
+ LanceIndexEntriesTableValuedFunction.NAME,
ImmutableMap.of("table", "ctl.db.tbl"));
+ Assert.assertTrue(tvf instanceof LanceIndexEntriesTableValuedFunction);
+ }
+}
diff --git a/gensrc/thrift/FrontendService.thrift
b/gensrc/thrift/FrontendService.thrift
index f06ceb3ceb8..7cf5bdc99b5 100644
--- a/gensrc/thrift/FrontendService.thrift
+++ b/gensrc/thrift/FrontendService.thrift
@@ -914,6 +914,7 @@ struct TMetadataTableRequestParams {
// Reserved for downstream field `current_roles` to keep thrift field ids
// wire-compatible across maintained branches. Do not reuse this id.
15: optional set<string> reserved_field_15
+ 16: optional PlanNodes.TLanceIndexMetadataParams lance_index_metadata_params
}
struct TSchemaTableRequestParams {
diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift
index 90833189d18..0e9f7e6d2f0 100644
--- a/gensrc/thrift/PlanNodes.thrift
+++ b/gensrc/thrift/PlanNodes.thrift
@@ -819,6 +819,13 @@ struct TParquetMetadataParams {
6: optional string bloom_literal
}
+// Identifies a Lance table for read-only physical index entry inspection.
+struct TLanceIndexMetadataParams {
+ 1: optional string catalog
+ 2: optional string database
+ 3: optional string table
+}
+
struct TMetaScanRange {
1: optional Types.TMetadataType metadata_type
2: optional TIcebergMetadataParams iceberg_params // deprecated
@@ -839,6 +846,7 @@ struct TMetaScanRange {
15: optional string serialized_table;
16: optional list<string> serialized_splits;
17: optional TParquetMetadataParams parquet_params;
+ 18: optional TLanceIndexMetadataParams lance_index_params;
}
// Specification of an individual data range which is held in its entirety
diff --git a/gensrc/thrift/Types.thrift b/gensrc/thrift/Types.thrift
index 23ce5153e01..aecf7053e18 100644
--- a/gensrc/thrift/Types.thrift
+++ b/gensrc/thrift/Types.thrift
@@ -764,6 +764,9 @@ enum TMetadataType {
HUDI = 11,
PAIMON = 12,
PARQUET = 13,
+ // 14 is STREAMS on master. Reserved to keep enum values wire-compatible
+ // across maintained branches. Do not reuse this value on branch-4.1.
+ LANCE_INDEX_ENTRIES = 15,
}
// deprecated
diff --git
a/regression-test/data/external_table_p0/lance/test_lance_index_entries.out
b/regression-test/data/external_table_p0/lance/test_lance_index_entries.out
new file mode 100644
index 00000000000..89b83252461
--- /dev/null
+++ b/regression-test/data/external_table_p0/lance/test_lance_index_entries.out
@@ -0,0 +1,14 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !entries_vector --
+test_lance_index_entries doris vs_ivf_pq_f32 embedding_ivf_pq_f32
UUID SET
+
+-- !entries_vector_count --
+1 1 1
+
+-- !entries_nested --
+test_lance_index_entries doris nested_index nested_label_btree
UUID SET
+
+-- !entries_no_indexes --
+0
+
+-- !entries_predicate --
diff --git
a/regression-test/suites/external_table_p0/lance/test_lance_index_entries.groovy
b/regression-test/suites/external_table_p0/lance/test_lance_index_entries.groovy
new file mode 100644
index 00000000000..c04008c163c
--- /dev/null
+++
b/regression-test/suites/external_table_p0/lance/test_lance_index_entries.groovy
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_lance_index_entries", "p0,external") {
+ String enabled = context.config.otherConfigs.get("enableIcebergTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disable Lance index entries test because the Iceberg
MinIO environment is disabled.")
+ return
+ }
+
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String minioPort = context.config.otherConfigs.get("iceberg_minio_port")
+ String lanceRestPort = context.config.otherConfigs.get("lance_rest_port")
+ String filesystemCatalog = "test_lance_index_entries"
+ String restCatalog = "test_lance_index_entries_rest"
+ String user = "test_lance_index_entries_user"
+ String password = "C123_567p"
+
+ // Index UUIDs are fixture-build artifacts; assert the shape instead of
the value.
+ String uuidShape =
"'^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\$'"
+
+ sql """DROP CATALOG IF EXISTS `${filesystemCatalog}`"""
+ sql """DROP CATALOG IF EXISTS `${restCatalog}`"""
+ try_sql "DROP USER '${user}'@'%'"
+
+ try {
+ sql """
+ CREATE CATALOG `${filesystemCatalog}` PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "filesystem",
+ "warehouse" = "s3://warehouse/lance",
+ "s3.endpoint" = "http://${externalEnvIp}:${minioPort}";,
+ "s3.access_key" = "admin",
+ "s3.secret_key" = "password",
+ "s3.region" = "us-east-1",
+ "use_path_style" = "true"
+ )
+ """
+
+ order_qt_entries_vector """
+ SELECT CatalogName, DatabaseName, TableName, IndexName,
+ IF(IndexUuid REGEXP ${uuidShape}, 'UUID', CONCAT('BAD:',
IndexUuid)) AS UuidShape,
+ IF(DatasetVersion >= 1, 'SET', 'BAD') AS VersionState
+ FROM lance_index_entries("table" =
"${filesystemCatalog}.doris.vs_ivf_pq_f32")
+ """
+
+ qt_entries_vector_count """
+ SELECT COUNT(*), COUNT(DISTINCT IndexName), COUNT(DISTINCT
IndexUuid)
+ FROM lance_index_entries("table" =
"${filesystemCatalog}.doris.vs_ivf_pq_f32")
+ """
+
+ order_qt_entries_nested """
+ SELECT CatalogName, DatabaseName, TableName, IndexName,
+ IF(IndexUuid REGEXP ${uuidShape}, 'UUID', CONCAT('BAD:',
IndexUuid)) AS UuidShape,
+ IF(DatasetVersion >= 1, 'SET', 'BAD') AS VersionState
+ FROM lance_index_entries("table" =
"${filesystemCatalog}.doris.nested_index")
+ """
+
+ qt_entries_no_indexes """
+ SELECT COUNT(*) FROM lance_index_entries("table" =
"${filesystemCatalog}.doris.predicate_pushdown")
+ """
+
+ // An ordinary predicate filters the bounded result.
+ qt_entries_predicate """
+ SELECT IndexName FROM lance_index_entries("table" =
"${filesystemCatalog}.doris.vs_ivf_pq_f32")
+ WHERE IndexName = "no_such_index"
+ """
+
+ sql """
+ CREATE CATALOG `${restCatalog}` PROPERTIES (
+ "type" = "lance",
+ "lance.catalog.type" = "rest",
+ "lance.rest.uri" = "http://${externalEnvIp}:${lanceRestPort}";,
+ "lance.rest.security.type" = "bearer",
+ "lance.rest.bearer-token" = "doris-lance-rest-test-token",
+ "lance.namespace.root_database" = "default",
+ "s3.endpoint" = "http://${externalEnvIp}:${minioPort}";,
+ "s3.region" = "us-east-1",
+ "use_path_style" = "true",
+ "test_connection" = "true"
+ )
+ """
+
+ test {
+ sql """SELECT * FROM lance_index_entries("table" =
"${restCatalog}.`default`.all_types")"""
+ exception "lance_index_entries is not supported for Lance REST
catalogs"
+ }
+
+ sql """CREATE USER '${user}'@'%' IDENTIFIED BY '${password}'"""
+ sql """GRANT SELECT_PRIV ON regression_test TO '${user}'@'%'"""
+ if (isCloudMode()) {
+ def clusters = sql "SHOW CLUSTERS"
+ assertTrue(!clusters.isEmpty())
+ sql """GRANT USAGE_PRIV ON CLUSTER `${clusters[0][0]}` TO
'${user}'@'%'"""
+ }
+
+ connect(user, password, context.config.jdbcUrl) {
+ test {
+ sql """SELECT * FROM lance_index_entries("table" =
"${filesystemCatalog}.doris.vs_ivf_pq_f32")"""
+ exception "denied"
+ }
+ }
+ } finally {
+ try_sql "DROP USER '${user}'@'%'"
+ // Keep both catalogs for debugging when the suite fails.
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
