pvary commented on code in PR #16961: URL: https://github.com/apache/iceberg/pull/16961#discussion_r3543784125
########## format/index.md: ########## @@ -0,0 +1,309 @@ +--- +title: "Index Spec" +--- +<!-- + - Licensed to the Apache Software Foundation (ASF) under one or more + - contributor license agreements. See the NOTICE file distributed with + - this work for additional information regarding copyright ownership. + - The ASF licenses this file to You under the Apache License, Version 2.0 + - (the "License"); you may not use this file except in compliance with + - the License. You may obtain a copy of the License at + - + - http://www.apache.org/licenses/LICENSE-2.0 + - + - Unless required by applicable law or agreed to in writing, software + - distributed under the License is distributed on an "AS IS" BASIS, + - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + - See the License for the specific language governing permissions and + - limitations under the License. + --> + +# Iceberg Index Specification + +## Background and Motivation + +Indexes enable query engines to locate relevant rows without scanning entire datasets. +They can accelerate point lookups, range predicates, and other retrieval patterns +while preserving Iceberg's table format, snapshot isolation, and interoperability. + +Indexes are optional. Engines may choose to create, maintain, consume, or ignore them. + +## Goals + +- Define a portable metadata format for indexes +- Provide a common storage architecture for index data +- Expose indexes as catalog-managed objects +- Allow indexes to be operated independently from source table metadata +- Enable index sharing across engines +- Provide a framework for defining new index types and transform functions + +## Overview + +Indexes are stored as a collection of files with some Iceberg table like semantics. At a high level they consist of a tracking file (similar to a root manifest file) which contains listings for a defined set of leaf files (similar to data files.) Leaf files store an ordered set of rows containing at least a key and the path of a Iceberg Table data file and the position within that file where the row where that key is stored. The organization of leaf files is defined by an Index Transform Function which varies based on the type of index. This structure is recorded in an Index metadata.json file which contains a set of snapshots, each of which points to a single tracking file mapping to the complete state of an Iceberg table at a given Iceberg table snapshot. + +Like Iceberg tables, views, and functions: + +- Metadata files (index metadata and tracking files) and data files (leaf files) are immutable +- Updates create new metadata files +- Catalogs perform atomic metadata swaps + +Each index snapshot references a tracking file which describes the leaf files belonging to the snapshot. + +```text +Index Metadata + | + +-- Index Snapshot + | + +-- Tracking File + | + +-- Leaf Data Files +``` + +Transform functions derive a transform value from the key columns and determine how index entries are organized within +the leaf files. +- The transform value space is divided into non-overlapping ranges. +- Each leaf file stores entries for a single range. +- The tracking file stores range bounds for each leaf file. + +This structure enables efficient planning while keeping the data layout flexible for different index implementations. + +## Definitions + +### Index Type + +The index type defines the logical category of an index and the class of queries it is designed to accelerate. + +The metadata, snapshot, tracking-file, and leaf-file structures defined in this specification form a generic framework shared by all index types. Each index type builds on this framework by defining its type-specific details, such as the leaf schema and the applicable transform functions. + +The following index type is fully defined in this specification: + +| Type | Description | +|--------|------------------------------------------------------------------| +| SCALAR | Maps scalar key values to their locations for equality lookups. | + +The following index types are reserved for future specifications. Their identifiers are claimed so that engines and catalog implementations recognize them as valid type names and handle them gracefully, but this specification defines no type-specific requirements (leaf schema, transforms, or query semantics) for them: + +| Type | Description | +|--------|----------------------------------------------------------| +| VECTOR | Reserved for similarity search over vector embeddings. | +| TERM | Reserved for text/term search. | + +The index type communicates the capabilities of an index to query engines and helps determine whether an index is +applicable to a particular query. + +### Index Transform Function + +The index transform function defines how the transform value is derived from the key columns when rows are stored in the +index. The following terms are used throughout this specification: + +- **Key columns**: the source-table columns the transform function is applied to. +- **Transform value**: the value produced by applying the transform function to a row's key columns. Index entries are organized by transform value. +- **Included columns**: optional source-table columns copied into the index for read convenience. They do not affect how the index is organized. + +The transform function determines the physical organization of the indexed data and therefore influences which query +patterns can efficiently leverage the index. + +The following transform functions are defined in this specification. The bound interpretation describes what the +transform-value bounds stored in the tracking file represent for each transform: + +| Transform | Bound Interpretation | +|-----------|----------------------| +| IDENTITY | Original value range | +| HASH | Hash bucket range | +| HILBERT | Hilbert key range | + +The following transform function is reserved for future specifications: + +| Transform | Bound Interpretation | +|-----------|---------------------------| +| IVF | Centroid identifier range | + +An index type does not fix a single transform function; the same index type can be realized with different transform functions. + +### Index Instance + +An index instance is a concrete realization of an index type and function applied to a specific table. + +Users create index instances by specifying: + +- The source table +- The index type +- The transform function +- The key columns +- The included columns (optional) +- Index properties (optional) + +Multiple instances of the same index type may exist for a table. + +### Index Snapshot + +An index snapshot is an immutable version of the index data generated from a specific table snapshot. Review Comment: >> Use the outdated index and apply incremental read for the new data > If the outdated index still reflects correct information [..] **Index snapshots are explicitly tied to table snapshots.** If a query is executed against an older table snapshot, whether due to snapshot isolation or time travel, the corresponding older index snapshot still points to the correct set of files and can be used safely. If the query is running against the current snapshot of the table, lets imagine the following situation: - Table T has snapshots S1 and S2 - Index I has snapshot IS1, which was built against T.S1 - No index snapshot exists yet for T.S2 In this case, a query against T.S2 could still leverage IS1. The engine would use the index for the data covered by S1 and then evaluate the delta between S1 and S2, applying any additions and removals on top of the indexed result. Since Iceberg records the complete set of file-level changes between snapshots, including rewrites and compactions, the engine can determine which files were added and removed after S1. It can then adjust the result produced by IS1 to reflect the state of T.S2. If the delta is relatively small compared to the size of the table, this approach may still be significantly more efficient than executing the query against the full table without using the index. > During this time period, the stale index can point to files that are not even part of the table any longer. The design explicitly associates each index snapshot with a specific table snapshot, so there is no concept of a "stale" index, as the index itself doesn't represent the current state of the table. Instead, there are table snapshots for which a corresponding index snapshot exists, and table snapshots for which it does not. An index snapshot is always valid for the table snapshot it was built against and therefore only contains references to files that are part of that snapshot. If a query targets that table snapshot, the index remains correct, even if later table snapshots have rewritten or removed those files. The real question is how an engine handles a table snapshot that does not yet have a corresponding index snapshot. In that case, it may choose to leverage the latest available index snapshot and reconcile the delta using the table history, or simply fall back to a table scan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
