ftauheed commented on code in PR #16961: URL: https://github.com/apache/iceberg/pull/16961#discussion_r3720353402
########## format/index-spec.md: ########## @@ -0,0 +1,406 @@ +--- +title: "Index Spec" +--- +<!-- + - Licensed to the Apache Software Foundation (ASF) under one or more + - contributor license agreements. See the NOTICE file distributed with + - this work for additional information regarding copyright ownership. + - The ASF licenses this file to You under the Apache License, Version 2.0 + - (the "License"); you may not use this file except in compliance with + - the License. You may obtain a copy of the License at + - + - http://www.apache.org/licenses/LICENSE-2.0 + - + - Unless required by applicable law or agreed to in writing, software + - distributed under the License is distributed on an "AS IS" BASIS, + - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + - See the License for the specific language governing permissions and + - limitations under the License. + --> + +# Iceberg Index Specification + +## Background and Motivation + +Indexes enable query engines to locate relevant rows without scanning entire datasets. +They can accelerate point lookups, range predicates, and other retrieval patterns +while preserving Iceberg's table format, snapshot isolation, and interoperability. + +Indexes are optional. Engines may choose to create, maintain, consume, or ignore them. + +## Goals + +- Define a portable metadata format for indexes +- Provide a common storage architecture for index data +- Allow indexes to be operated independently of source table metadata +- Enable index sharing across engines +- Provide a framework for defining new index types + +## Overview + +An index is recorded in an Index `metadata.json` file which contains the index definition and a set of index snapshots. +Each index snapshot maps to the complete state of an Iceberg table at a given Iceberg table snapshot and references the +index data for that state. + +Iceberg standardizes the index lifecycle, discovery model, snapshot relationship, and the minimum metadata needed for +safe cross-engine use. Engines remain free to ignore unsupported indexes, use exact snapshot matches only, or implement +more advanced stale-index and incremental-query logic. + +Like Iceberg tables, views, and functions: + +- Index metadata files and index data files are immutable +- Updates create new metadata files +- Catalogs perform atomic metadata swaps + +The index data of a snapshot is organized as a [tracking file](#tracking-file) (similar to a root manifest file) that +lists a set of [leaf files](#leaf-files) (similar to data files): + +```text +Index Metadata + | + +-- Index Snapshot(s) + | + +-- Tracking File + | + +-- Leaf Data Files +``` + +## Definitions + +### Index Type + +The index type defines the logical category of an index and the class of queries it is designed to accelerate. It +communicates the capabilities of an index to query engines and helps determine whether an index is applicable to a +particular query. + +The following index type is defined in this specification: + +| Type | Description | +|--------|-----------------------------------------------------------------------| +| SCALAR | Accelerates point lookups and possibly range filters over key columns | + +The following index type is reserved for future specifications. + +| Type | Description | +|--------|--------------------------------------------------------------------| +| VECTOR | Reserved for accelerating similarity search over vector embeddings | + +### Key Columns + +The source-table columns the index is built on and optimized for retrieval. + +### Transform + +The transform is a function applied to the key columns that produces the values used to order the index entries. + +The following transforms are defined in this specification: + +| Transform | Description | +|-----------|--------------------------------------------------------| +| IDENTITY | Uses the original key values | +| HASH | Hashes the key columns into hash buckets | +| HILBERT | Maps multi-column keys to their Hilbert curve position | + +The ordering each transform defines is described in [Ordering](#ordering). + +### Index Instance + +An index instance is a concrete realization of an index type applied to a specific table. + +Users create index instances by specifying: + +- Source table +- Index type +- Transform +- Key columns +- Index properties (optional) + +Multiple instances of the same index type may exist for a table. + +### Index Snapshot + +An index snapshot is an immutable version of the index data generated from a specific table snapshot. + +Each index snapshot references a complete set of index files and contains all data from the referenced +table snapshot. + +## Index Metadata Review Comment: Do we have puffin support, partition and file pruning related metadata similar to iceberg tables. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
