pvary commented on code in PR #16961: URL: https://github.com/apache/iceberg/pull/16961#discussion_r3528486020
########## format/index.md: ########## @@ -0,0 +1,322 @@ +--- +title: "Index Spec" +--- +<!-- + - Licensed to the Apache Software Foundation (ASF) under one or more + - contributor license agreements. See the NOTICE file distributed with + - this work for additional information regarding copyright ownership. + - The ASF licenses this file to You under the Apache License, Version 2.0 + - (the "License"); you may not use this file except in compliance with + - the License. You may obtain a copy of the License at + - + - http://www.apache.org/licenses/LICENSE-2.0 + - + - Unless required by applicable law or agreed to in writing, software + - distributed under the License is distributed on an "AS IS" BASIS, + - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + - See the License for the specific language governing permissions and + - limitations under the License. + --> + +# Iceberg Index Specification + +## Background and Motivation + +Indexes enable query engines to locate relevant rows without scanning entire datasets. +They can accelerate point lookups, range predicates, and other retrieval patterns +while preserving Iceberg's table format, snapshot isolation, and interoperability. + +This specification defines: + +- A portable metadata format for indexes +- A catalog-managed index object independent from table metadata +- A common storage architecture for index data +- A framework for defining new index types and transform functions + +Indexes are optional. Engines may choose to create, maintain, consume, or ignore them. + +## Goals + +- Define a common metadata format for indexes +- Allow indexes to evolve independently from table metadata +- Enable index sharing across engines +- Provide a foundation for future index types + +## Definitions + +### Index Type + +The index type defines the logical category of an index and the class of queries it is designed to accelerate. + +The following index type is defined in this specification: + +| Type | +|--------| +| SCALAR | + +The following index types are reserved for future specifications: + +| Type | +|--------| +| VECTOR | +| TERM | + +The index type communicates the capabilities of an index to query engines and helps determine whether an index is +applicable to a particular query. + +### Index Transform Function + +The index transform function defines how the index organization key is derived from source table columns when rows are +stored in the index. + +The transform function determines the physical organization of the indexed data and therefore influences which query +patterns can efficiently leverage the index. + +The following index types are reserved for future specifications: + +| Transform | +|-----------| +| IDENTITY | +| HASH | +| HILBERT | Review Comment: I'm a bit reluctant to couple index transforms with partition transforms. For one, some transform functions (like Hilbert transform) we definitely want for indexes, but don't have a meaningful partition transform equivalent today, and it's not obvious that they should. Their purpose and requirements are fundamentally different. Looking ahead, this separation becomes even more important for more advanced index types such as IVF. An IVF index needs to assign vectors to centroids, but those centroids are derived from the table data itself and cannot be defined upfront at table creation time. Modeling this as a partition transform would therefore be awkward and unintuitive, as it introduces data-dependent behavior that is very different from the role of existing partition transforms. Overall, index transforms seem to serve a distinct purpose and have different requirements than partition transforms, so I'd prefer to keep the two concepts separate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
