One correction to the previous summary: I originally wrote that *Range* files would store the index data, but I was later corrected that the decision was actually to use *Region* files.
The correct index file naming is: - *Tracking file*: contains the list of region files (analogous to a manifest file for tables). - *Region file*: contains the index data rows (analogous to a data file for tables). Sorry for the confusion, Peter Péter Váry <[email protected]> ezt írta (időpont: 2026. szept. 3., Cs, 13:18): > > Hi everyone, > > Here is the recording of the Monday sync: > > https://apache-iceberg.slack.com/files/U06538UFDH6/F0BTT739Y7P/impromptu_google_meet_meeting_-_aug_31_2026.mp4 > > Thanks, Shawn, for making it available. > > A brief summary of the discussion: > > - We finalized the file naming: > - Tracking file: contains the list of range files (similar to a > manifest file for tables). > - Range file: contains the index data rows (similar to a data file > for tables). > - We agreed not to prescribe how expression values are materialized > (for example, Hilbert or hash transformations), leaving room for > implementation flexibility. > - Ryan mentioned that he is working on a v4 proposal for generating > Field IDs for non-materialized columns. The current thinking is that index > columns should follow the same approach. > - We agreed to retain older versions of the index metadata JSON. > - We agreed not to require a strict one-to-one mapping between index > snapshots and table snapshots. An index may contain multiple snapshots > corresponding to the same table snapshot. > > Based on the discussion, I have updated the spec PR. Following Ryan's > pattern for non-materialized column definitions, the proposal now > introduces: > > - *materialized-fields* for range file columns > - *non-materialized-fields* for tracking file statistics > - *cluster-spec* to define which columns participate in range file > clustering > > Overall, I think we reached good alignment on the remaining design points, > and I do not see any open questions at the moment. The PR is in good shape > for review, so I would greatly appreciate any feedback or comments. > > Thanks everyone for the productive discussion and continued collaboration. > Peter > > Péter Váry <[email protected]> ezt írta (időpont: 2026. aug. > 18., K, 14:17): > >> Hi everyone, >> >> Unfortunately, yesterday's sync was not recorded due to a technical >> issue. :( >> >> Here is a brief summary of the discussion and the decisions we reached: >> >> 1. We agreed to use a list of Iceberg expressions to define the >> *index_keys*, which determine how the index is ordered. >> 2. We agreed not to restrict the functions that can be used in those >> expressions. Engines that do not understand a particular expression can >> simply choose not to use the index. >> 3. We agreed that index region files (leaf files) should be sorted by >> the values generated by the index_keys. >> 4. We discussed using *included_column_ids* to define the contents >> and schema of the index region files, and generally felt this could be a >> good approach. >> 5. We started discussing whether the results of the index_key >> expressions should also be stored in the index region files alongside the >> included columns. In some cases, storing them could help readers optimize >> lookups within a region file. In other cases, the values could be derived >> when needed, making them redundant and potentially hurting performance by >> increasing the size of the region files. >> >> I think we should continue the discussion around point 5, as the outcome >> may influence, or even reopen, the decisions around point 4. >> >> Thanks! >> >> >> Flavio Junqueira <[email protected]> ezt írta (időpont: 2026. >> aug. 4., K, 18:57): >> >>> Thank you for the summary, the proposal and the overall discussion. I >>> have one main question about the target and I apologize in advance if I'm >>> missing some important detail as I might not have the full context. >>> >>> I noticed that partitioning, sorting and clustering (via Hilbert) were >>> discussed in the call in this context of index transform functions. My >>> understanding is that engines currently are already pruning partitions and >>> files using partition information and table metadata like per-column >>> min-max. If that's the case, then what's it that this proposal is trying to >>> achieve by having transform functions that capture partitioning, sorting >>> and clustering when engines are already able to prune without a secondary >>> index? I can more easily see the primary key index example, but I'm not >>> entirely sure about these data layout mechanisms. >>> >>> I also didn't quite get how engines are expected to consume such an >>> index. There was a discussion about where the mapping of the input to files >>> happens, whether that's in Iceberg or the engine, and I didn't quite get >>> the recommendation. Checking Peter's proposal, I couldn't see it clearly >>> either, and if I just missed it, I'd appreciate a pointer to where I should >>> look into. >>> >>> -Flavio >>> >>> On 4 Aug 2026, at 10:52, Péter Váry <[email protected]> wrote: >>> >>> Hi everyone, >>> >>> Here is the recording from yesterday’s sync: >>> https://www.youtube.com/watch?v=Wg9orP9JAmk >>> >>> To summarize: >>> - We agreed to retain the history of index snapshots, but not the >>> history of other index properties. If readers require information that may >>> later be modified by a user, that information should be copied into the >>> corresponding snapshot. >>> - We discussed the possibility of defining index ordering using a list >>> of transform functions. >>> >>> Dan and Yingyi suggested something along the following lines: >>> >>> *"index_keys" : [* >>> * {* >>> * "type": "apply",* >>> * "func-name": {* >>> * "catalog": "iceberg_functions",* >>> * "name": "day"* >>> * },* >>> * "arguments": [* >>> * { "type": "reference", "id": 3 }* >>> * ]* >>> * },* >>> * {* >>> * "type": "apply",* >>> * "func-name": {* >>> * "catalog": "iceberg_functions",* >>> * "name": "truncate"* >>> * },* >>> * "arguments": [* >>> * { "type": "reference", "id": 4 },* >>> * 2* >>> * ]* >>> * }* >>> *]* >>> >>> >>> In this example, rows would first be ordered by day(col3) and then by >>> truncate(col4, 2). To ensure a deterministic ordering, ties would be broken >>> using the original values of columns 3 and 4. If those are still equal, >>> ordering would fall back to the file name and finally the row position. >>> >>> I spent some time thinking about this after the sync, and here are my >>> thoughts. >>> >>> Pros: >>> - Provides a high degree of flexibility. >>> - Enables expression-based indexes. >>> >>> Cons: >>> - Do we need this flexibility at this stage? >>> - Equality deletes, the primary use cases we are targeting, do not >>> require expressions. >>> - Most indexes I have seen in production are not expression-based. >>> - Increased complexity for engines: >>> - Engines must parse and understand transforms to determine whether >>> using an index can improve a query. For example, is ordering by >>> day(timestamp) sufficient for a key lookup? >>> - Complexity increases further as new UDFs are introduced. >>> - Expressions are not pushed into scans today, so index handling >>> cannot be fully hidden behind Iceberg scan planning. >>> - Risk of fragmentation: >>> - The additional complexity may slow engine adoption. >>> - Adopters may choose to support only exact transform matches created >>> by them, rather than learn to reason about the general transforms. >>> - UDFs are prone to missing implementations and behavioral >>> differences across engines. In the worst case, such differences could lead >>> to incorrect query results when indexes are used. >>> >>> Given the points above, I still prefer defining HASH, HILBERT, IDENTITY >>> as a set of standardized values combined with an explicit list of key >>> columns. This keeps adoption simple, unlocks the primary use cases we are >>> targeting today, and allows the Iceberg library to provide the transform >>> implementations directly. >>> >>> Internally, these transforms could still be represented as Iceberg >>> expressions, giving us a migration path toward future index types that may >>> require full expression support. >>> >>> For example: >>> >>> *{* >>> * "transform": "HILBERT",* >>> * "key-column-ids": [3, 4]* >>> *}* >>> >>> >>> I'm interested in hearing other opinions. >>> >>> Thanks, >>> Peter >>> >>> Renjie Liu <[email protected]> ezt írta (időpont: 2026. júl. 28., >>> K, 10:07): >>> >>>> Hi, Peter: >>>> >>>> Thanks for driving the proposal. >>>> >>>> > *Hash Index Type*: Should hash-based indexes be a separate index >>>> type? They are only applicable to equality and IN predicates, and the hash >>>> algorithm must always be known. A dedicated HASH type could make these >>>> constraints explicit and distinguish hash-based indexes from ordered >>>> indexes. >>>> >>>> I'm leaning toward keeping them in scalar index type due to the >>>> similarity with others transformation functions like identity. As with the >>>> constraints, I think they are properties of transformation functions. >>>> >>>> > *Transform Function/Layout Representation*: Should transform >>>> functions be represented as an enum or a string? I lean toward strings, but >>>> I do not have a strong preference. In many cases, an index can still be >>>> used based on standard column statistics, even if a reader does not >>>> recognize a particular transform. >>>> >>>> This seems more like a problem in java implementation? >>>> >>>> Also I prefer the name transform function compared with layout >>>> representation. Layout representation makes me feel that it's not about the >>>> definition of an index function. >>>> >>>> > *Future Index Types*: How should we handle index types that are >>>> expected in the long term but fall outside the scope of this proposal (for >>>> example, VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved >>>> concepts, or omitted entirely? I lean toward at least mentioning them. The >>>> rationale for introducing an index type is easier to explain when >>>> considering future index families. >>>> >>>> +1 >>>> >>>> *> Index History Handling:* Most database systems treat index changes >>>> as a drop-and-recreate operation rather than an in-place update. If we >>>> follow the same approach, there is limited value in maintaining index >>>> metadata history. During index maintenance, a new metadata.json can be >>>> created and atomically swapped in, with the catalog responsible for >>>> cleaning up obsolete metadata and index data files according to its >>>> retention policy. >>>> >>>> +1, I think we only need to keep index data for different snapshots, >>>> but if an index definition changed, we should drop and recreate a new >>>> index. >>>> >>>> > *File Format Location*: Should file format be specified at the >>>> leaf-file level or only at the index level? My preference is to stay >>>> aligned with the table metadata model and avoid introducing differences >>>> unless there is a compelling requirement. This would simplify >>>> implementation and maximize code reuse. >>>> >>>> +1. >>>> >>>> >>>> >>>> On Fri, Jul 24, 2026 at 11:57 PM Péter Váry < >>>> [email protected]> wrote: >>>> >>>>> Hi everyone, >>>>> >>>>> Here are the key takeaways from Monday’s meeting (July 20): >>>>> >>>>> - *Index Type*: An index type enum will be introduced in v1, >>>>> initially with a single value (SCALAR). This establishes a >>>>> future-proof API >>>>> boundary while leaving room for additional index families. >>>>> - *Strict Total Ordering*: The specification should require >>>>> transform functions to produce values that preserve a well-defined >>>>> total >>>>> ordering across the key space. The transform function does not need to >>>>> return a long. Instead, leaf files can store the first and last values >>>>> according to the ordering, enabling generic filtering capabilities for >>>>> readers. >>>>> >>>>> Open Questions: >>>>> >>>>> - *Hash Index Type*: Should hash-based indexes be a separate index >>>>> type? They are only applicable to equality and IN predicates, and the >>>>> hash >>>>> algorithm must always be known. A dedicated HASH type could make these >>>>> constraints explicit and distinguish hash-based indexes from ordered >>>>> indexes. >>>>> - *Transform Function/Layout Representation*: Should transform >>>>> functions be represented as an enum or a string? I lean toward >>>>> strings, but >>>>> I do not have a strong preference. In many cases, an index can still be >>>>> used based on standard column statistics, even if a reader does not >>>>> recognize a particular transform. >>>>> - *Future Index Types*: How should we handle index types that are >>>>> expected in the long term but fall outside the scope of this proposal >>>>> (for >>>>> example, VECTOR, IVF, TERM)? Should they be mentioned as >>>>> planned/reserved >>>>> concepts, or omitted entirely? I lean toward at least mentioning them. >>>>> The >>>>> rationale for introducing an index type is easier to explain when >>>>> considering future index families. >>>>> - *Index History Handling:* Most database systems treat index >>>>> changes as a drop-and-recreate operation rather than an in-place >>>>> update. If >>>>> we follow the same approach, there is limited value in maintaining >>>>> index >>>>> metadata history. During index maintenance, a new metadata.json can be >>>>> created and atomically swapped in, with the catalog responsible for >>>>> cleaning up obsolete metadata and index data files according to its >>>>> retention policy. >>>>> - *Min/Max Statistics Representation*: Should transform-specific >>>>> or sort-order min/max values be represented using the existing file >>>>> statistics structure, or exposed through dedicated fields? My >>>>> preference is >>>>> to keep them within the statistics structure defined by the file schema >>>>> while exposing them directly through the Java API if needed. >>>>> - *File Format Location*: Should file format be specified at the >>>>> leaf-file level or only at the index level? My preference is to stay >>>>> aligned with the table metadata model and avoid introducing differences >>>>> unless there is a compelling requirement. This would simplify >>>>> implementation and maximize code reuse. >>>>> >>>>> I have updated the spec PR based on the decisions above: >>>>> https://github.com/apache/iceberg/pull/16961 >>>>> >>>>> Thanks, >>>>> Peter >>>>> >>>>> huaxin gao <[email protected]> ezt írta (időpont: 2026. jún. >>>>> 26., P, 19:42): >>>>> >>>>>> Hi all, Here is the summary of this Monday's index meeting: >>>>>> >>>>>> Two blocking decisions closed: >>>>>> >>>>>> 1. Index is *not* a table— it's its own object (reuses table >>>>>> machinery under the hood). Reasoning: requires a sort order, no column >>>>>> updates, no partition spec, no overlapping ranges between leaves, >>>>>> inherits >>>>>> base-table permissions, and has its own CREATE/DROP/UPDATE INDEX DDL. >>>>>> 2. Index is a separate catalog entity, with no pointers in table >>>>>> metadata— has its own REST endpoints; the catalog can optionally >>>>>> return >>>>>> index metadata withloadTable to avoid extra round-trips. Keeps table >>>>>> and >>>>>> index updates independent/async. >>>>>> >>>>>> >>>>>> Next steps: Start writing the spec and build out the copy-on-write >>>>>> path now. >>>>>> >>>>>> Here are the draft spec: >>>>>> secondary index spec <https://github.com/apache/iceberg/pull/16961> >>>>>> irc spec <https://github.com/apache/iceberg/pull/16963> >>>>>> >>>>>> Thanks, >>>>>> Huaxin >>>>>> >>>>>> On Sat, Jun 20, 2026 at 11:11 AM huaxin gao <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I built a standalone PoC to validate that the basic index structure >>>>>>> works: that we can build a PK index, convert equality deletes to >>>>>>> position >>>>>>> deletes through it, and have every converted delete land on the correct >>>>>>> live row. I ran it up to *100M keys*. >>>>>>> >>>>>>> *Headline: the structure works.* The index builds over up to 100M >>>>>>> keys, the eq-delete → position-delete conversion resolved correctly at >>>>>>> *every* size (100% of converted deletes mapped to the right live >>>>>>> row), and the resulting position deletes are *~8× cheaper to apply* at >>>>>>> query time than the equality deletes they replace. >>>>>>> >>>>>>> Beyond correctness, the run also shows how the index’s *maintenance* >>>>>>> cost >>>>>>> scales, comparing copy-on-write (COW, rewrite touched leaves) vs an >>>>>>> append/merge (MOR) option, under a realistic mixed CDC checkpoint (1,000 >>>>>>> insert + 500 update + 500 delete), local wall-clock: >>>>>>> keys EQ baseline INDEX (COW) % of 60s (COW) INDEX (MOR) % of 60s >>>>>>> (MOR) correct >>>>>>> 5M 6 ms 6.7s 11.2% 2.2s 3.7% PASS >>>>>>> 20M 8 ms 24.2s 40.4% 6.4s 10.6% PASS >>>>>>> 50M 7 ms 51.6s 86.1% 12.2s 20.4% PASS >>>>>>> *100M* 6 ms *75.0s* 125% (BEHIND) *16.9s* 28.2% (keeps up) PASS >>>>>>> >>>>>>> COW maintenance crosses the 60 s checkpoint around 100M (75 s/cycle, >>>>>>> 125%); MOR stays at ~28% and keeps pace; the equality-delete baseline is >>>>>>> ~6 ms and flat. So the structure works, but *COW alone can’t >>>>>>> sustain scattered CDC at hundreds of millions of keys on a single >>>>>>> writer*. >>>>>>> It’s worth allowing a merge-on-read / update-file maintenance option >>>>>>> alongside COW (or sharding the index across parallel writers). >>>>>>> >>>>>>> *Full write-up, all tables, and the in-region reality-check:* link >>>>>>> <https://docs.google.com/document/d/1G3zxbW8X0eU3UrouslZfp42bBc9CvgJGnJyDONCB4PU/edit?tab=t.0> >>>>>>> >>>>>>> Feedback welcome, especially on the spec direction (whether to allow >>>>>>> a merge-on-read / update-file maintenance option alongside COW) and on >>>>>>> the >>>>>>> read-side modeling. >>>>>>> >>>>>>> Thanks, >>>>>>> Huaxin >>>>>>> >>>>>>> On Tue, Jun 9, 2026 at 5:45 PM huaxin gao <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Sorry, we've skipped posting a few of the dedicated index-sync >>>>>>>> summaries to the mailing list; you can find those in the Google doc >>>>>>>> <https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?pli=1&tab=t.8041k7j2n7y3> >>>>>>>> and the Slack channel. Here's yesterday's summary: >>>>>>>> >>>>>>>> *Decided* >>>>>>>> >>>>>>>> - >>>>>>>> >>>>>>>> Index vs. table (what we agreed): >>>>>>>> - Reuse table implementation/library code and a near-identical >>>>>>>> spec — the commit path will be custom regardless, so reuse isn't >>>>>>>> the >>>>>>>> deciding factor. >>>>>>>> - An index is not a table from a user/API view: loading or >>>>>>>> writing an index as a table must fail(it would violate index >>>>>>>> invariants). >>>>>>>> - The spec forbids most table behaviors: no overlapping >>>>>>>> files, one mandatory transform sort order, no column updates, no >>>>>>>> partition >>>>>>>> spec. >>>>>>>> - Delete vectors: reuse Iceberg's existing DV — benchmarks >>>>>>>> showed no new delete format is worth introducing. >>>>>>>> - Incremental updates: start with copy-on-write only (no update >>>>>>>> files). For object-store-sized leaves, a full leaf rewrite is about >>>>>>>> as >>>>>>>> cheap as maintaining an overlay update file + DV, so we'll skip the >>>>>>>> MOR >>>>>>>> machinery for now and add it later only if benchmarks prove we need >>>>>>>> it >>>>>>>> (likely just the very-large-leaf case). >>>>>>>> - >>>>>>>> >>>>>>>> Validate the spec first: build a quick, hand-wired prototype >>>>>>>> (Parquet files structured per the spec) and benchmark it on real >>>>>>>> scales >>>>>>>> before formalizing. >>>>>>>> >>>>>>>> *Leaning, not final* >>>>>>>> >>>>>>>> - >>>>>>>> >>>>>>>> Indexes are likely separate catalog objects, linked from the >>>>>>>> table by storing just an identifier (like materialized views) and >>>>>>>> not >>>>>>>> visible in LIST TABLES. >>>>>>>> - >>>>>>>> >>>>>>>> We'll need a commit path for indexes, but simpler than tables >>>>>>>> (no stage-create). >>>>>>>> >>>>>>>> *Still open* >>>>>>>> >>>>>>>> >>>>>>>> - >>>>>>>> >>>>>>>> Permissions model — separate vs. inherited (action: look at >>>>>>>> what real DBs do for index permissions). >>>>>>>> - REST/catalog RPC design — minimize round-trips; index >>>>>>>> metadata ideally returned with LOAD TABLE. Catalog RPC cost may >>>>>>>> dominate Parquet IO, so this needs real design. >>>>>>>> - >>>>>>>> >>>>>>>> Scale modeling — target rows-per-leaf vs. leaf size vs. >>>>>>>> metadata-file count. >>>>>>>> - >>>>>>>> >>>>>>>> DDL-on-index semantics (reuse table schema-update actions or >>>>>>>> separate) >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Huaxin >>>>>>>> >>>>>>>> On Wed, Apr 22, 2026 at 8:47 AM Péter Váry < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> TL;DR >>>>>>>>> We still need to validate with ADLS and S3, but based on the local >>>>>>>>> tests, the MPHF approach looks more promising if we can tolerate >>>>>>>>> larger >>>>>>>>> files and longer index maintenance times. >>>>>>>>> >>>>>>>>> Details: >>>>>>>>> Here are the results from the local experiments on my Mac. I >>>>>>>>> removed unnecessary statistics from the Parquet files and tested >>>>>>>>> different >>>>>>>>> row group sizes: >>>>>>>>> >>>>>>>>> - For an index file with 1M records, a row group size of 5,000 >>>>>>>>> appears to be the sweet spot. >>>>>>>>> - For 10M records, 10,000 rows per row group works best. >>>>>>>>> >>>>>>>>> If you have additional ideas for optimizing Parquet-based indexes, >>>>>>>>> I’d be very interested to hear them. >>>>>>>>> The test code is available on this branch: >>>>>>>>> https://github.com/pvary/iceberg/tree/leaf_bench >>>>>>>>> >>>>>>>>> Best results: >>>>>>>>> *1m records/file* >>>>>>>>> >>>>>>>>> - Parquet - 5000 row/RowGroup >>>>>>>>> - Read: 1191 µs - 1 file open, 3 seek, 123KB read per lookup >>>>>>>>> - Write: 1.7 s, 15 MB >>>>>>>>> - MPHF >>>>>>>>> - Read: 202 µs - 1 file open, 1 seek, 282KB read per lookup >>>>>>>>> - Write: 0.8 s, 34 MB >>>>>>>>> >>>>>>>>> *10m records/file* >>>>>>>>> >>>>>>>>> - Parquet - 10000 row/RowGroup >>>>>>>>> - Read: 4168 µs - 1 file open, 3 seek, 395KB read per lookup >>>>>>>>> - Write: 19.5s s, 144 MB >>>>>>>>> - MPHF >>>>>>>>> - Read: 1086 µs - 1 file open, 1 seek, 2.8 MB (2812KB) >>>>>>>>> read per lookup >>>>>>>>> - Write: 6.5 s, 34 MB, 353 MB >>>>>>>>> >>>>>>>>> Below are the full results. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Benchmark (indexType) >>>>>>>>> (keyType) (numRows) Mode Cnt Score Error >>>>>>>>> UnitsInvertedIndexBenchmark.lookup PARQUET_1000 >>>>>>>>> LONG >>>>>>>>> 1000000 ss 10000 3285.284 ± 5.138 >>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_1000 >>>>>>>>> LONG >>>>>>>>> 1000000 ss 10000 2522168989.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_1000 LONG >>>>>>>>> 1000000 ss 10000 10000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_1000 LONG >>>>>>>>> 1000000 ss 10000 30000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup PARQUET_1000 LONG >>>>>>>>> 10000000 ss 10000 35449.614 ± 34.673 >>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_1000 >>>>>>>>> LONG >>>>>>>>> 10000000 ss 10000 24302649201.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_1000 LONG >>>>>>>>> 10000000 ss 10000 10000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_1000 LONG >>>>>>>>> 10000000 ss 10000 30000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup PARQUET_5000 LONG >>>>>>>>> 1000000 ss 10000 1191.959 ± 4.169 >>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_5000 >>>>>>>>> LONG >>>>>>>>> 1000000 ss 10000 1230877229.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_5000 LONG >>>>>>>>> 1000000 ss 10000 10000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_5000 LONG >>>>>>>>> 1000000 ss 10000 30000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup PARQUET_5000 LONG >>>>>>>>> 10000000 ss 10000 7236.447 ± 10.374 >>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_5000 >>>>>>>>> LONG >>>>>>>>> 10000000 ss 10000 5650715973.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_5000 LONG >>>>>>>>> 10000000 ss 10000 10000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_5000 LONG >>>>>>>>> 10000000 ss 10000 30000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup PARQUET_10000 LONG >>>>>>>>> 1000000 ss 10000 1349.946 ± 7.834 >>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_10000 >>>>>>>>> LONG >>>>>>>>> 1000000 ss 10000 1730219377.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_10000 LONG >>>>>>>>> 1000000 ss 10000 10000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_10000 LONG >>>>>>>>> 1000000 ss 10000 30000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup PARQUET_10000 LONG >>>>>>>>> 10000000 ss 10000 4168.635 ± 11.051 >>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_10000 >>>>>>>>> LONG >>>>>>>>> 10000000 ss 10000 3946341532.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_10000 LONG >>>>>>>>> 10000000 ss 10000 10000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_10000 LONG >>>>>>>>> 10000000 ss 10000 30000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup PARQUET_50000 LONG >>>>>>>>> 1000000 ss 10000 4736.466 ± 38.179 >>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_50000 >>>>>>>>> LONG >>>>>>>>> 1000000 ss 10000 7427413541.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_50000 LONG >>>>>>>>> 1000000 ss 10000 10000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_50000 LONG >>>>>>>>> 1000000 ss 10000 30000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup PARQUET_50000 LONG >>>>>>>>> 10000000 ss 10000 4979.031 ± 34.708 >>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_50000 >>>>>>>>> LONG >>>>>>>>> 10000000 ss 10000 7694887636.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_50000 LONG >>>>>>>>> 10000000 ss 10000 10000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_50000 LONG >>>>>>>>> 10000000 ss 10000 30000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup MPHF LONG >>>>>>>>> 1000000 ss 10000 202.571 ± 2.336 >>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead MPHF >>>>>>>>> LONG >>>>>>>>> 1000000 ss 10000 2821570000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams MPHF LONG >>>>>>>>> 1000000 ss 10000 10000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:seeks MPHF LONG >>>>>>>>> 1000000 ss 10000 10000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup MPHF LONG >>>>>>>>> 10000000 ss 10000 1086.957 ± 4.524 >>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead MPHF >>>>>>>>> LONG >>>>>>>>> 10000000 ss 10000 28119460000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams MPHF LONG >>>>>>>>> 10000000 ss 10000 10000.000 >>>>>>>>> #InvertedIndexBenchmark.lookup:seeks MPHF LONG >>>>>>>>> 10000000 ss 10000 10000.000 >>>>>>>>> #InvertedIndexBenchmark.write PARQUET_1000 LONG >>>>>>>>> 1000000 ss 3 1720731.014 ± 876636.004 >>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_1000 >>>>>>>>> LONG >>>>>>>>> 1000000 ss 3 46453317.000 >>>>>>>>> #InvertedIndexBenchmark.write PARQUET_1000 LONG >>>>>>>>> 10000000 ss 3 18547947.876 ± 12258125.307 >>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_1000 >>>>>>>>> LONG >>>>>>>>> 10000000 ss 3 452655675.000 >>>>>>>>> #InvertedIndexBenchmark.write PARQUET_5000 LONG >>>>>>>>> 1000000 ss 3 1718345.583 ± 1103928.016 >>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_5000 >>>>>>>>> LONG >>>>>>>>> 1000000 ss 3 44845788.000 >>>>>>>>> #InvertedIndexBenchmark.write PARQUET_5000 LONG >>>>>>>>> 10000000 ss 3 18604229.931 ± 2668361.915 >>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_5000 >>>>>>>>> LONG >>>>>>>>> 10000000 ss 3 435388818.000 >>>>>>>>> #InvertedIndexBenchmark.write PARQUET_10000 LONG >>>>>>>>> 1000000 ss 3 1761555.389 ± 535857.675 >>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_10000 >>>>>>>>> LONG >>>>>>>>> 1000000 ss 3 44536635.000 >>>>>>>>> #InvertedIndexBenchmark.write PARQUET_10000 LONG >>>>>>>>> 10000000 ss 3 19501588.264 ± 2130054.558 >>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_10000 >>>>>>>>> LONG >>>>>>>>> 10000000 ss 3 433189623.000 >>>>>>>>> #InvertedIndexBenchmark.write PARQUET_50000 LONG >>>>>>>>> 1000000 ss 3 1936624.889 ± 6601363.985 >>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_50000 >>>>>>>>> LONG >>>>>>>>> 1000000 ss 3 44264655.000 >>>>>>>>> #InvertedIndexBenchmark.write PARQUET_50000 LONG >>>>>>>>> 10000000 ss 3 20471742.278 ± 10705206.310 >>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_50000 >>>>>>>>> LONG >>>>>>>>> 10000000 ss 3 431311305.000 >>>>>>>>> #InvertedIndexBenchmark.write MPHF LONG >>>>>>>>> 1000000 ss 3 896573.958 ± 1408024.851 >>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes MPHF >>>>>>>>> LONG >>>>>>>>> 1000000 ss 3 102846369.000 >>>>>>>>> #InvertedIndexBenchmark.write MPHF LONG >>>>>>>>> 10000000 ss 3 6509348.875 ± 15519975.479 >>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes MPHF >>>>>>>>> LONG >>>>>>>>> 10000000 ss 3 1058435733.000 #* >>>>>>>>> >>>>>>>>> huaxin gao <[email protected]> ezt írta (időpont: 2026. ápr. >>>>>>>>> 21., K, 20:53): >>>>>>>>> >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> In recent secondary index sync meetings, the discussion converged >>>>>>>>>> on the need to define what an index is from first principles before >>>>>>>>>> settling on physical layout. >>>>>>>>>> >>>>>>>>>> To address that, Peter and I have drafted a requirements document >>>>>>>>>> for a key lookup index (renamed from "primary key index" to avoid >>>>>>>>>> implying >>>>>>>>>> uniqueness enforcement), the goal is to nail down one well-scoped >>>>>>>>>> index >>>>>>>>>> type first. >>>>>>>>>> >>>>>>>>>> Doc: Key Lookup Index Requirements >>>>>>>>>> <https://docs.google.com/document/d/1e0zxK-jA0LBDq8YQlQgFipTHelDFiga8lCkgDTmYub8/edit?tab=t.0#heading=h.8shrgabvl19> >>>>>>>>>> >>>>>>>>>> It covers requirements, three design options (manifest + sorted >>>>>>>>>> Parquet, hash + sorted Parquet, hash + MPHF) and open questions. We >>>>>>>>>> will >>>>>>>>>> add preliminary benchmark results shortly. >>>>>>>>>> >>>>>>>>>> Feedback welcome — inline in the doc, on this thread, or at the >>>>>>>>>> next index sync. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Huaxin >>>>>>>>>> >>>>>>>>>> On Mon, Apr 13, 2026 at 7:22 AM Steven Wu <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Do we need the special index identifier that was originally >>>>>>>>>>> proposed? A generic CatalogObjectIdentifier (with namespace and >>>>>>>>>>> name) would >>>>>>>>>>> be consistent with all object types in the catalog. I have a >>>>>>>>>>> discussion >>>>>>>>>>> thread on the generic identifier topic: [DISCUSS] REST Spec: >>>>>>>>>>> generic CatalogObjectIdentifier. >>>>>>>>>>> >>>>>>>>>>> Should we add an indexes array field to table metadata? It only >>>>>>>>>>> contains a list of index object identifiers. It doesn't contain any >>>>>>>>>>> index >>>>>>>>>>> metadata which should live in the index objects. Yufei was trying >>>>>>>>>>> to bring >>>>>>>>>>> this up at the end of the first sync. But we didn't get enough time >>>>>>>>>>> to >>>>>>>>>>> really discuss it. It will be great to discuss this as the first >>>>>>>>>>> agenda >>>>>>>>>>> item today. >>>>>>>>>>> >>>>>>>>>>> On Mon, Apr 13, 2026 at 3:17 AM Péter Váry < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi everyone, >>>>>>>>>>>> >>>>>>>>>>>> We had several engaging discussions at the Iceberg Summit, and >>>>>>>>>>>> it was great to finally catch up with many of you in person. We >>>>>>>>>>>> truly >>>>>>>>>>>> missed those who couldn’t attend, hopefully we’ll all meet again >>>>>>>>>>>> at the >>>>>>>>>>>> next summit. >>>>>>>>>>>> >>>>>>>>>>>> To keep the conversation going, Huaxin and I have put together >>>>>>>>>>>> the agenda for our next meeting. As a reminder, we’ll meet on >>>>>>>>>>>> *April >>>>>>>>>>>> 13th, 9:00–10:00 AM *PDT (6:00–7:00 PM CEST). >>>>>>>>>>>> >>>>>>>>>>>> Proposed agenda: >>>>>>>>>>>> >>>>>>>>>>>> - Continue first-principles index design discussion from >>>>>>>>>>>> Mar 30 >>>>>>>>>>>> - *Index Ownership and Write Responsibility* >>>>>>>>>>>> - Should writers be allowed to update indexes, or >>>>>>>>>>>> - Should all index writes be handled exclusively by >>>>>>>>>>>> the Index Maintenance process? >>>>>>>>>>>> - If writers can update indexes then we need to >>>>>>>>>>>> define what guarantees are required (compaction, file >>>>>>>>>>>> splitting, layout >>>>>>>>>>>> expectations)? >>>>>>>>>>>> - If only Index Maintenance updates indexes then we >>>>>>>>>>>> only need to define what observable properties should be >>>>>>>>>>>> exposed to >>>>>>>>>>>> consumers? Like: >>>>>>>>>>>> - Expected max files for a single key >>>>>>>>>>>> - Current max files for a single key >>>>>>>>>>>> - Deletes allowed/present >>>>>>>>>>>> - Sorted by >>>>>>>>>>>> - Partitioned by >>>>>>>>>>>> - *Specification Scope: What Belongs in the Spec?* >>>>>>>>>>>> - Related to the ownership question above >>>>>>>>>>>> - Light spec: Just define that the index table should >>>>>>>>>>>> be optimized for retrieval by key columns and the index >>>>>>>>>>>> columns should be >>>>>>>>>>>> contained in the table. This could give us more >>>>>>>>>>>> flexibility if better >>>>>>>>>>>> organization methods come up, or >>>>>>>>>>>> - Detailed spec: We could define the max number of >>>>>>>>>>>> files per index to read for a single key, or even the >>>>>>>>>>>> partitioning and the >>>>>>>>>>>> exact sort order. This could allow more use-cases for a >>>>>>>>>>>> given index, like >>>>>>>>>>>> joins or cardinality estimations. >>>>>>>>>>>> - I would go for light spec for the main types (PK, >>>>>>>>>>>> Containing) and only the Index Maintenance processes >>>>>>>>>>>> should update the >>>>>>>>>>>> Indexes, as for many use-cases the details are not >>>>>>>>>>>> important, and writers >>>>>>>>>>>> will very rarely update the Indexes themselves. >>>>>>>>>>>> - *Logical Placement of Indexes* >>>>>>>>>>>> - Index as a child object of an Iceberg Table, or >>>>>>>>>>>> - Index as a first‑class entity under >>>>>>>>>>>> /namespace/indexes/{index} >>>>>>>>>>>> - Based on the discussions on the summit we are >>>>>>>>>>>> leaning in this direction. This means the index id should >>>>>>>>>>>> be unique in the >>>>>>>>>>>> namespace but helps the catalog implementations quite a >>>>>>>>>>>> bit >>>>>>>>>>>> - *Physical Placement of Index Data* >>>>>>>>>>>> - I don’t think we should specify this. We should >>>>>>>>>>>> have a base location for the index, but can rely on the >>>>>>>>>>>> catalog >>>>>>>>>>>> implementations to decide on their own, like they do with >>>>>>>>>>>> the tables, >>>>>>>>>>>> views, udfs. >>>>>>>>>>>> - *Iceberg Reader Based indexes* (Containing indexes and >>>>>>>>>>>> potentially PK indexes). These are the indexes which could >>>>>>>>>>>> be read by the >>>>>>>>>>>> existing Iceberg readers. We might decide to store the PK >>>>>>>>>>>> index similarly >>>>>>>>>>>> to an Iceberg Table and treat it as a reader based index. >>>>>>>>>>>> - What are the table properties/features exposed to >>>>>>>>>>>> the readers >>>>>>>>>>>> - Maybe just some behavioral descriptors for the >>>>>>>>>>>> optimizer to decide if the index could be used or >>>>>>>>>>>> should be skipped, like: >>>>>>>>>>>> - Expected max files for a single key >>>>>>>>>>>> - max files for a single key >>>>>>>>>>>> - Deletes allowed/present >>>>>>>>>>>> - Sorted by >>>>>>>>>>>> - Partitioned by >>>>>>>>>>>> - The Tasks when reading the index based on the >>>>>>>>>>>> filters and projection >>>>>>>>>>>> - What are the table properties/features exposed to >>>>>>>>>>>> the Index Maintenance. I think this could be internal to >>>>>>>>>>>> the Index >>>>>>>>>>>> Maintenance process and might not be exposed through the >>>>>>>>>>>> spec. The Index >>>>>>>>>>>> Maintenance process could handle this as a standard >>>>>>>>>>>> Iceberg Table and could >>>>>>>>>>>> be based on the Table Maintenance process, but there >>>>>>>>>>>> might be some totally >>>>>>>>>>>> different processes. >>>>>>>>>>>> - It should be possible to add properties to an index >>>>>>>>>>>> defined by the Index Maintenance process which could be used >>>>>>>>>>>> and updated in >>>>>>>>>>>> the next Index Maintenance run. >>>>>>>>>>>> - *PK index storage format benchmark results* >>>>>>>>>>>> - Flat Parquet (baseline) >>>>>>>>>>>> - BTree with Parquet leaves >>>>>>>>>>>> - Vortex >>>>>>>>>>>> - *Open items / next steps* >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Peter >>>>>>>>>>>> >>>>>>>>>>>> huaxin gao <[email protected]> ezt írta (időpont: 2026. >>>>>>>>>>>> márc. 23., H, 3:03): >>>>>>>>>>>> >>>>>>>>>>>>> Hi everyone, I wanted to share an update on the primary key >>>>>>>>>>>>> index work. >>>>>>>>>>>>> Since there are still open questions on whether bloom filter >>>>>>>>>>>>> indexes fit in the secondary index framework or should be treated >>>>>>>>>>>>> as >>>>>>>>>>>>> extended stats, I've shifted focus to the primary key index since >>>>>>>>>>>>> it's a >>>>>>>>>>>>> clearer fit for the framework. >>>>>>>>>>>>> I've put together a proposal for a primary key reverse-lookup >>>>>>>>>>>>> index that maps each key to its physical location (file_path, >>>>>>>>>>>>> row_position). It enables: >>>>>>>>>>>>> >>>>>>>>>>>>> - Scan-time file pruning for point lookups >>>>>>>>>>>>> - Converting key-based deletes into position deletes >>>>>>>>>>>>> (eliminating equality deletes for Flink CDC) >>>>>>>>>>>>> - Accelerating Spark MERGE INTO by replacing full-table >>>>>>>>>>>>> joins with direct file lookups >>>>>>>>>>>>> >>>>>>>>>>>>> Proposal: >>>>>>>>>>>>> https://docs.google.com/document/d/1HuhCZ0n2FqDh8yqQb9oEj1CPM5yXpEsMPGZno2aSf8E/edit?tab=t.0#heading=h.tbevg4q0m9 >>>>>>>>>>>>> Feedback welcome! >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Huaxin >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Mar 18, 2026 at 11:42 PM Péter Váry < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Key takeaways from the general index discussion at the May 16 >>>>>>>>>>>>>> meeting. >>>>>>>>>>>>>> Thanks to everyone who participated! The recording is >>>>>>>>>>>>>> available here: https://www.youtube.com/watch?v=btmjhtRWUCE >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Q: Do we need to tie index types to the algorithms used >>>>>>>>>>>>>> to access them? >>>>>>>>>>>>>> - A: From a specification perspective, the goal is to >>>>>>>>>>>>>> define the storage-level data layout so it can be shared >>>>>>>>>>>>>> across engines. >>>>>>>>>>>>>> Engines are free to interpret and use the data as they see >>>>>>>>>>>>>> fit, but the >>>>>>>>>>>>>> on-disk data layout itself must be strictly defined and >>>>>>>>>>>>>> interoperable. >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Q: Should we introduce an additional abstraction layer >>>>>>>>>>>>>> (e.g., Vector Index) with sub-types such as IVF and DiskANN? >>>>>>>>>>>>>> - A: This is possible if we decide it is beneficial. I >>>>>>>>>>>>>> explored potential naming, but it is not yet clear how such a >>>>>>>>>>>>>> layer would >>>>>>>>>>>>>> be used in practice. >>>>>>>>>>>>>> *Question to Yingyi Bu*: could you provide examples where >>>>>>>>>>>>>> this additional layer would be useful? Should this >>>>>>>>>>>>>> abstraction be defined >>>>>>>>>>>>>> at the spec level, or is it better handled at the engine >>>>>>>>>>>>>> level? >>>>>>>>>>>>>> My initial idea was that users would create a generic >>>>>>>>>>>>>> Vector Index and let the engine choose the concrete >>>>>>>>>>>>>> implementation. >>>>>>>>>>>>>> However, this would limit user control and users likely need >>>>>>>>>>>>>> to specify the >>>>>>>>>>>>>> exact index representation, which implies they must be aware >>>>>>>>>>>>>> of the >>>>>>>>>>>>>> available representations. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Q: Do we want to allow extensibility for index types? >>>>>>>>>>>>>> - A: Yes. The intent is to support a small set of >>>>>>>>>>>>>> well-defined index types while allowing experimentation with >>>>>>>>>>>>>> new ones. If a >>>>>>>>>>>>>> new index type proves broadly useful, a follow-up proposal >>>>>>>>>>>>>> can standardize >>>>>>>>>>>>>> it and incorporate it into the spec. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Q: Do we allow multiple versions of an index for the >>>>>>>>>>>>>> same table snapshot? >>>>>>>>>>>>>> - A: Yes. Older index versions must be retained for >>>>>>>>>>>>>> readers that have already started using them, while new >>>>>>>>>>>>>> readers should >>>>>>>>>>>>>> automatically use the latest available version >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Q: Do we need to use materialized views for these >>>>>>>>>>>>>> indexes? >>>>>>>>>>>>>> - A: No. These indexes are primarily examples, and >>>>>>>>>>>>>> different types may require different storage methods. >>>>>>>>>>>>>> However, the Primary >>>>>>>>>>>>>> Key, Containing, and parts of the IVF indexes can be >>>>>>>>>>>>>> structured as Iceberg >>>>>>>>>>>>>> tables. This allows engines to read them natively; in some >>>>>>>>>>>>>> cases, Iceberg >>>>>>>>>>>>>> planners can automatically redirect queries to the index >>>>>>>>>>>>>> table without >>>>>>>>>>>>>> engine modifications. Furthermore, index maintenance for >>>>>>>>>>>>>> these tables can >>>>>>>>>>>>>> leverage existing materialized view maintenance workflows. >>>>>>>>>>>>>> Other index >>>>>>>>>>>>>> types may instead rely on Puffin files or alternative storage >>>>>>>>>>>>>> approaches. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Q: How should index metadata be accessed? Should we add >>>>>>>>>>>>>> explicit pointers for the indexes in the table metadata? >>>>>>>>>>>>>> - A: We did not have sufficient time to fully explore and >>>>>>>>>>>>>> conclude this topic. >>>>>>>>>>>>>> *Question for Yufei Gu*: Did I understand correctly that >>>>>>>>>>>>>> your main concern stems from endpoint resolution from a REST >>>>>>>>>>>>>> Catalog >>>>>>>>>>>>>> perspective? Specifically, if indexes are exposed under a URI >>>>>>>>>>>>>> such as >>>>>>>>>>>>>> >>>>>>>>>>>>>> v1/{prefix}/namespaces/{namespace}/tables/{table}/indexes/{index}, >>>>>>>>>>>>>> would >>>>>>>>>>>>>> this make it more difficult for the REST Catalog to resolve >>>>>>>>>>>>>> and route >>>>>>>>>>>>>> requests to the appropriate endpoint? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Suhas Jayaram Subramanya via dev <[email protected]> >>>>>>>>>>>>>> ezt írta (időpont: 2026. márc. 13., P, 23:32): >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi everyone, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Here's a proposal for native Vector Index support in Iceberg >>>>>>>>>>>>>>> tables -- >>>>>>>>>>>>>>> https://docs.google.com/document/d/1KL4qLOwdqnhOcqTc0EjO1O16NV3M3c-gZCEINDWw4lA/edit?usp=sharing >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We've been working on this proposal with Peter internally at >>>>>>>>>>>>>>> Microsoft and he suggested we post it here to bring this to the >>>>>>>>>>>>>>> community's >>>>>>>>>>>>>>> attention, ahead of the next Secondary Index Sync. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Suhas >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 2026/02/19 04:34:34 huaxin gao wrote: >>>>>>>>>>>>>>> > Hi Everyone, >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Here are the recording and notes from the Iceberg Index >>>>>>>>>>>>>>> Support Sync on >>>>>>>>>>>>>>> > 2/11. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Recording: https://www.youtube.com/watch?v=3sFfQ0A50yk >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Notes: >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.8041k7j2n7y3 >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > The meeting will move to biweekly, Mondays 9–10am PST, >>>>>>>>>>>>>>> starting March 2. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Since the sync, I updated the Bloom skipping index proposal >>>>>>>>>>>>>>> > < >>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.5r5kl6k3fqwu >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > to address the discussion questions, specifically: >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > - Performance justification: when this helps >>>>>>>>>>>>>>> (high-cardinality = / IN, >>>>>>>>>>>>>>> > many data files, high object-store latency) and how it >>>>>>>>>>>>>>> differs from Parquet >>>>>>>>>>>>>>> > row-group Bloom filters (which still require opening the >>>>>>>>>>>>>>> data file). >>>>>>>>>>>>>>> > - Cost / scalability: rough sizing (Bloom blob size per >>>>>>>>>>>>>>> file, Puffin >>>>>>>>>>>>>>> > file size), the planning cost trade-off (driver index >>>>>>>>>>>>>>> reads vs executor >>>>>>>>>>>>>>> > file opens), and mitigations via caching. >>>>>>>>>>>>>>> > - Lifecycle / maintenance: incremental production as new >>>>>>>>>>>>>>> data files >>>>>>>>>>>>>>> > arrive, behavior when the index is missing/behind, and >>>>>>>>>>>>>>> sharding/compaction >>>>>>>>>>>>>>> > plus cleanup to avoid accumulating too many small Puffin >>>>>>>>>>>>>>> files over time. >>>>>>>>>>>>>>> > - Writer expectations: inline (optional) vs asynchronous >>>>>>>>>>>>>>> (primary) index >>>>>>>>>>>>>>> > creation. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > I also implemented a Spark 4.1 POC >>>>>>>>>>>>>>> > <https://github.com/apache/iceberg/pull/15311> and a >>>>>>>>>>>>>>> local benchmark to >>>>>>>>>>>>>>> > quantify both the pruning impact (plannedFiles → >>>>>>>>>>>>>>> afterBloom) and the index >>>>>>>>>>>>>>> > read overhead (statsFiles, statsBytes, bloomPayloadBytes) >>>>>>>>>>>>>>> for point >>>>>>>>>>>>>>> > predicates on high-cardinality columns. Please take a look >>>>>>>>>>>>>>> and let me know >>>>>>>>>>>>>>> > if you have any questions or feedback. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Thanks, >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Huaxin >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > On Tue, Feb 10, 2026 at 1:43 PM huaxin gao < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > > Reminder for tomorrow's sync on Iceberg Index Support. >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > Wednesday: Feb. 11 9:00 – 10:00am >>>>>>>>>>>>>>> > > Time zone: America/Los_Angeles >>>>>>>>>>>>>>> > > Google Meet joining info >>>>>>>>>>>>>>> > > Video call link: meet.google.com/nsp-ctyr-khk >>>>>>>>>>>>>>> > > Design doc: >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2 >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7 >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > Thanks, >>>>>>>>>>>>>>> > > Huaxin >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > > On Tue, Feb 3, 2026 at 10:52 PM Péter Váry < >>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>> > > wrote: >>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>> > >> Thanks Huaxin and Steven for organizing this. Looking >>>>>>>>>>>>>>> forward to meet you >>>>>>>>>>>>>>> > >> all next week! >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >> On Wed, Feb 4, 2026, 02:48 Steven Wu <[email protected]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> > >> >>>>>>>>>>>>>>> > >>> We set up the dev calendar event with a new google >>>>>>>>>>>>>>> meet link. Please >>>>>>>>>>>>>>> > >>> ignore the link from Huaxin's original email. >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>> The dev calendar has the correct info (including the >>>>>>>>>>>>>>> new meeting link) >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>> Iceberg Index Support Sync >>>>>>>>>>>>>>> > >>> Wednesday, February 11 · 9:00 – 10:00am >>>>>>>>>>>>>>> > >>> Time zone: America/Los_Angeles >>>>>>>>>>>>>>> > >>> Google Meet joining info >>>>>>>>>>>>>>> > >>> Video call link: https://meet.google.com/nsp-ctyr-khk >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>> On Tue, Feb 3, 2026 at 5:08 PM huaxin gao < >>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>> > >>> wrote: >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> > >>>> Sorry, I meant PST (not EST) :) >>>>>>>>>>>>>>> > >>>> Looking forward to the discussion! >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>> On Tue, Feb 3, 2026 at 4:58 PM Shawn Chang < >>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>> > >>>> wrote: >>>>>>>>>>>>>>> > >>>> >>>>>>>>>>>>>>> > >>>>> Hi Huaxin, >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> Thanks for starting the sync! >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> The meeting seems to be 9-10AM PST on the dev events >>>>>>>>>>>>>>> calendar >>>>>>>>>>>>>>> > >>>>> < >>>>>>>>>>>>>>> https://calendar.google.com/calendar/u/0?cid=MzkwNWQ0OTJmMWI0NTBiYTA3MTJmMmFlNmFmYTc2ZWI3NTdmMTNkODUyMjBjYzAzYWE0NTI3ODg1YWRjNTYyOUBncm91cC5jYWxlbmRhci5nb29nbGUuY29t >>>>>>>>>>>>>>> >, >>>>>>>>>>>>>>> > >>>>> not EST. Maybe it's a typo? >>>>>>>>>>>>>>> > >>>>> Otherwise, looking forward to the discussion! >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> Best, >>>>>>>>>>>>>>> > >>>>> Shawn >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>> On Tue, Feb 3, 2026 at 9:18 AM huaxin gao < >>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>> > >>>>> wrote: >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>>> Hi all, >>>>>>>>>>>>>>> > >>>>>> I'd like to start a dedicated sync to discuss >>>>>>>>>>>>>>> Iceberg Index support. >>>>>>>>>>>>>>> > >>>>>> Here is the existing discussion thread: >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> https://lists.apache.org/thread/fzqk3jjf0xpj5m4cfqb3v4c65p0t04ty >>>>>>>>>>>>>>> . >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> To ground the discussion, here are the two >>>>>>>>>>>>>>> proposals: >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> - Peter's proposal >>>>>>>>>>>>>>> > >>>>>> < >>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2> >>>>>>>>>>>>>>> (overall >>>>>>>>>>>>>>> > >>>>>> index support) >>>>>>>>>>>>>>> > >>>>>> - My proposal >>>>>>>>>>>>>>> > >>>>>> < >>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7 >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>> (bloom filter skipping index) >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> Time slot: Every 3 weeks, Wednesdays at 9 AM to 10 >>>>>>>>>>>>>>> AM EST, starting >>>>>>>>>>>>>>> > >>>>>> next Wednesday (2/11). After FileFormat sync >>>>>>>>>>>>>>> finishes, we plan to use that >>>>>>>>>>>>>>> > >>>>>> slot and switch to every other Monday, 9 AM to 10 >>>>>>>>>>>>>>> AM EST. >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> Meet link: https://meet.google.com/fjn-tyze-mko >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>>> Thanks, >>>>>>>>>>>>>>> > >>>>>> Huaxin >>>>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>
