Hi everyone,
Here is the recording of the Monday sync:
https://apache-iceberg.slack.com/files/U06538UFDH6/F0BTT739Y7P/impromptu_google_meet_meeting_-_aug_31_2026.mp4
Thanks, Shawn, for making it available.
A brief summary of the discussion:
- We finalized the file naming:
- Tracking file: contains the list of range files (similar to a
manifest file for tables).
- Range file: contains the index data rows (similar to a data file
for tables).
- We agreed not to prescribe how expression values are materialized (for
example, Hilbert or hash transformations), leaving room for implementation
flexibility.
- Ryan mentioned that he is working on a v4 proposal for generating
Field IDs for non-materialized columns. The current thinking is that index
columns should follow the same approach.
- We agreed to retain older versions of the index metadata JSON.
- We agreed not to require a strict one-to-one mapping between index
snapshots and table snapshots. An index may contain multiple snapshots
corresponding to the same table snapshot.
Based on the discussion, I have updated the spec PR. Following Ryan's
pattern for non-materialized column definitions, the proposal now
introduces:
- *materialized-fields* for range file columns
- *non-materialized-fields* for tracking file statistics
- *cluster-spec* to define which columns participate in range file
clustering
Overall, I think we reached good alignment on the remaining design points,
and I do not see any open questions at the moment. The PR is in good shape
for review, so I would greatly appreciate any feedback or comments.
Thanks everyone for the productive discussion and continued collaboration.
Peter
Péter Váry <[email protected]> ezt írta (időpont: 2026. aug. 18.,
K, 14:17):
> Hi everyone,
>
> Unfortunately, yesterday's sync was not recorded due to a technical issue.
> :(
>
> Here is a brief summary of the discussion and the decisions we reached:
>
> 1. We agreed to use a list of Iceberg expressions to define the
> *index_keys*, which determine how the index is ordered.
> 2. We agreed not to restrict the functions that can be used in those
> expressions. Engines that do not understand a particular expression can
> simply choose not to use the index.
> 3. We agreed that index region files (leaf files) should be sorted by
> the values generated by the index_keys.
> 4. We discussed using *included_column_ids* to define the contents and
> schema of the index region files, and generally felt this could be a good
> approach.
> 5. We started discussing whether the results of the index_key
> expressions should also be stored in the index region files alongside the
> included columns. In some cases, storing them could help readers optimize
> lookups within a region file. In other cases, the values could be derived
> when needed, making them redundant and potentially hurting performance by
> increasing the size of the region files.
>
> I think we should continue the discussion around point 5, as the outcome
> may influence, or even reopen, the decisions around point 4.
>
> Thanks!
>
>
> Flavio Junqueira <[email protected]> ezt írta (időpont: 2026.
> aug. 4., K, 18:57):
>
>> Thank you for the summary, the proposal and the overall discussion. I
>> have one main question about the target and I apologize in advance if I'm
>> missing some important detail as I might not have the full context.
>>
>> I noticed that partitioning, sorting and clustering (via Hilbert) were
>> discussed in the call in this context of index transform functions. My
>> understanding is that engines currently are already pruning partitions and
>> files using partition information and table metadata like per-column
>> min-max. If that's the case, then what's it that this proposal is trying to
>> achieve by having transform functions that capture partitioning, sorting
>> and clustering when engines are already able to prune without a secondary
>> index? I can more easily see the primary key index example, but I'm not
>> entirely sure about these data layout mechanisms.
>>
>> I also didn't quite get how engines are expected to consume such an
>> index. There was a discussion about where the mapping of the input to files
>> happens, whether that's in Iceberg or the engine, and I didn't quite get
>> the recommendation. Checking Peter's proposal, I couldn't see it clearly
>> either, and if I just missed it, I'd appreciate a pointer to where I should
>> look into.
>>
>> -Flavio
>>
>> On 4 Aug 2026, at 10:52, Péter Váry <[email protected]> wrote:
>>
>> Hi everyone,
>>
>> Here is the recording from yesterday’s sync:
>> https://www.youtube.com/watch?v=Wg9orP9JAmk
>>
>> To summarize:
>> - We agreed to retain the history of index snapshots, but not the history
>> of other index properties. If readers require information that may later be
>> modified by a user, that information should be copied into the
>> corresponding snapshot.
>> - We discussed the possibility of defining index ordering using a list of
>> transform functions.
>>
>> Dan and Yingyi suggested something along the following lines:
>>
>> *"index_keys" : [*
>> * {*
>> * "type": "apply",*
>> * "func-name": {*
>> * "catalog": "iceberg_functions",*
>> * "name": "day"*
>> * },*
>> * "arguments": [*
>> * { "type": "reference", "id": 3 }*
>> * ]*
>> * },*
>> * {*
>> * "type": "apply",*
>> * "func-name": {*
>> * "catalog": "iceberg_functions",*
>> * "name": "truncate"*
>> * },*
>> * "arguments": [*
>> * { "type": "reference", "id": 4 },*
>> * 2*
>> * ]*
>> * }*
>> *]*
>>
>>
>> In this example, rows would first be ordered by day(col3) and then by
>> truncate(col4, 2). To ensure a deterministic ordering, ties would be broken
>> using the original values of columns 3 and 4. If those are still equal,
>> ordering would fall back to the file name and finally the row position.
>>
>> I spent some time thinking about this after the sync, and here are my
>> thoughts.
>>
>> Pros:
>> - Provides a high degree of flexibility.
>> - Enables expression-based indexes.
>>
>> Cons:
>> - Do we need this flexibility at this stage?
>> - Equality deletes, the primary use cases we are targeting, do not
>> require expressions.
>> - Most indexes I have seen in production are not expression-based.
>> - Increased complexity for engines:
>> - Engines must parse and understand transforms to determine whether
>> using an index can improve a query. For example, is ordering by
>> day(timestamp) sufficient for a key lookup?
>> - Complexity increases further as new UDFs are introduced.
>> - Expressions are not pushed into scans today, so index handling
>> cannot be fully hidden behind Iceberg scan planning.
>> - Risk of fragmentation:
>> - The additional complexity may slow engine adoption.
>> - Adopters may choose to support only exact transform matches created
>> by them, rather than learn to reason about the general transforms.
>> - UDFs are prone to missing implementations and behavioral differences
>> across engines. In the worst case, such differences could lead to incorrect
>> query results when indexes are used.
>>
>> Given the points above, I still prefer defining HASH, HILBERT, IDENTITY
>> as a set of standardized values combined with an explicit list of key
>> columns. This keeps adoption simple, unlocks the primary use cases we are
>> targeting today, and allows the Iceberg library to provide the transform
>> implementations directly.
>>
>> Internally, these transforms could still be represented as Iceberg
>> expressions, giving us a migration path toward future index types that may
>> require full expression support.
>>
>> For example:
>>
>> *{*
>> * "transform": "HILBERT",*
>> * "key-column-ids": [3, 4]*
>> *}*
>>
>>
>> I'm interested in hearing other opinions.
>>
>> Thanks,
>> Peter
>>
>> Renjie Liu <[email protected]> ezt írta (időpont: 2026. júl. 28.,
>> K, 10:07):
>>
>>> Hi, Peter:
>>>
>>> Thanks for driving the proposal.
>>>
>>> > *Hash Index Type*: Should hash-based indexes be a separate index
>>> type? They are only applicable to equality and IN predicates, and the hash
>>> algorithm must always be known. A dedicated HASH type could make these
>>> constraints explicit and distinguish hash-based indexes from ordered
>>> indexes.
>>>
>>> I'm leaning toward keeping them in scalar index type due to the
>>> similarity with others transformation functions like identity. As with the
>>> constraints, I think they are properties of transformation functions.
>>>
>>> > *Transform Function/Layout Representation*: Should transform
>>> functions be represented as an enum or a string? I lean toward strings, but
>>> I do not have a strong preference. In many cases, an index can still be
>>> used based on standard column statistics, even if a reader does not
>>> recognize a particular transform.
>>>
>>> This seems more like a problem in java implementation?
>>>
>>> Also I prefer the name transform function compared with layout
>>> representation. Layout representation makes me feel that it's not about the
>>> definition of an index function.
>>>
>>> > *Future Index Types*: How should we handle index types that are
>>> expected in the long term but fall outside the scope of this proposal (for
>>> example, VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved
>>> concepts, or omitted entirely? I lean toward at least mentioning them. The
>>> rationale for introducing an index type is easier to explain when
>>> considering future index families.
>>>
>>> +1
>>>
>>> *> Index History Handling:* Most database systems treat index changes
>>> as a drop-and-recreate operation rather than an in-place update. If we
>>> follow the same approach, there is limited value in maintaining index
>>> metadata history. During index maintenance, a new metadata.json can be
>>> created and atomically swapped in, with the catalog responsible for
>>> cleaning up obsolete metadata and index data files according to its
>>> retention policy.
>>>
>>> +1, I think we only need to keep index data for different snapshots, but
>>> if an index definition changed, we should drop and recreate a new index.
>>>
>>> > *File Format Location*: Should file format be specified at the
>>> leaf-file level or only at the index level? My preference is to stay
>>> aligned with the table metadata model and avoid introducing differences
>>> unless there is a compelling requirement. This would simplify
>>> implementation and maximize code reuse.
>>>
>>> +1.
>>>
>>>
>>>
>>> On Fri, Jul 24, 2026 at 11:57 PM Péter Váry <[email protected]>
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> Here are the key takeaways from Monday’s meeting (July 20):
>>>>
>>>> - *Index Type*: An index type enum will be introduced in v1,
>>>> initially with a single value (SCALAR). This establishes a future-proof
>>>> API
>>>> boundary while leaving room for additional index families.
>>>> - *Strict Total Ordering*: The specification should require
>>>> transform functions to produce values that preserve a well-defined total
>>>> ordering across the key space. The transform function does not need to
>>>> return a long. Instead, leaf files can store the first and last values
>>>> according to the ordering, enabling generic filtering capabilities for
>>>> readers.
>>>>
>>>> Open Questions:
>>>>
>>>> - *Hash Index Type*: Should hash-based indexes be a separate index
>>>> type? They are only applicable to equality and IN predicates, and the
>>>> hash
>>>> algorithm must always be known. A dedicated HASH type could make these
>>>> constraints explicit and distinguish hash-based indexes from ordered
>>>> indexes.
>>>> - *Transform Function/Layout Representation*: Should transform
>>>> functions be represented as an enum or a string? I lean toward strings,
>>>> but
>>>> I do not have a strong preference. In many cases, an index can still be
>>>> used based on standard column statistics, even if a reader does not
>>>> recognize a particular transform.
>>>> - *Future Index Types*: How should we handle index types that are
>>>> expected in the long term but fall outside the scope of this proposal
>>>> (for
>>>> example, VECTOR, IVF, TERM)? Should they be mentioned as
>>>> planned/reserved
>>>> concepts, or omitted entirely? I lean toward at least mentioning them.
>>>> The
>>>> rationale for introducing an index type is easier to explain when
>>>> considering future index families.
>>>> - *Index History Handling:* Most database systems treat index
>>>> changes as a drop-and-recreate operation rather than an in-place
>>>> update. If
>>>> we follow the same approach, there is limited value in maintaining index
>>>> metadata history. During index maintenance, a new metadata.json can be
>>>> created and atomically swapped in, with the catalog responsible for
>>>> cleaning up obsolete metadata and index data files according to its
>>>> retention policy.
>>>> - *Min/Max Statistics Representation*: Should transform-specific or
>>>> sort-order min/max values be represented using the existing file
>>>> statistics
>>>> structure, or exposed through dedicated fields? My preference is to keep
>>>> them within the statistics structure defined by the file schema while
>>>> exposing them directly through the Java API if needed.
>>>> - *File Format Location*: Should file format be specified at the
>>>> leaf-file level or only at the index level? My preference is to stay
>>>> aligned with the table metadata model and avoid introducing differences
>>>> unless there is a compelling requirement. This would simplify
>>>> implementation and maximize code reuse.
>>>>
>>>> I have updated the spec PR based on the decisions above:
>>>> https://github.com/apache/iceberg/pull/16961
>>>>
>>>> Thanks,
>>>> Peter
>>>>
>>>> huaxin gao <[email protected]> ezt írta (időpont: 2026. jún. 26.,
>>>> P, 19:42):
>>>>
>>>>> Hi all, Here is the summary of this Monday's index meeting:
>>>>>
>>>>> Two blocking decisions closed:
>>>>>
>>>>> 1. Index is *not* a table— it's its own object (reuses table
>>>>> machinery under the hood). Reasoning: requires a sort order, no column
>>>>> updates, no partition spec, no overlapping ranges between leaves,
>>>>> inherits
>>>>> base-table permissions, and has its own CREATE/DROP/UPDATE INDEX DDL.
>>>>> 2. Index is a separate catalog entity, with no pointers in table
>>>>> metadata— has its own REST endpoints; the catalog can optionally return
>>>>> index metadata withloadTable to avoid extra round-trips. Keeps table
>>>>> and
>>>>> index updates independent/async.
>>>>>
>>>>>
>>>>> Next steps: Start writing the spec and build out the copy-on-write
>>>>> path now.
>>>>>
>>>>> Here are the draft spec:
>>>>> secondary index spec <https://github.com/apache/iceberg/pull/16961>
>>>>> irc spec <https://github.com/apache/iceberg/pull/16963>
>>>>>
>>>>> Thanks,
>>>>> Huaxin
>>>>>
>>>>> On Sat, Jun 20, 2026 at 11:11 AM huaxin gao <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I built a standalone PoC to validate that the basic index structure
>>>>>> works: that we can build a PK index, convert equality deletes to position
>>>>>> deletes through it, and have every converted delete land on the correct
>>>>>> live row. I ran it up to *100M keys*.
>>>>>>
>>>>>> *Headline: the structure works.* The index builds over up to 100M
>>>>>> keys, the eq-delete → position-delete conversion resolved correctly at
>>>>>> *every* size (100% of converted deletes mapped to the right live
>>>>>> row), and the resulting position deletes are *~8× cheaper to apply* at
>>>>>> query time than the equality deletes they replace.
>>>>>>
>>>>>> Beyond correctness, the run also shows how the index’s *maintenance* cost
>>>>>> scales, comparing copy-on-write (COW, rewrite touched leaves) vs an
>>>>>> append/merge (MOR) option, under a realistic mixed CDC checkpoint (1,000
>>>>>> insert + 500 update + 500 delete), local wall-clock:
>>>>>> keys EQ baseline INDEX (COW) % of 60s (COW) INDEX (MOR) % of 60s
>>>>>> (MOR) correct
>>>>>> 5M 6 ms 6.7s 11.2% 2.2s 3.7% PASS
>>>>>> 20M 8 ms 24.2s 40.4% 6.4s 10.6% PASS
>>>>>> 50M 7 ms 51.6s 86.1% 12.2s 20.4% PASS
>>>>>> *100M* 6 ms *75.0s* 125% (BEHIND) *16.9s* 28.2% (keeps up) PASS
>>>>>>
>>>>>> COW maintenance crosses the 60 s checkpoint around 100M (75 s/cycle,
>>>>>> 125%); MOR stays at ~28% and keeps pace; the equality-delete baseline is
>>>>>> ~6 ms and flat. So the structure works, but *COW alone can’t sustain
>>>>>> scattered CDC at hundreds of millions of keys on a single writer*.
>>>>>> It’s worth allowing a merge-on-read / update-file maintenance option
>>>>>> alongside COW (or sharding the index across parallel writers).
>>>>>>
>>>>>> *Full write-up, all tables, and the in-region reality-check:* link
>>>>>> <https://docs.google.com/document/d/1G3zxbW8X0eU3UrouslZfp42bBc9CvgJGnJyDONCB4PU/edit?tab=t.0>
>>>>>>
>>>>>> Feedback welcome, especially on the spec direction (whether to allow
>>>>>> a merge-on-read / update-file maintenance option alongside COW) and on
>>>>>> the
>>>>>> read-side modeling.
>>>>>>
>>>>>> Thanks,
>>>>>> Huaxin
>>>>>>
>>>>>> On Tue, Jun 9, 2026 at 5:45 PM huaxin gao <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Sorry, we've skipped posting a few of the dedicated index-sync
>>>>>>> summaries to the mailing list; you can find those in the Google doc
>>>>>>> <https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?pli=1&tab=t.8041k7j2n7y3>
>>>>>>> and the Slack channel. Here's yesterday's summary:
>>>>>>>
>>>>>>> *Decided*
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>>> Index vs. table (what we agreed):
>>>>>>> - Reuse table implementation/library code and a near-identical
>>>>>>> spec — the commit path will be custom regardless, so reuse isn't
>>>>>>> the
>>>>>>> deciding factor.
>>>>>>> - An index is not a table from a user/API view: loading or
>>>>>>> writing an index as a table must fail(it would violate index
>>>>>>> invariants).
>>>>>>> - The spec forbids most table behaviors: no overlapping
>>>>>>> files, one mandatory transform sort order, no column updates, no
>>>>>>> partition
>>>>>>> spec.
>>>>>>> - Delete vectors: reuse Iceberg's existing DV — benchmarks
>>>>>>> showed no new delete format is worth introducing.
>>>>>>> - Incremental updates: start with copy-on-write only (no update
>>>>>>> files). For object-store-sized leaves, a full leaf rewrite is about
>>>>>>> as
>>>>>>> cheap as maintaining an overlay update file + DV, so we'll skip the
>>>>>>> MOR
>>>>>>> machinery for now and add it later only if benchmarks prove we need
>>>>>>> it
>>>>>>> (likely just the very-large-leaf case).
>>>>>>> -
>>>>>>>
>>>>>>> Validate the spec first: build a quick, hand-wired prototype
>>>>>>> (Parquet files structured per the spec) and benchmark it on real
>>>>>>> scales
>>>>>>> before formalizing.
>>>>>>>
>>>>>>> *Leaning, not final*
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>>> Indexes are likely separate catalog objects, linked from the
>>>>>>> table by storing just an identifier (like materialized views) and not
>>>>>>> visible in LIST TABLES.
>>>>>>> -
>>>>>>>
>>>>>>> We'll need a commit path for indexes, but simpler than tables
>>>>>>> (no stage-create).
>>>>>>>
>>>>>>> *Still open*
>>>>>>>
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>>> Permissions model — separate vs. inherited (action: look at what
>>>>>>> real DBs do for index permissions).
>>>>>>> - REST/catalog RPC design — minimize round-trips; index metadata
>>>>>>> ideally returned with LOAD TABLE. Catalog RPC cost may dominate
>>>>>>> Parquet IO, so this needs real design.
>>>>>>> -
>>>>>>>
>>>>>>> Scale modeling — target rows-per-leaf vs. leaf size vs.
>>>>>>> metadata-file count.
>>>>>>> -
>>>>>>>
>>>>>>> DDL-on-index semantics (reuse table schema-update actions or
>>>>>>> separate)
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Huaxin
>>>>>>>
>>>>>>> On Wed, Apr 22, 2026 at 8:47 AM Péter Váry <
>>>>>>> [email protected]> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> TL;DR
>>>>>>>> We still need to validate with ADLS and S3, but based on the local
>>>>>>>> tests, the MPHF approach looks more promising if we can tolerate larger
>>>>>>>> files and longer index maintenance times.
>>>>>>>>
>>>>>>>> Details:
>>>>>>>> Here are the results from the local experiments on my Mac. I
>>>>>>>> removed unnecessary statistics from the Parquet files and tested
>>>>>>>> different
>>>>>>>> row group sizes:
>>>>>>>>
>>>>>>>> - For an index file with 1M records, a row group size of 5,000
>>>>>>>> appears to be the sweet spot.
>>>>>>>> - For 10M records, 10,000 rows per row group works best.
>>>>>>>>
>>>>>>>> If you have additional ideas for optimizing Parquet-based indexes,
>>>>>>>> I’d be very interested to hear them.
>>>>>>>> The test code is available on this branch:
>>>>>>>> https://github.com/pvary/iceberg/tree/leaf_bench
>>>>>>>>
>>>>>>>> Best results:
>>>>>>>> *1m records/file*
>>>>>>>>
>>>>>>>> - Parquet - 5000 row/RowGroup
>>>>>>>> - Read: 1191 µs - 1 file open, 3 seek, 123KB read per lookup
>>>>>>>> - Write: 1.7 s, 15 MB
>>>>>>>> - MPHF
>>>>>>>> - Read: 202 µs - 1 file open, 1 seek, 282KB read per lookup
>>>>>>>> - Write: 0.8 s, 34 MB
>>>>>>>>
>>>>>>>> *10m records/file*
>>>>>>>>
>>>>>>>> - Parquet - 10000 row/RowGroup
>>>>>>>> - Read: 4168 µs - 1 file open, 3 seek, 395KB read per lookup
>>>>>>>> - Write: 19.5s s, 144 MB
>>>>>>>> - MPHF
>>>>>>>> - Read: 1086 µs - 1 file open, 1 seek, 2.8 MB (2812KB) read
>>>>>>>> per lookup
>>>>>>>> - Write: 6.5 s, 34 MB, 353 MB
>>>>>>>>
>>>>>>>> Below are the full results.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *Benchmark (indexType)
>>>>>>>> (keyType) (numRows) Mode Cnt Score Error
>>>>>>>> UnitsInvertedIndexBenchmark.lookup PARQUET_1000
>>>>>>>> LONG
>>>>>>>> 1000000 ss 10000 3285.284 ± 5.138
>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_1000
>>>>>>>> LONG
>>>>>>>> 1000000 ss 10000 2522168989.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_1000 LONG
>>>>>>>> 1000000 ss 10000 10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_1000 LONG
>>>>>>>> 1000000 ss 10000 30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup PARQUET_1000 LONG
>>>>>>>> 10000000 ss 10000 35449.614 ± 34.673
>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_1000
>>>>>>>> LONG
>>>>>>>> 10000000 ss 10000 24302649201.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_1000 LONG
>>>>>>>> 10000000 ss 10000 10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_1000 LONG
>>>>>>>> 10000000 ss 10000 30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup PARQUET_5000 LONG
>>>>>>>> 1000000 ss 10000 1191.959 ± 4.169
>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_5000
>>>>>>>> LONG
>>>>>>>> 1000000 ss 10000 1230877229.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_5000 LONG
>>>>>>>> 1000000 ss 10000 10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_5000 LONG
>>>>>>>> 1000000 ss 10000 30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup PARQUET_5000 LONG
>>>>>>>> 10000000 ss 10000 7236.447 ± 10.374
>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_5000
>>>>>>>> LONG
>>>>>>>> 10000000 ss 10000 5650715973.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_5000 LONG
>>>>>>>> 10000000 ss 10000 10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_5000 LONG
>>>>>>>> 10000000 ss 10000 30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup PARQUET_10000 LONG
>>>>>>>> 1000000 ss 10000 1349.946 ± 7.834
>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_10000
>>>>>>>> LONG
>>>>>>>> 1000000 ss 10000 1730219377.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_10000 LONG
>>>>>>>> 1000000 ss 10000 10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_10000 LONG
>>>>>>>> 1000000 ss 10000 30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup PARQUET_10000 LONG
>>>>>>>> 10000000 ss 10000 4168.635 ± 11.051
>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_10000
>>>>>>>> LONG
>>>>>>>> 10000000 ss 10000 3946341532.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_10000 LONG
>>>>>>>> 10000000 ss 10000 10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_10000 LONG
>>>>>>>> 10000000 ss 10000 30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup PARQUET_50000 LONG
>>>>>>>> 1000000 ss 10000 4736.466 ± 38.179
>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_50000
>>>>>>>> LONG
>>>>>>>> 1000000 ss 10000 7427413541.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_50000 LONG
>>>>>>>> 1000000 ss 10000 10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_50000 LONG
>>>>>>>> 1000000 ss 10000 30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup PARQUET_50000 LONG
>>>>>>>> 10000000 ss 10000 4979.031 ± 34.708
>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead PARQUET_50000
>>>>>>>> LONG
>>>>>>>> 10000000 ss 10000 7694887636.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams PARQUET_50000 LONG
>>>>>>>> 10000000 ss 10000 10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks PARQUET_50000 LONG
>>>>>>>> 10000000 ss 10000 30000.000
>>>>>>>> #InvertedIndexBenchmark.lookup MPHF LONG
>>>>>>>> 1000000 ss 10000 202.571 ± 2.336
>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead MPHF
>>>>>>>> LONG
>>>>>>>> 1000000 ss 10000 2821570000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams MPHF LONG
>>>>>>>> 1000000 ss 10000 10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks MPHF LONG
>>>>>>>> 1000000 ss 10000 10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup MPHF LONG
>>>>>>>> 10000000 ss 10000 1086.957 ± 4.524
>>>>>>>> us/opInvertedIndexBenchmark.lookup:bytesRead MPHF
>>>>>>>> LONG
>>>>>>>> 10000000 ss 10000 28119460000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:openStreams MPHF LONG
>>>>>>>> 10000000 ss 10000 10000.000
>>>>>>>> #InvertedIndexBenchmark.lookup:seeks MPHF LONG
>>>>>>>> 10000000 ss 10000 10000.000
>>>>>>>> #InvertedIndexBenchmark.write PARQUET_1000 LONG
>>>>>>>> 1000000 ss 3 1720731.014 ± 876636.004
>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_1000
>>>>>>>> LONG
>>>>>>>> 1000000 ss 3 46453317.000
>>>>>>>> #InvertedIndexBenchmark.write PARQUET_1000 LONG
>>>>>>>> 10000000 ss 3 18547947.876 ± 12258125.307
>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_1000
>>>>>>>> LONG
>>>>>>>> 10000000 ss 3 452655675.000
>>>>>>>> #InvertedIndexBenchmark.write PARQUET_5000 LONG
>>>>>>>> 1000000 ss 3 1718345.583 ± 1103928.016
>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_5000
>>>>>>>> LONG
>>>>>>>> 1000000 ss 3 44845788.000
>>>>>>>> #InvertedIndexBenchmark.write PARQUET_5000 LONG
>>>>>>>> 10000000 ss 3 18604229.931 ± 2668361.915
>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_5000
>>>>>>>> LONG
>>>>>>>> 10000000 ss 3 435388818.000
>>>>>>>> #InvertedIndexBenchmark.write PARQUET_10000 LONG
>>>>>>>> 1000000 ss 3 1761555.389 ± 535857.675
>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_10000
>>>>>>>> LONG
>>>>>>>> 1000000 ss 3 44536635.000
>>>>>>>> #InvertedIndexBenchmark.write PARQUET_10000 LONG
>>>>>>>> 10000000 ss 3 19501588.264 ± 2130054.558
>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_10000
>>>>>>>> LONG
>>>>>>>> 10000000 ss 3 433189623.000
>>>>>>>> #InvertedIndexBenchmark.write PARQUET_50000 LONG
>>>>>>>> 1000000 ss 3 1936624.889 ± 6601363.985
>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_50000
>>>>>>>> LONG
>>>>>>>> 1000000 ss 3 44264655.000
>>>>>>>> #InvertedIndexBenchmark.write PARQUET_50000 LONG
>>>>>>>> 10000000 ss 3 20471742.278 ± 10705206.310
>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes PARQUET_50000
>>>>>>>> LONG
>>>>>>>> 10000000 ss 3 431311305.000
>>>>>>>> #InvertedIndexBenchmark.write MPHF LONG
>>>>>>>> 1000000 ss 3 896573.958 ± 1408024.851
>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes MPHF
>>>>>>>> LONG
>>>>>>>> 1000000 ss 3 102846369.000
>>>>>>>> #InvertedIndexBenchmark.write MPHF LONG
>>>>>>>> 10000000 ss 3 6509348.875 ± 15519975.479
>>>>>>>> us/opInvertedIndexBenchmark.write:indexFileBytes MPHF
>>>>>>>> LONG
>>>>>>>> 10000000 ss 3 1058435733.000 #*
>>>>>>>>
>>>>>>>> huaxin gao <[email protected]> ezt írta (időpont: 2026. ápr.
>>>>>>>> 21., K, 20:53):
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>>
>>>>>>>>> In recent secondary index sync meetings, the discussion converged
>>>>>>>>> on the need to define what an index is from first principles before
>>>>>>>>> settling on physical layout.
>>>>>>>>>
>>>>>>>>> To address that, Peter and I have drafted a requirements document
>>>>>>>>> for a key lookup index (renamed from "primary key index" to avoid
>>>>>>>>> implying
>>>>>>>>> uniqueness enforcement), the goal is to nail down one well-scoped
>>>>>>>>> index
>>>>>>>>> type first.
>>>>>>>>>
>>>>>>>>> Doc: Key Lookup Index Requirements
>>>>>>>>> <https://docs.google.com/document/d/1e0zxK-jA0LBDq8YQlQgFipTHelDFiga8lCkgDTmYub8/edit?tab=t.0#heading=h.8shrgabvl19>
>>>>>>>>>
>>>>>>>>> It covers requirements, three design options (manifest + sorted
>>>>>>>>> Parquet, hash + sorted Parquet, hash + MPHF) and open questions. We
>>>>>>>>> will
>>>>>>>>> add preliminary benchmark results shortly.
>>>>>>>>>
>>>>>>>>> Feedback welcome — inline in the doc, on this thread, or at the
>>>>>>>>> next index sync.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Huaxin
>>>>>>>>>
>>>>>>>>> On Mon, Apr 13, 2026 at 7:22 AM Steven Wu <[email protected]>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Do we need the special index identifier that was originally
>>>>>>>>>> proposed? A generic CatalogObjectIdentifier (with namespace and
>>>>>>>>>> name) would
>>>>>>>>>> be consistent with all object types in the catalog. I have a
>>>>>>>>>> discussion
>>>>>>>>>> thread on the generic identifier topic: [DISCUSS] REST Spec:
>>>>>>>>>> generic CatalogObjectIdentifier.
>>>>>>>>>>
>>>>>>>>>> Should we add an indexes array field to table metadata? It only
>>>>>>>>>> contains a list of index object identifiers. It doesn't contain any
>>>>>>>>>> index
>>>>>>>>>> metadata which should live in the index objects. Yufei was trying to
>>>>>>>>>> bring
>>>>>>>>>> this up at the end of the first sync. But we didn't get enough time
>>>>>>>>>> to
>>>>>>>>>> really discuss it. It will be great to discuss this as the first
>>>>>>>>>> agenda
>>>>>>>>>> item today.
>>>>>>>>>>
>>>>>>>>>> On Mon, Apr 13, 2026 at 3:17 AM Péter Váry <
>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>
>>>>>>>>>>> We had several engaging discussions at the Iceberg Summit, and
>>>>>>>>>>> it was great to finally catch up with many of you in person. We
>>>>>>>>>>> truly
>>>>>>>>>>> missed those who couldn’t attend, hopefully we’ll all meet again at
>>>>>>>>>>> the
>>>>>>>>>>> next summit.
>>>>>>>>>>>
>>>>>>>>>>> To keep the conversation going, Huaxin and I have put together
>>>>>>>>>>> the agenda for our next meeting. As a reminder, we’ll meet on *April
>>>>>>>>>>> 13th, 9:00–10:00 AM *PDT (6:00–7:00 PM CEST).
>>>>>>>>>>>
>>>>>>>>>>> Proposed agenda:
>>>>>>>>>>>
>>>>>>>>>>> - Continue first-principles index design discussion from Mar
>>>>>>>>>>> 30
>>>>>>>>>>> - *Index Ownership and Write Responsibility*
>>>>>>>>>>> - Should writers be allowed to update indexes, or
>>>>>>>>>>> - Should all index writes be handled exclusively by
>>>>>>>>>>> the Index Maintenance process?
>>>>>>>>>>> - If writers can update indexes then we need to define
>>>>>>>>>>> what guarantees are required (compaction, file splitting,
>>>>>>>>>>> layout
>>>>>>>>>>> expectations)?
>>>>>>>>>>> - If only Index Maintenance updates indexes then we
>>>>>>>>>>> only need to define what observable properties should be
>>>>>>>>>>> exposed to
>>>>>>>>>>> consumers? Like:
>>>>>>>>>>> - Expected max files for a single key
>>>>>>>>>>> - Current max files for a single key
>>>>>>>>>>> - Deletes allowed/present
>>>>>>>>>>> - Sorted by
>>>>>>>>>>> - Partitioned by
>>>>>>>>>>> - *Specification Scope: What Belongs in the Spec?*
>>>>>>>>>>> - Related to the ownership question above
>>>>>>>>>>> - Light spec: Just define that the index table should
>>>>>>>>>>> be optimized for retrieval by key columns and the index
>>>>>>>>>>> columns should be
>>>>>>>>>>> contained in the table. This could give us more
>>>>>>>>>>> flexibility if better
>>>>>>>>>>> organization methods come up, or
>>>>>>>>>>> - Detailed spec: We could define the max number of
>>>>>>>>>>> files per index to read for a single key, or even the
>>>>>>>>>>> partitioning and the
>>>>>>>>>>> exact sort order. This could allow more use-cases for a
>>>>>>>>>>> given index, like
>>>>>>>>>>> joins or cardinality estimations.
>>>>>>>>>>> - I would go for light spec for the main types (PK,
>>>>>>>>>>> Containing) and only the Index Maintenance processes
>>>>>>>>>>> should update the
>>>>>>>>>>> Indexes, as for many use-cases the details are not
>>>>>>>>>>> important, and writers
>>>>>>>>>>> will very rarely update the Indexes themselves.
>>>>>>>>>>> - *Logical Placement of Indexes*
>>>>>>>>>>> - Index as a child object of an Iceberg Table, or
>>>>>>>>>>> - Index as a first‑class entity under
>>>>>>>>>>> /namespace/indexes/{index}
>>>>>>>>>>> - Based on the discussions on the summit we are
>>>>>>>>>>> leaning in this direction. This means the index id should
>>>>>>>>>>> be unique in the
>>>>>>>>>>> namespace but helps the catalog implementations quite a bit
>>>>>>>>>>> - *Physical Placement of Index Data*
>>>>>>>>>>> - I don’t think we should specify this. We should have
>>>>>>>>>>> a base location for the index, but can rely on the catalog
>>>>>>>>>>> implementations
>>>>>>>>>>> to decide on their own, like they do with the tables,
>>>>>>>>>>> views, udfs.
>>>>>>>>>>> - *Iceberg Reader Based indexes* (Containing indexes and
>>>>>>>>>>> potentially PK indexes). These are the indexes which could be
>>>>>>>>>>> read by the
>>>>>>>>>>> existing Iceberg readers. We might decide to store the PK
>>>>>>>>>>> index similarly
>>>>>>>>>>> to an Iceberg Table and treat it as a reader based index.
>>>>>>>>>>> - What are the table properties/features exposed to
>>>>>>>>>>> the readers
>>>>>>>>>>> - Maybe just some behavioral descriptors for the
>>>>>>>>>>> optimizer to decide if the index could be used or
>>>>>>>>>>> should be skipped, like:
>>>>>>>>>>> - Expected max files for a single key
>>>>>>>>>>> - max files for a single key
>>>>>>>>>>> - Deletes allowed/present
>>>>>>>>>>> - Sorted by
>>>>>>>>>>> - Partitioned by
>>>>>>>>>>> - The Tasks when reading the index based on the
>>>>>>>>>>> filters and projection
>>>>>>>>>>> - What are the table properties/features exposed to
>>>>>>>>>>> the Index Maintenance. I think this could be internal to
>>>>>>>>>>> the Index
>>>>>>>>>>> Maintenance process and might not be exposed through the
>>>>>>>>>>> spec. The Index
>>>>>>>>>>> Maintenance process could handle this as a standard
>>>>>>>>>>> Iceberg Table and could
>>>>>>>>>>> be based on the Table Maintenance process, but there might
>>>>>>>>>>> be some totally
>>>>>>>>>>> different processes.
>>>>>>>>>>> - It should be possible to add properties to an index
>>>>>>>>>>> defined by the Index Maintenance process which could be used
>>>>>>>>>>> and updated in
>>>>>>>>>>> the next Index Maintenance run.
>>>>>>>>>>> - *PK index storage format benchmark results*
>>>>>>>>>>> - Flat Parquet (baseline)
>>>>>>>>>>> - BTree with Parquet leaves
>>>>>>>>>>> - Vortex
>>>>>>>>>>> - *Open items / next steps*
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Peter
>>>>>>>>>>>
>>>>>>>>>>> huaxin gao <[email protected]> ezt írta (időpont: 2026.
>>>>>>>>>>> márc. 23., H, 3:03):
>>>>>>>>>>>
>>>>>>>>>>>> Hi everyone, I wanted to share an update on the primary key
>>>>>>>>>>>> index work.
>>>>>>>>>>>> Since there are still open questions on whether bloom filter
>>>>>>>>>>>> indexes fit in the secondary index framework or should be treated
>>>>>>>>>>>> as
>>>>>>>>>>>> extended stats, I've shifted focus to the primary key index since
>>>>>>>>>>>> it's a
>>>>>>>>>>>> clearer fit for the framework.
>>>>>>>>>>>> I've put together a proposal for a primary key reverse-lookup
>>>>>>>>>>>> index that maps each key to its physical location (file_path,
>>>>>>>>>>>> row_position). It enables:
>>>>>>>>>>>>
>>>>>>>>>>>> - Scan-time file pruning for point lookups
>>>>>>>>>>>> - Converting key-based deletes into position deletes
>>>>>>>>>>>> (eliminating equality deletes for Flink CDC)
>>>>>>>>>>>> - Accelerating Spark MERGE INTO by replacing full-table
>>>>>>>>>>>> joins with direct file lookups
>>>>>>>>>>>>
>>>>>>>>>>>> Proposal:
>>>>>>>>>>>> https://docs.google.com/document/d/1HuhCZ0n2FqDh8yqQb9oEj1CPM5yXpEsMPGZno2aSf8E/edit?tab=t.0#heading=h.tbevg4q0m9
>>>>>>>>>>>> Feedback welcome!
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Huaxin
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Mar 18, 2026 at 11:42 PM Péter Váry <
>>>>>>>>>>>> [email protected]> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Key takeaways from the general index discussion at the May 16
>>>>>>>>>>>>> meeting.
>>>>>>>>>>>>> Thanks to everyone who participated! The recording is
>>>>>>>>>>>>> available here: https://www.youtube.com/watch?v=btmjhtRWUCE
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Q: Do we need to tie index types to the algorithms used
>>>>>>>>>>>>> to access them?
>>>>>>>>>>>>> - A: From a specification perspective, the goal is to
>>>>>>>>>>>>> define the storage-level data layout so it can be shared
>>>>>>>>>>>>> across engines.
>>>>>>>>>>>>> Engines are free to interpret and use the data as they see
>>>>>>>>>>>>> fit, but the
>>>>>>>>>>>>> on-disk data layout itself must be strictly defined and
>>>>>>>>>>>>> interoperable.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Q: Should we introduce an additional abstraction layer
>>>>>>>>>>>>> (e.g., Vector Index) with sub-types such as IVF and DiskANN?
>>>>>>>>>>>>> - A: This is possible if we decide it is beneficial. I
>>>>>>>>>>>>> explored potential naming, but it is not yet clear how such a
>>>>>>>>>>>>> layer would
>>>>>>>>>>>>> be used in practice.
>>>>>>>>>>>>> *Question to Yingyi Bu*: could you provide examples where
>>>>>>>>>>>>> this additional layer would be useful? Should this abstraction
>>>>>>>>>>>>> be defined
>>>>>>>>>>>>> at the spec level, or is it better handled at the engine level?
>>>>>>>>>>>>> My initial idea was that users would create a generic
>>>>>>>>>>>>> Vector Index and let the engine choose the concrete
>>>>>>>>>>>>> implementation.
>>>>>>>>>>>>> However, this would limit user control and users likely need
>>>>>>>>>>>>> to specify the
>>>>>>>>>>>>> exact index representation, which implies they must be aware
>>>>>>>>>>>>> of the
>>>>>>>>>>>>> available representations.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Q: Do we want to allow extensibility for index types?
>>>>>>>>>>>>> - A: Yes. The intent is to support a small set of
>>>>>>>>>>>>> well-defined index types while allowing experimentation with
>>>>>>>>>>>>> new ones. If a
>>>>>>>>>>>>> new index type proves broadly useful, a follow-up proposal can
>>>>>>>>>>>>> standardize
>>>>>>>>>>>>> it and incorporate it into the spec.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Q: Do we allow multiple versions of an index for the
>>>>>>>>>>>>> same table snapshot?
>>>>>>>>>>>>> - A: Yes. Older index versions must be retained for
>>>>>>>>>>>>> readers that have already started using them, while new
>>>>>>>>>>>>> readers should
>>>>>>>>>>>>> automatically use the latest available version
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Q: Do we need to use materialized views for these
>>>>>>>>>>>>> indexes?
>>>>>>>>>>>>> - A: No. These indexes are primarily examples, and
>>>>>>>>>>>>> different types may require different storage methods.
>>>>>>>>>>>>> However, the Primary
>>>>>>>>>>>>> Key, Containing, and parts of the IVF indexes can be
>>>>>>>>>>>>> structured as Iceberg
>>>>>>>>>>>>> tables. This allows engines to read them natively; in some
>>>>>>>>>>>>> cases, Iceberg
>>>>>>>>>>>>> planners can automatically redirect queries to the index table
>>>>>>>>>>>>> without
>>>>>>>>>>>>> engine modifications. Furthermore, index maintenance for these
>>>>>>>>>>>>> tables can
>>>>>>>>>>>>> leverage existing materialized view maintenance workflows.
>>>>>>>>>>>>> Other index
>>>>>>>>>>>>> types may instead rely on Puffin files or alternative storage
>>>>>>>>>>>>> approaches.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Q: How should index metadata be accessed? Should we add
>>>>>>>>>>>>> explicit pointers for the indexes in the table metadata?
>>>>>>>>>>>>> - A: We did not have sufficient time to fully explore and
>>>>>>>>>>>>> conclude this topic.
>>>>>>>>>>>>> *Question for Yufei Gu*: Did I understand correctly that
>>>>>>>>>>>>> your main concern stems from endpoint resolution from a REST
>>>>>>>>>>>>> Catalog
>>>>>>>>>>>>> perspective? Specifically, if indexes are exposed under a URI
>>>>>>>>>>>>> such as
>>>>>>>>>>>>>
>>>>>>>>>>>>> v1/{prefix}/namespaces/{namespace}/tables/{table}/indexes/{index},
>>>>>>>>>>>>> would
>>>>>>>>>>>>> this make it more difficult for the REST Catalog to resolve
>>>>>>>>>>>>> and route
>>>>>>>>>>>>> requests to the appropriate endpoint?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Suhas Jayaram Subramanya via dev <[email protected]> ezt
>>>>>>>>>>>>> írta (időpont: 2026. márc. 13., P, 23:32):
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi everyone,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here's a proposal for native Vector Index support in Iceberg
>>>>>>>>>>>>>> tables --
>>>>>>>>>>>>>> https://docs.google.com/document/d/1KL4qLOwdqnhOcqTc0EjO1O16NV3M3c-gZCEINDWw4lA/edit?usp=sharing
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We've been working on this proposal with Peter internally at
>>>>>>>>>>>>>> Microsoft and he suggested we post it here to bring this to the
>>>>>>>>>>>>>> community's
>>>>>>>>>>>>>> attention, ahead of the next Secondary Index Sync.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Suhas
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 2026/02/19 04:34:34 huaxin gao wrote:
>>>>>>>>>>>>>> > Hi Everyone,
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Here are the recording and notes from the Iceberg Index
>>>>>>>>>>>>>> Support Sync on
>>>>>>>>>>>>>> > 2/11.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Recording: https://www.youtube.com/watch?v=3sFfQ0A50yk
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Notes:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.8041k7j2n7y3
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > The meeting will move to biweekly, Mondays 9–10am PST,
>>>>>>>>>>>>>> starting March 2.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Since the sync, I updated the Bloom skipping index proposal
>>>>>>>>>>>>>> > <
>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.5r5kl6k3fqwu
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > to address the discussion questions, specifically:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > - Performance justification: when this helps
>>>>>>>>>>>>>> (high-cardinality = / IN,
>>>>>>>>>>>>>> > many data files, high object-store latency) and how it
>>>>>>>>>>>>>> differs from Parquet
>>>>>>>>>>>>>> > row-group Bloom filters (which still require opening the
>>>>>>>>>>>>>> data file).
>>>>>>>>>>>>>> > - Cost / scalability: rough sizing (Bloom blob size per
>>>>>>>>>>>>>> file, Puffin
>>>>>>>>>>>>>> > file size), the planning cost trade-off (driver index reads
>>>>>>>>>>>>>> vs executor
>>>>>>>>>>>>>> > file opens), and mitigations via caching.
>>>>>>>>>>>>>> > - Lifecycle / maintenance: incremental production as new
>>>>>>>>>>>>>> data files
>>>>>>>>>>>>>> > arrive, behavior when the index is missing/behind, and
>>>>>>>>>>>>>> sharding/compaction
>>>>>>>>>>>>>> > plus cleanup to avoid accumulating too many small Puffin
>>>>>>>>>>>>>> files over time.
>>>>>>>>>>>>>> > - Writer expectations: inline (optional) vs asynchronous
>>>>>>>>>>>>>> (primary) index
>>>>>>>>>>>>>> > creation.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > I also implemented a Spark 4.1 POC
>>>>>>>>>>>>>> > <https://github.com/apache/iceberg/pull/15311> and a local
>>>>>>>>>>>>>> benchmark to
>>>>>>>>>>>>>> > quantify both the pruning impact (plannedFiles →
>>>>>>>>>>>>>> afterBloom) and the index
>>>>>>>>>>>>>> > read overhead (statsFiles, statsBytes, bloomPayloadBytes)
>>>>>>>>>>>>>> for point
>>>>>>>>>>>>>> > predicates on high-cardinality columns. Please take a look
>>>>>>>>>>>>>> and let me know
>>>>>>>>>>>>>> > if you have any questions or feedback.
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Thanks,
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > Huaxin
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > On Tue, Feb 10, 2026 at 1:43 PM huaxin gao <[email protected]>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > > Reminder for tomorrow's sync on Iceberg Index Support.
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > Wednesday: Feb. 11 9:00 – 10:00am
>>>>>>>>>>>>>> > > Time zone: America/Los_Angeles
>>>>>>>>>>>>>> > > Google Meet joining info
>>>>>>>>>>>>>> > > Video call link: meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>> > > Design doc:
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > Thanks,
>>>>>>>>>>>>>> > > Huaxin
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > On Tue, Feb 3, 2026 at 10:52 PM Péter Váry <
>>>>>>>>>>>>>> [email protected]>
>>>>>>>>>>>>>> > > wrote:
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >> Thanks Huaxin and Steven for organizing this. Looking
>>>>>>>>>>>>>> forward to meet you
>>>>>>>>>>>>>> > >> all next week!
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >> On Wed, Feb 4, 2026, 02:48 Steven Wu <[email protected]>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> > >>
>>>>>>>>>>>>>> > >>> We set up the dev calendar event with a new google meet
>>>>>>>>>>>>>> link. Please
>>>>>>>>>>>>>> > >>> ignore the link from Huaxin's original email.
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>> The dev calendar has the correct info (including the
>>>>>>>>>>>>>> new meeting link)
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>> Iceberg Index Support Sync
>>>>>>>>>>>>>> > >>> Wednesday, February 11 · 9:00 – 10:00am
>>>>>>>>>>>>>> > >>> Time zone: America/Los_Angeles
>>>>>>>>>>>>>> > >>> Google Meet joining info
>>>>>>>>>>>>>> > >>> Video call link: https://meet.google.com/nsp-ctyr-khk
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>> On Tue, Feb 3, 2026 at 5:08 PM huaxin gao <
>>>>>>>>>>>>>> [email protected]>
>>>>>>>>>>>>>> > >>> wrote:
>>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>>> > >>>> Sorry, I meant PST (not EST) :)
>>>>>>>>>>>>>> > >>>> Looking forward to the discussion!
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>> On Tue, Feb 3, 2026 at 4:58 PM Shawn Chang <
>>>>>>>>>>>>>> [email protected]>
>>>>>>>>>>>>>> > >>>> wrote:
>>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>>> > >>>>> Hi Huaxin,
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>> Thanks for starting the sync!
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>> The meeting seems to be 9-10AM PST on the dev events
>>>>>>>>>>>>>> calendar
>>>>>>>>>>>>>> > >>>>> <
>>>>>>>>>>>>>> https://calendar.google.com/calendar/u/0?cid=MzkwNWQ0OTJmMWI0NTBiYTA3MTJmMmFlNmFmYTc2ZWI3NTdmMTNkODUyMjBjYzAzYWE0NTI3ODg1YWRjNTYyOUBncm91cC5jYWxlbmRhci5nb29nbGUuY29t
>>>>>>>>>>>>>> >,
>>>>>>>>>>>>>> > >>>>> not EST. Maybe it's a typo?
>>>>>>>>>>>>>> > >>>>> Otherwise, looking forward to the discussion!
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>> Best,
>>>>>>>>>>>>>> > >>>>> Shawn
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>> On Tue, Feb 3, 2026 at 9:18 AM huaxin gao <
>>>>>>>>>>>>>> [email protected]>
>>>>>>>>>>>>>> > >>>>> wrote:
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> > >>>>>> Hi all,
>>>>>>>>>>>>>> > >>>>>> I'd like to start a dedicated sync to discuss
>>>>>>>>>>>>>> Iceberg Index support.
>>>>>>>>>>>>>> > >>>>>> Here is the existing discussion thread:
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> https://lists.apache.org/thread/fzqk3jjf0xpj5m4cfqb3v4c65p0t04ty
>>>>>>>>>>>>>> .
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>> To ground the discussion, here are the two proposals:
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>> - Peter's proposal
>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>> https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2>
>>>>>>>>>>>>>> (overall
>>>>>>>>>>>>>> > >>>>>> index support)
>>>>>>>>>>>>>> > >>>>>> - My proposal
>>>>>>>>>>>>>> > >>>>>> <
>>>>>>>>>>>>>> https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > >>>>>> (bloom filter skipping index)
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>> Time slot: Every 3 weeks, Wednesdays at 9 AM to 10
>>>>>>>>>>>>>> AM EST, starting
>>>>>>>>>>>>>> > >>>>>> next Wednesday (2/11). After FileFormat sync
>>>>>>>>>>>>>> finishes, we plan to use that
>>>>>>>>>>>>>> > >>>>>> slot and switch to every other Monday, 9 AM to 10 AM
>>>>>>>>>>>>>> EST.
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>> Meet link: https://meet.google.com/fjn-tyze-mko
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>>> > >>>>>> Huaxin
>>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>
