Thank you for the summary, the proposal and the overall discussion. I have one main question about the target and I apologize in advance if I'm missing some important detail as I might not have the full context.
I noticed that partitioning, sorting and clustering (via Hilbert) were discussed in the call in this context of index transform functions. My understanding is that engines currently are already pruning partitions and files using partition information and table metadata like per-column min-max. If that's the case, then what's it that this proposal is trying to achieve by having transform functions that capture partitioning, sorting and clustering when engines are already able to prune without a secondary index? I can more easily see the primary key index example, but I'm not entirely sure about these data layout mechanisms. I also didn't quite get how engines are expected to consume such an index. There was a discussion about where the mapping of the input to files happens, whether that's in Iceberg or the engine, and I didn't quite get the recommendation. Checking Peter's proposal, I couldn't see it clearly either, and if I just missed it, I'd appreciate a pointer to where I should look into. -Flavio > On 4 Aug 2026, at 10:52, Péter Váry <[email protected]> wrote: > > Hi everyone, > > Here is the recording from yesterday’s sync: > https://www.youtube.com/watch?v=Wg9orP9JAmk > > To summarize: > - We agreed to retain the history of index snapshots, but not the history of > other index properties. If readers require information that may later be > modified by a user, that information should be copied into the corresponding > snapshot. > - We discussed the possibility of defining index ordering using a list of > transform functions. > > Dan and Yingyi suggested something along the following lines: > > "index_keys" : [ > { > "type": "apply", > "func-name": { > "catalog": "iceberg_functions", > "name": "day" > }, > "arguments": [ > { "type": "reference", "id": 3 } > ] > }, > { > "type": "apply", > "func-name": { > "catalog": "iceberg_functions", > "name": "truncate" > }, > "arguments": [ > { "type": "reference", "id": 4 }, > 2 > ] > } > ] > > In this example, rows would first be ordered by day(col3) and then by > truncate(col4, 2). To ensure a deterministic ordering, ties would be broken > using the original values of columns 3 and 4. If those are still equal, > ordering would fall back to the file name and finally the row position. > > I spent some time thinking about this after the sync, and here are my > thoughts. > > Pros: > - Provides a high degree of flexibility. > - Enables expression-based indexes. > > Cons: > - Do we need this flexibility at this stage? > - Equality deletes, the primary use cases we are targeting, do not require > expressions. > - Most indexes I have seen in production are not expression-based. > - Increased complexity for engines: > - Engines must parse and understand transforms to determine whether using > an index can improve a query. For example, is ordering by day(timestamp) > sufficient for a key lookup? > - Complexity increases further as new UDFs are introduced. > - Expressions are not pushed into scans today, so index handling cannot be > fully hidden behind Iceberg scan planning. > - Risk of fragmentation: > - The additional complexity may slow engine adoption. > - Adopters may choose to support only exact transform matches created by > them, rather than learn to reason about the general transforms. > - UDFs are prone to missing implementations and behavioral differences > across engines. In the worst case, such differences could lead to incorrect > query results when indexes are used. > > Given the points above, I still prefer defining HASH, HILBERT, IDENTITY as a > set of standardized values combined with an explicit list of key columns. > This keeps adoption simple, unlocks the primary use cases we are targeting > today, and allows the Iceberg library to provide the transform > implementations directly. > > Internally, these transforms could still be represented as Iceberg > expressions, giving us a migration path toward future index types that may > require full expression support. > > For example: > > { > "transform": "HILBERT", > "key-column-ids": [3, 4] > } > > I'm interested in hearing other opinions. > > Thanks, > Peter > > Renjie Liu <[email protected] <mailto:[email protected]>> ezt > írta (időpont: 2026. júl. 28., K, 10:07): >> Hi, Peter: >> >> Thanks for driving the proposal. >> >> > Hash Index Type: Should hash-based indexes be a separate index type? They >> > are only applicable to equality and IN predicates, and the hash algorithm >> > must always be known. A dedicated HASH type could make these constraints >> > explicit and distinguish hash-based indexes from ordered indexes. >> >> I'm leaning toward keeping them in scalar index type due to the similarity >> with others transformation functions like identity. As with the constraints, >> I think they are properties of transformation functions. >> >> > Transform Function/Layout Representation: Should transform functions be >> > represented as an enum or a string? I lean toward strings, but I do not >> > have a strong preference. In many cases, an index can still be used based >> > on standard column statistics, even if a reader does not recognize a >> > particular transform. >> >> This seems more like a problem in java implementation? >> >> Also I prefer the name transform function compared with layout >> representation. Layout representation makes me feel that it's not about the >> definition of an index function. >> >> > Future Index Types: How should we handle index types that are expected in >> > the long term but fall outside the scope of this proposal (for example, >> > VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved concepts, >> > or omitted entirely? I lean toward at least mentioning them. The rationale >> > for introducing an index type is easier to explain when considering future >> > index families. >> >> +1 >> >> > Index History Handling: Most database systems treat index changes as a >> > drop-and-recreate operation rather than an in-place update. If we follow >> > the same approach, there is limited value in maintaining index metadata >> > history. During index maintenance, a new metadata.json can be created and >> > atomically swapped in, with the catalog responsible for cleaning up >> > obsolete metadata and index data files according to its retention policy. >> >> +1, I think we only need to keep index data for different snapshots, but if >> an index definition changed, we should drop and recreate a new index. >> >> > File Format Location: Should file format be specified at the leaf-file >> > level or only at the index level? My preference is to stay aligned with >> > the table metadata model and avoid introducing differences unless there is >> > a compelling requirement. This would simplify implementation and maximize >> > code reuse. >> >> +1. >> >> >> >> On Fri, Jul 24, 2026 at 11:57 PM Péter Váry <[email protected] >> <mailto:[email protected]>> wrote: >>> Hi everyone, >>> >>> Here are the key takeaways from Monday’s meeting (July 20): >>> Index Type: An index type enum will be introduced in v1, initially with a >>> single value (SCALAR). This establishes a future-proof API boundary while >>> leaving room for additional index families. >>> Strict Total Ordering: The specification should require transform functions >>> to produce values that preserve a well-defined total ordering across the >>> key space. The transform function does not need to return a long. Instead, >>> leaf files can store the first and last values according to the ordering, >>> enabling generic filtering capabilities for readers. >>> Open Questions: >>> Hash Index Type: Should hash-based indexes be a separate index type? They >>> are only applicable to equality and IN predicates, and the hash algorithm >>> must always be known. A dedicated HASH type could make these constraints >>> explicit and distinguish hash-based indexes from ordered indexes. >>> Transform Function/Layout Representation: Should transform functions be >>> represented as an enum or a string? I lean toward strings, but I do not >>> have a strong preference. In many cases, an index can still be used based >>> on standard column statistics, even if a reader does not recognize a >>> particular transform. >>> Future Index Types: How should we handle index types that are expected in >>> the long term but fall outside the scope of this proposal (for example, >>> VECTOR, IVF, TERM)? Should they be mentioned as planned/reserved concepts, >>> or omitted entirely? I lean toward at least mentioning them. The rationale >>> for introducing an index type is easier to explain when considering future >>> index families. >>> Index History Handling: Most database systems treat index changes as a >>> drop-and-recreate operation rather than an in-place update. If we follow >>> the same approach, there is limited value in maintaining index metadata >>> history. During index maintenance, a new metadata.json can be created and >>> atomically swapped in, with the catalog responsible for cleaning up >>> obsolete metadata and index data files according to its retention policy. >>> Min/Max Statistics Representation: Should transform-specific or sort-order >>> min/max values be represented using the existing file statistics structure, >>> or exposed through dedicated fields? My preference is to keep them within >>> the statistics structure defined by the file schema while exposing them >>> directly through the Java API if needed. >>> File Format Location: Should file format be specified at the leaf-file >>> level or only at the index level? My preference is to stay aligned with the >>> table metadata model and avoid introducing differences unless there is a >>> compelling requirement. This would simplify implementation and maximize >>> code reuse. >>> I have updated the spec PR based on the decisions above: >>> https://github.com/apache/iceberg/pull/16961 >>> >>> Thanks, >>> Peter >>> >>> huaxin gao <[email protected] <mailto:[email protected]>> ezt >>> írta (időpont: 2026. jún. 26., P, 19:42): >>>> Hi all, Here is the summary of this Monday's index meeting: >>>> >>>> Two blocking decisions closed: >>>> Index is not a table— it's its own object (reuses table machinery under >>>> the hood). Reasoning: requires a sort order, no column updates, no >>>> partition spec, no overlapping ranges between leaves, inherits base-table >>>> permissions, and has its own CREATE/DROP/UPDATE INDEX DDL. >>>> Index is a separate catalog entity, with no pointers in table metadata— >>>> has its own REST endpoints; the catalog can optionally return index >>>> metadata withloadTable to avoid extra round-trips. Keeps table and index >>>> updates independent/async. >>>> >>>> Next steps: Start writing the spec and build out the copy-on-write path >>>> now. >>>> >>>> Here are the draft spec: >>>> secondary index spec <https://github.com/apache/iceberg/pull/16961> >>>> irc spec <https://github.com/apache/iceberg/pull/16963> >>>> >>>> Thanks, >>>> Huaxin >>>> >>>> On Sat, Jun 20, 2026 at 11:11 AM huaxin gao <[email protected] >>>> <mailto:[email protected]>> wrote: >>>>> Hi all, >>>>> >>>>> I built a standalone PoC to validate that the basic index structure >>>>> works: that we can build a PK index, convert equality deletes to position >>>>> deletes through it, and have every converted delete land on the correct >>>>> live row. I ran it up to 100M keys. >>>>> >>>>> Headline: the structure works. The index builds over up to 100M keys, the >>>>> eq-delete → position-delete conversion resolved correctly at every size >>>>> (100% of converted deletes mapped to the right live row), and the >>>>> resulting position deletes are ~8× cheaper to apply at query time than >>>>> the equality deletes they replace. >>>>> >>>>> Beyond correctness, the run also shows how the index’s maintenance cost >>>>> scales, comparing copy-on-write (COW, rewrite touched leaves) vs an >>>>> append/merge (MOR) option, under a realistic mixed CDC checkpoint (1,000 >>>>> insert + 500 update + 500 delete), local wall-clock: >>>>> >>>>> keys EQ baseline INDEX (COW) % of 60s (COW) INDEX (MOR) >>>>> % of 60s (MOR) correct >>>>> 5M 6 ms 6.7s 11.2% 2.2s 3.7% PASS >>>>> 20M 8 ms 24.2s 40.4% 6.4s 10.6% PASS >>>>> 50M 7 ms 51.6s 86.1% 12.2s 20.4% PASS >>>>> 100M 6 ms 75.0s 125% (BEHIND) 16.9s 28.2% (keeps up) >>>>> PASS >>>>> COW maintenance crosses the 60 s checkpoint around 100M (75 s/cycle, >>>>> 125%); MOR stays at ~28% and keeps pace; the equality-delete baseline is >>>>> ~6 ms and flat. So the structure works, but COW alone can’t sustain >>>>> scattered CDC at hundreds of millions of keys on a single writer. It’s >>>>> worth allowing a merge-on-read / update-file maintenance option alongside >>>>> COW (or sharding the index across parallel writers). >>>>> >>>>> Full write-up, all tables, and the in-region reality-check: link >>>>> <https://docs.google.com/document/d/1G3zxbW8X0eU3UrouslZfp42bBc9CvgJGnJyDONCB4PU/edit?tab=t.0> >>>>> Feedback welcome, especially on the spec direction (whether to allow a >>>>> merge-on-read / update-file maintenance option alongside COW) and on the >>>>> read-side modeling. >>>>> >>>>> Thanks, >>>>> Huaxin >>>>> >>>>> >>>>> On Tue, Jun 9, 2026 at 5:45 PM huaxin gao <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>>> Sorry, we've skipped posting a few of the dedicated index-sync >>>>>> summaries to the mailing list; you can find those in the Google doc >>>>>> <https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?pli=1&tab=t.8041k7j2n7y3> >>>>>> and the Slack channel. Here's yesterday's summary: >>>>>> >>>>>> Decided >>>>>> Index vs. table (what we agreed): >>>>>> Reuse table implementation/library code and a near-identical spec — the >>>>>> commit path will be custom regardless, so reuse isn't the deciding >>>>>> factor. >>>>>> An index is not a table from a user/API view: loading or writing an >>>>>> index as a table must fail(it would violate index invariants). >>>>>> The spec forbids most table behaviors: no overlapping files, one >>>>>> mandatory transform sort order, no column updates, no partition spec. >>>>>> Delete vectors: reuse Iceberg's existing DV — benchmarks showed no new >>>>>> delete format is worth introducing. >>>>>> Incremental updates: start with copy-on-write only (no update files). >>>>>> For object-store-sized leaves, a full leaf rewrite is about as cheap as >>>>>> maintaining an overlay update file + DV, so we'll skip the MOR machinery >>>>>> for now and add it later only if benchmarks prove we need it (likely >>>>>> just the very-large-leaf case). >>>>>> Validate the spec first: build a quick, hand-wired prototype (Parquet >>>>>> files structured per the spec) and benchmark it on real scales before >>>>>> formalizing. >>>>>> Leaning, not final >>>>>> >>>>>> Indexes are likely separate catalog objects, linked from the table by >>>>>> storing just an identifier (like materialized views) and not visible in >>>>>> LIST TABLES. >>>>>> We'll need a commit path for indexes, but simpler than tables (no >>>>>> stage-create). >>>>>> Still open >>>>>> >>>>>> >>>>>> Permissions model — separate vs. inherited (action: look at what real >>>>>> DBs do for index permissions). >>>>>> REST/catalog RPC design — minimize round-trips; index metadata ideally >>>>>> returned with LOAD TABLE. Catalog RPC cost may dominate Parquet IO, so >>>>>> this needs real design. >>>>>> Scale modeling — target rows-per-leaf vs. leaf size vs. metadata-file >>>>>> count. >>>>>> DDL-on-index semantics (reuse table schema-update actions or separate) >>>>>> >>>>>> Thanks, >>>>>> Huaxin >>>>>> >>>>>> On Wed, Apr 22, 2026 at 8:47 AM Péter Váry <[email protected] >>>>>> <mailto:[email protected]>> wrote: >>>>>>> Hi All, >>>>>>> >>>>>>> TL;DR >>>>>>> We still need to validate with ADLS and S3, but based on the local >>>>>>> tests, the MPHF approach looks more promising if we can tolerate larger >>>>>>> files and longer index maintenance times. >>>>>>> >>>>>>> Details: >>>>>>> Here are the results from the local experiments on my Mac. I removed >>>>>>> unnecessary statistics from the Parquet files and tested different row >>>>>>> group sizes: >>>>>>> For an index file with 1M records, a row group size of 5,000 appears to >>>>>>> be the sweet spot. >>>>>>> For 10M records, 10,000 rows per row group works best. >>>>>>> If you have additional ideas for optimizing Parquet-based indexes, I’d >>>>>>> be very interested to hear them. >>>>>>> The test code is available on this branch: >>>>>>> https://github.com/pvary/iceberg/tree/leaf_bench >>>>>>> >>>>>>> Best results: >>>>>>> 1m records/file >>>>>>> Parquet - 5000 row/RowGroup >>>>>>> Read: 1191 µs - 1 file open, 3 seek, 123KB read per lookup >>>>>>> Write: 1.7 s, 15 MB >>>>>>> MPHF >>>>>>> Read: 202 µs - 1 file open, 1 seek, 282KB read per lookup >>>>>>> Write: 0.8 s, 34 MB >>>>>>> 10m records/file >>>>>>> Parquet - 10000 row/RowGroup >>>>>>> Read: 4168 µs - 1 file open, 3 seek, 395KB read per lookup >>>>>>> Write: 19.5s s, 144 MB >>>>>>> MPHF >>>>>>> Read: 1086 µs - 1 file open, 1 seek, 2.8 MB (2812KB) read per lookup >>>>>>> Write: 6.5 s, 34 MB, 353 MB >>>>>>> Below are the full results. >>>>>>> Benchmark (indexType) (keyType) >>>>>>> (numRows) Mode Cnt Score Error Units >>>>>>> InvertedIndexBenchmark.lookup PARQUET_1000 LONG >>>>>>> 1000000 ss 10000 3285.284 ± 5.138 us/op >>>>>>> InvertedIndexBenchmark.lookup:bytesRead PARQUET_1000 LONG >>>>>>> 1000000 ss 10000 2522168989.000 # >>>>>>> InvertedIndexBenchmark.lookup:openStreams PARQUET_1000 LONG >>>>>>> 1000000 ss 10000 10000.000 # >>>>>>> InvertedIndexBenchmark.lookup:seeks PARQUET_1000 LONG >>>>>>> 1000000 ss 10000 30000.000 # >>>>>>> InvertedIndexBenchmark.lookup PARQUET_1000 LONG >>>>>>> 10000000 ss 10000 35449.614 ± 34.673 us/op >>>>>>> InvertedIndexBenchmark.lookup:bytesRead PARQUET_1000 LONG >>>>>>> 10000000 ss 10000 24302649201.000 # >>>>>>> InvertedIndexBenchmark.lookup:openStreams PARQUET_1000 LONG >>>>>>> 10000000 ss 10000 10000.000 # >>>>>>> InvertedIndexBenchmark.lookup:seeks PARQUET_1000 LONG >>>>>>> 10000000 ss 10000 30000.000 # >>>>>>> InvertedIndexBenchmark.lookup PARQUET_5000 LONG >>>>>>> 1000000 ss 10000 1191.959 ± 4.169 us/op >>>>>>> InvertedIndexBenchmark.lookup:bytesRead PARQUET_5000 LONG >>>>>>> 1000000 ss 10000 1230877229.000 # >>>>>>> InvertedIndexBenchmark.lookup:openStreams PARQUET_5000 LONG >>>>>>> 1000000 ss 10000 10000.000 # >>>>>>> InvertedIndexBenchmark.lookup:seeks PARQUET_5000 LONG >>>>>>> 1000000 ss 10000 30000.000 # >>>>>>> InvertedIndexBenchmark.lookup PARQUET_5000 LONG >>>>>>> 10000000 ss 10000 7236.447 ± 10.374 us/op >>>>>>> InvertedIndexBenchmark.lookup:bytesRead PARQUET_5000 LONG >>>>>>> 10000000 ss 10000 5650715973.000 # >>>>>>> InvertedIndexBenchmark.lookup:openStreams PARQUET_5000 LONG >>>>>>> 10000000 ss 10000 10000.000 # >>>>>>> InvertedIndexBenchmark.lookup:seeks PARQUET_5000 LONG >>>>>>> 10000000 ss 10000 30000.000 # >>>>>>> InvertedIndexBenchmark.lookup PARQUET_10000 LONG >>>>>>> 1000000 ss 10000 1349.946 ± 7.834 us/op >>>>>>> InvertedIndexBenchmark.lookup:bytesRead PARQUET_10000 LONG >>>>>>> 1000000 ss 10000 1730219377.000 # >>>>>>> InvertedIndexBenchmark.lookup:openStreams PARQUET_10000 LONG >>>>>>> 1000000 ss 10000 10000.000 # >>>>>>> InvertedIndexBenchmark.lookup:seeks PARQUET_10000 LONG >>>>>>> 1000000 ss 10000 30000.000 # >>>>>>> InvertedIndexBenchmark.lookup PARQUET_10000 LONG >>>>>>> 10000000 ss 10000 4168.635 ± 11.051 us/op >>>>>>> InvertedIndexBenchmark.lookup:bytesRead PARQUET_10000 LONG >>>>>>> 10000000 ss 10000 3946341532.000 # >>>>>>> InvertedIndexBenchmark.lookup:openStreams PARQUET_10000 LONG >>>>>>> 10000000 ss 10000 10000.000 # >>>>>>> InvertedIndexBenchmark.lookup:seeks PARQUET_10000 LONG >>>>>>> 10000000 ss 10000 30000.000 # >>>>>>> InvertedIndexBenchmark.lookup PARQUET_50000 LONG >>>>>>> 1000000 ss 10000 4736.466 ± 38.179 us/op >>>>>>> InvertedIndexBenchmark.lookup:bytesRead PARQUET_50000 LONG >>>>>>> 1000000 ss 10000 7427413541.000 # >>>>>>> InvertedIndexBenchmark.lookup:openStreams PARQUET_50000 LONG >>>>>>> 1000000 ss 10000 10000.000 # >>>>>>> InvertedIndexBenchmark.lookup:seeks PARQUET_50000 LONG >>>>>>> 1000000 ss 10000 30000.000 # >>>>>>> InvertedIndexBenchmark.lookup PARQUET_50000 LONG >>>>>>> 10000000 ss 10000 4979.031 ± 34.708 us/op >>>>>>> InvertedIndexBenchmark.lookup:bytesRead PARQUET_50000 LONG >>>>>>> 10000000 ss 10000 7694887636.000 # >>>>>>> InvertedIndexBenchmark.lookup:openStreams PARQUET_50000 LONG >>>>>>> 10000000 ss 10000 10000.000 # >>>>>>> InvertedIndexBenchmark.lookup:seeks PARQUET_50000 LONG >>>>>>> 10000000 ss 10000 30000.000 # >>>>>>> InvertedIndexBenchmark.lookup MPHF LONG >>>>>>> 1000000 ss 10000 202.571 ± 2.336 us/op >>>>>>> InvertedIndexBenchmark.lookup:bytesRead MPHF LONG >>>>>>> 1000000 ss 10000 2821570000.000 # >>>>>>> InvertedIndexBenchmark.lookup:openStreams MPHF LONG >>>>>>> 1000000 ss 10000 10000.000 # >>>>>>> InvertedIndexBenchmark.lookup:seeks MPHF LONG >>>>>>> 1000000 ss 10000 10000.000 # >>>>>>> InvertedIndexBenchmark.lookup MPHF LONG >>>>>>> 10000000 ss 10000 1086.957 ± 4.524 us/op >>>>>>> InvertedIndexBenchmark.lookup:bytesRead MPHF LONG >>>>>>> 10000000 ss 10000 28119460000.000 # >>>>>>> InvertedIndexBenchmark.lookup:openStreams MPHF LONG >>>>>>> 10000000 ss 10000 10000.000 # >>>>>>> InvertedIndexBenchmark.lookup:seeks MPHF LONG >>>>>>> 10000000 ss 10000 10000.000 # >>>>>>> InvertedIndexBenchmark.write PARQUET_1000 LONG >>>>>>> 1000000 ss 3 1720731.014 ± 876636.004 us/op >>>>>>> InvertedIndexBenchmark.write:indexFileBytes PARQUET_1000 LONG >>>>>>> 1000000 ss 3 46453317.000 # >>>>>>> InvertedIndexBenchmark.write PARQUET_1000 LONG >>>>>>> 10000000 ss 3 18547947.876 ± 12258125.307 us/op >>>>>>> InvertedIndexBenchmark.write:indexFileBytes PARQUET_1000 LONG >>>>>>> 10000000 ss 3 452655675.000 # >>>>>>> InvertedIndexBenchmark.write PARQUET_5000 LONG >>>>>>> 1000000 ss 3 1718345.583 ± 1103928.016 us/op >>>>>>> InvertedIndexBenchmark.write:indexFileBytes PARQUET_5000 LONG >>>>>>> 1000000 ss 3 44845788.000 # >>>>>>> InvertedIndexBenchmark.write PARQUET_5000 LONG >>>>>>> 10000000 ss 3 18604229.931 ± 2668361.915 us/op >>>>>>> InvertedIndexBenchmark.write:indexFileBytes PARQUET_5000 LONG >>>>>>> 10000000 ss 3 435388818.000 # >>>>>>> InvertedIndexBenchmark.write PARQUET_10000 LONG >>>>>>> 1000000 ss 3 1761555.389 ± 535857.675 us/op >>>>>>> InvertedIndexBenchmark.write:indexFileBytes PARQUET_10000 LONG >>>>>>> 1000000 ss 3 44536635.000 # >>>>>>> InvertedIndexBenchmark.write PARQUET_10000 LONG >>>>>>> 10000000 ss 3 19501588.264 ± 2130054.558 us/op >>>>>>> InvertedIndexBenchmark.write:indexFileBytes PARQUET_10000 LONG >>>>>>> 10000000 ss 3 433189623.000 # >>>>>>> InvertedIndexBenchmark.write PARQUET_50000 LONG >>>>>>> 1000000 ss 3 1936624.889 ± 6601363.985 us/op >>>>>>> InvertedIndexBenchmark.write:indexFileBytes PARQUET_50000 LONG >>>>>>> 1000000 ss 3 44264655.000 # >>>>>>> InvertedIndexBenchmark.write PARQUET_50000 LONG >>>>>>> 10000000 ss 3 20471742.278 ± 10705206.310 us/op >>>>>>> InvertedIndexBenchmark.write:indexFileBytes PARQUET_50000 LONG >>>>>>> 10000000 ss 3 431311305.000 # >>>>>>> InvertedIndexBenchmark.write MPHF LONG >>>>>>> 1000000 ss 3 896573.958 ± 1408024.851 us/op >>>>>>> InvertedIndexBenchmark.write:indexFileBytes MPHF LONG >>>>>>> 1000000 ss 3 102846369.000 # >>>>>>> InvertedIndexBenchmark.write MPHF LONG >>>>>>> 10000000 ss 3 6509348.875 ± 15519975.479 us/op >>>>>>> InvertedIndexBenchmark.write:indexFileBytes MPHF LONG >>>>>>> 10000000 ss 3 1058435733.000 # >>>>>>> >>>>>>> huaxin gao <[email protected] <mailto:[email protected]>> ezt >>>>>>> írta (időpont: 2026. ápr. 21., K, 20:53): >>>>>>>> Hi all, >>>>>>>> >>>>>>>> In recent secondary index sync meetings, the discussion converged on >>>>>>>> the need to define what an index is from first principles before >>>>>>>> settling on physical layout. >>>>>>>> >>>>>>>> To address that, Peter and I have drafted a requirements document for >>>>>>>> a key lookup index (renamed from "primary key index" to avoid implying >>>>>>>> uniqueness enforcement), the goal is to nail down one well-scoped >>>>>>>> index type first. >>>>>>>> >>>>>>>> Doc: Key Lookup Index Requirements >>>>>>>> <https://docs.google.com/document/d/1e0zxK-jA0LBDq8YQlQgFipTHelDFiga8lCkgDTmYub8/edit?tab=t.0#heading=h.8shrgabvl19> >>>>>>>> It covers requirements, three design options (manifest + sorted >>>>>>>> Parquet, hash + sorted Parquet, hash + MPHF) and open questions. We >>>>>>>> will add preliminary benchmark results shortly. >>>>>>>> >>>>>>>> Feedback welcome — inline in the doc, on this thread, or at the next >>>>>>>> index sync. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Huaxin >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Apr 13, 2026 at 7:22 AM Steven Wu <[email protected] >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>> Do we need the special index identifier that was originally proposed? >>>>>>>>> A generic CatalogObjectIdentifier (with namespace and name) would be >>>>>>>>> consistent with all object types in the catalog. I have a discussion >>>>>>>>> thread on the generic identifier topic: [DISCUSS] REST Spec: generic >>>>>>>>> CatalogObjectIdentifier. >>>>>>>>> >>>>>>>>> Should we add an indexes array field to table metadata? It only >>>>>>>>> contains a list of index object identifiers. It doesn't contain any >>>>>>>>> index metadata which should live in the index objects. Yufei was >>>>>>>>> trying to bring this up at the end of the first sync. But we didn't >>>>>>>>> get enough time to really discuss it. It will be great to discuss >>>>>>>>> this as the first agenda item today. >>>>>>>>> >>>>>>>>> On Mon, Apr 13, 2026 at 3:17 AM Péter Váry >>>>>>>>> <[email protected] <mailto:[email protected]>> >>>>>>>>> wrote: >>>>>>>>>> Hi everyone, >>>>>>>>>> >>>>>>>>>> We had several engaging discussions at the Iceberg Summit, and it >>>>>>>>>> was great to finally catch up with many of you in person. We truly >>>>>>>>>> missed those who couldn’t attend, hopefully we’ll all meet again at >>>>>>>>>> the next summit. >>>>>>>>>> >>>>>>>>>> To keep the conversation going, Huaxin and I have put together the >>>>>>>>>> agenda for our next meeting. As a reminder, we’ll meet on April >>>>>>>>>> 13th, 9:00–10:00 AM PDT (6:00–7:00 PM CEST). >>>>>>>>>> >>>>>>>>>> Proposed agenda: >>>>>>>>>> Continue first-principles index design discussion from Mar 30 >>>>>>>>>> Index Ownership and Write Responsibility >>>>>>>>>> Should writers be allowed to update indexes, or >>>>>>>>>> Should all index writes be handled exclusively by the Index >>>>>>>>>> Maintenance process? >>>>>>>>>> If writers can update indexes then we need to define what guarantees >>>>>>>>>> are required (compaction, file splitting, layout expectations)? >>>>>>>>>> If only Index Maintenance updates indexes then we only need to >>>>>>>>>> define what observable properties should be exposed to consumers? >>>>>>>>>> Like: >>>>>>>>>> Expected max files for a single key >>>>>>>>>> Current max files for a single key >>>>>>>>>> Deletes allowed/present >>>>>>>>>> Sorted by >>>>>>>>>> Partitioned by >>>>>>>>>> Specification Scope: What Belongs in the Spec? >>>>>>>>>> Related to the ownership question above >>>>>>>>>> Light spec: Just define that the index table should be optimized for >>>>>>>>>> retrieval by key columns and the index columns should be contained >>>>>>>>>> in the table. This could give us more flexibility if better >>>>>>>>>> organization methods come up, or >>>>>>>>>> Detailed spec: We could define the max number of files per index to >>>>>>>>>> read for a single key, or even the partitioning and the exact sort >>>>>>>>>> order. This could allow more use-cases for a given index, like joins >>>>>>>>>> or cardinality estimations. >>>>>>>>>> I would go for light spec for the main types (PK, Containing) and >>>>>>>>>> only the Index Maintenance processes should update the Indexes, as >>>>>>>>>> for many use-cases the details are not important, and writers will >>>>>>>>>> very rarely update the Indexes themselves. >>>>>>>>>> Logical Placement of Indexes >>>>>>>>>> Index as a child object of an Iceberg Table, or >>>>>>>>>> Index as a first‑class entity under /namespace/indexes/{index} >>>>>>>>>> Based on the discussions on the summit we are leaning in this >>>>>>>>>> direction. This means the index id should be unique in the namespace >>>>>>>>>> but helps the catalog implementations quite a bit >>>>>>>>>> Physical Placement of Index Data >>>>>>>>>> I don’t think we should specify this. We should have a base location >>>>>>>>>> for the index, but can rely on the catalog implementations to decide >>>>>>>>>> on their own, like they do with the tables, views, udfs. >>>>>>>>>> Iceberg Reader Based indexes (Containing indexes and potentially PK >>>>>>>>>> indexes). These are the indexes which could be read by the existing >>>>>>>>>> Iceberg readers. We might decide to store the PK index similarly to >>>>>>>>>> an Iceberg Table and treat it as a reader based index. >>>>>>>>>> What are the table properties/features exposed to the readers >>>>>>>>>> Maybe just some behavioral descriptors for the optimizer to decide >>>>>>>>>> if the index could be used or should be skipped, like: >>>>>>>>>> Expected max files for a single key >>>>>>>>>> max files for a single key >>>>>>>>>> Deletes allowed/present >>>>>>>>>> Sorted by >>>>>>>>>> Partitioned by >>>>>>>>>> The Tasks when reading the index based on the filters and projection >>>>>>>>>> What are the table properties/features exposed to the Index >>>>>>>>>> Maintenance. I think this could be internal to the Index Maintenance >>>>>>>>>> process and might not be exposed through the spec. The Index >>>>>>>>>> Maintenance process could handle this as a standard Iceberg Table >>>>>>>>>> and could be based on the Table Maintenance process, but there might >>>>>>>>>> be some totally different processes. >>>>>>>>>> It should be possible to add properties to an index defined by the >>>>>>>>>> Index Maintenance process which could be used and updated in the >>>>>>>>>> next Index Maintenance run. >>>>>>>>>> PK index storage format benchmark results >>>>>>>>>> Flat Parquet (baseline) >>>>>>>>>> BTree with Parquet leaves >>>>>>>>>> Vortex >>>>>>>>>> Open items / next steps >>>>>>>>>> Thanks, >>>>>>>>>> Peter >>>>>>>>>> >>>>>>>>>> huaxin gao <[email protected] <mailto:[email protected]>> >>>>>>>>>> ezt írta (időpont: 2026. márc. 23., H, 3:03): >>>>>>>>>>> Hi everyone, I wanted to share an update on the primary key index >>>>>>>>>>> work. >>>>>>>>>>> Since there are still open questions on whether bloom filter >>>>>>>>>>> indexes fit in the secondary index framework or should be treated >>>>>>>>>>> as extended stats, I've shifted focus to the primary key index >>>>>>>>>>> since it's a clearer fit for the framework. >>>>>>>>>>> I've put together a proposal for a primary key reverse-lookup index >>>>>>>>>>> that maps each key to its physical location (file_path, >>>>>>>>>>> row_position). It enables: >>>>>>>>>>> Scan-time file pruning for point lookups >>>>>>>>>>> Converting key-based deletes into position deletes (eliminating >>>>>>>>>>> equality deletes for Flink CDC) >>>>>>>>>>> Accelerating Spark MERGE INTO by replacing full-table joins with >>>>>>>>>>> direct file lookups >>>>>>>>>>> Proposal: >>>>>>>>>>> https://docs.google.com/document/d/1HuhCZ0n2FqDh8yqQb9oEj1CPM5yXpEsMPGZno2aSf8E/edit?tab=t.0#heading=h.tbevg4q0m9 >>>>>>>>>>> Feedback welcome! >>>>>>>>>>> Thanks, >>>>>>>>>>> Huaxin >>>>>>>>>>> >>>>>>>>>>> On Wed, Mar 18, 2026 at 11:42 PM Péter Váry >>>>>>>>>>> <[email protected] <mailto:[email protected]>> >>>>>>>>>>> wrote: >>>>>>>>>>>> Key takeaways from the general index discussion at the May 16 >>>>>>>>>>>> meeting. >>>>>>>>>>>> Thanks to everyone who participated! The recording is available >>>>>>>>>>>> here: https://www.youtube.com/watch?v=btmjhtRWUCE >>>>>>>>>>>> Q: Do we need to tie index types to the algorithms used to access >>>>>>>>>>>> them? >>>>>>>>>>>> A: From a specification perspective, the goal is to define the >>>>>>>>>>>> storage-level data layout so it can be shared across engines. >>>>>>>>>>>> Engines are free to interpret and use the data as they see fit, >>>>>>>>>>>> but the on-disk data layout itself must be strictly defined and >>>>>>>>>>>> interoperable. >>>>>>>>>>>> >>>>>>>>>>>> Q: Should we introduce an additional abstraction layer (e.g., >>>>>>>>>>>> Vector Index) with sub-types such as IVF and DiskANN? >>>>>>>>>>>> A: This is possible if we decide it is beneficial. I explored >>>>>>>>>>>> potential naming, but it is not yet clear how such a layer would >>>>>>>>>>>> be used in practice. >>>>>>>>>>>> Question to Yingyi Bu: could you provide examples where this >>>>>>>>>>>> additional layer would be useful? Should this abstraction be >>>>>>>>>>>> defined at the spec level, or is it better handled at the engine >>>>>>>>>>>> level? >>>>>>>>>>>> My initial idea was that users would create a generic Vector Index >>>>>>>>>>>> and let the engine choose the concrete implementation. However, >>>>>>>>>>>> this would limit user control and users likely need to specify the >>>>>>>>>>>> exact index representation, which implies they must be aware of >>>>>>>>>>>> the available representations. >>>>>>>>>>>> >>>>>>>>>>>> Q: Do we want to allow extensibility for index types? >>>>>>>>>>>> A: Yes. The intent is to support a small set of well-defined index >>>>>>>>>>>> types while allowing experimentation with new ones. If a new index >>>>>>>>>>>> type proves broadly useful, a follow-up proposal can standardize >>>>>>>>>>>> it and incorporate it into the spec. >>>>>>>>>>>> >>>>>>>>>>>> Q: Do we allow multiple versions of an index for the same table >>>>>>>>>>>> snapshot? >>>>>>>>>>>> A: Yes. Older index versions must be retained for readers that >>>>>>>>>>>> have already started using them, while new readers should >>>>>>>>>>>> automatically use the latest available version >>>>>>>>>>>> >>>>>>>>>>>> Q: Do we need to use materialized views for these indexes? >>>>>>>>>>>> A: No. These indexes are primarily examples, and different types >>>>>>>>>>>> may require different storage methods. However, the Primary Key, >>>>>>>>>>>> Containing, and parts of the IVF indexes can be structured as >>>>>>>>>>>> Iceberg tables. This allows engines to read them natively; in some >>>>>>>>>>>> cases, Iceberg planners can automatically redirect queries to the >>>>>>>>>>>> index table without engine modifications. Furthermore, index >>>>>>>>>>>> maintenance for these tables can leverage existing materialized >>>>>>>>>>>> view maintenance workflows. Other index types may instead rely on >>>>>>>>>>>> Puffin files or alternative storage approaches. >>>>>>>>>>>> >>>>>>>>>>>> Q: How should index metadata be accessed? Should we add explicit >>>>>>>>>>>> pointers for the indexes in the table metadata? >>>>>>>>>>>> A: We did not have sufficient time to fully explore and conclude >>>>>>>>>>>> this topic. >>>>>>>>>>>> Question for Yufei Gu: Did I understand correctly that your main >>>>>>>>>>>> concern stems from endpoint resolution from a REST Catalog >>>>>>>>>>>> perspective? Specifically, if indexes are exposed under a URI such >>>>>>>>>>>> as >>>>>>>>>>>> v1/{prefix}/namespaces/{namespace}/tables/{table}/indexes/{index}, >>>>>>>>>>>> would this make it more difficult for the REST Catalog to resolve >>>>>>>>>>>> and route requests to the appropriate endpoint? >>>>>>>>>>>> >>>>>>>>>>>> Suhas Jayaram Subramanya via dev <[email protected] >>>>>>>>>>>> <mailto:[email protected]>> ezt írta (időpont: 2026. márc. >>>>>>>>>>>> 13., P, 23:32): >>>>>>>>>>>>> Hi everyone, >>>>>>>>>>>>> >>>>>>>>>>>>> Here's a proposal for native Vector Index support in Iceberg >>>>>>>>>>>>> tables -- >>>>>>>>>>>>> https://docs.google.com/document/d/1KL4qLOwdqnhOcqTc0EjO1O16NV3M3c-gZCEINDWw4lA/edit?usp=sharing >>>>>>>>>>>>> >>>>>>>>>>>>> We've been working on this proposal with Peter internally at >>>>>>>>>>>>> Microsoft and he suggested we post it here to bring this to the >>>>>>>>>>>>> community's attention, ahead of the next Secondary Index Sync. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Suhas >>>>>>>>>>>>> >>>>>>>>>>>>> On 2026/02/19 04:34:34 huaxin gao wrote: >>>>>>>>>>>>> > Hi Everyone, >>>>>>>>>>>>> > >>>>>>>>>>>>> > Here are the recording and notes from the Iceberg Index Support >>>>>>>>>>>>> > Sync on >>>>>>>>>>>>> > 2/11. >>>>>>>>>>>>> > >>>>>>>>>>>>> > Recording: https://www.youtube.com/watch?v=3sFfQ0A50yk >>>>>>>>>>>>> > >>>>>>>>>>>>> > Notes: >>>>>>>>>>>>> > https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.8041k7j2n7y3 >>>>>>>>>>>>> > >>>>>>>>>>>>> > The meeting will move to biweekly, Mondays 9–10am PST, starting >>>>>>>>>>>>> > March 2. >>>>>>>>>>>>> > >>>>>>>>>>>>> > Since the sync, I updated the Bloom skipping index proposal >>>>>>>>>>>>> > <https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.5r5kl6k3fqwu> >>>>>>>>>>>>> > to address the discussion questions, specifically: >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > - Performance justification: when this helps (high-cardinality >>>>>>>>>>>>> > = / IN, >>>>>>>>>>>>> > many data files, high object-store latency) and how it differs >>>>>>>>>>>>> > from Parquet >>>>>>>>>>>>> > row-group Bloom filters (which still require opening the data >>>>>>>>>>>>> > file). >>>>>>>>>>>>> > - Cost / scalability: rough sizing (Bloom blob size per file, >>>>>>>>>>>>> > Puffin >>>>>>>>>>>>> > file size), the planning cost trade-off (driver index reads vs >>>>>>>>>>>>> > executor >>>>>>>>>>>>> > file opens), and mitigations via caching. >>>>>>>>>>>>> > - Lifecycle / maintenance: incremental production as new data >>>>>>>>>>>>> > files >>>>>>>>>>>>> > arrive, behavior when the index is missing/behind, and >>>>>>>>>>>>> > sharding/compaction >>>>>>>>>>>>> > plus cleanup to avoid accumulating too many small Puffin files >>>>>>>>>>>>> > over time. >>>>>>>>>>>>> > - Writer expectations: inline (optional) vs asynchronous >>>>>>>>>>>>> > (primary) index >>>>>>>>>>>>> > creation. >>>>>>>>>>>>> > >>>>>>>>>>>>> > I also implemented a Spark 4.1 POC >>>>>>>>>>>>> > <https://github.com/apache/iceberg/pull/15311> and a local >>>>>>>>>>>>> > benchmark to >>>>>>>>>>>>> > quantify both the pruning impact (plannedFiles → afterBloom) >>>>>>>>>>>>> > and the index >>>>>>>>>>>>> > read overhead (statsFiles, statsBytes, bloomPayloadBytes) for >>>>>>>>>>>>> > point >>>>>>>>>>>>> > predicates on high-cardinality columns. Please take a look and >>>>>>>>>>>>> > let me know >>>>>>>>>>>>> > if you have any questions or feedback. >>>>>>>>>>>>> > >>>>>>>>>>>>> > Thanks, >>>>>>>>>>>>> > >>>>>>>>>>>>> > Huaxin >>>>>>>>>>>>> > >>>>>>>>>>>>> > On Tue, Feb 10, 2026 at 1:43 PM huaxin gao <[email protected] >>>>>>>>>>>>> > <mailto:[email protected]>> wrote: >>>>>>>>>>>>> > >>>>>>>>>>>>> > > Reminder for tomorrow's sync on Iceberg Index Support. >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > Wednesday: Feb. 11 9:00 – 10:00am >>>>>>>>>>>>> > > Time zone: America/Los_Angeles >>>>>>>>>>>>> > > Google Meet joining info >>>>>>>>>>>>> > > Video call link: meet.google.com/nsp-ctyr-khk >>>>>>>>>>>>> > > <http://meet.google.com/nsp-ctyr-khk> >>>>>>>>>>>>> > > Design doc: >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2 >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7 >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > Thanks, >>>>>>>>>>>>> > > Huaxin >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > On Tue, Feb 3, 2026 at 10:52 PM Péter Váry <[email protected] >>>>>>>>>>>>> > > <mailto:[email protected]>> >>>>>>>>>>>>> > > wrote: >>>>>>>>>>>>> > > >>>>>>>>>>>>> > >> Thanks Huaxin and Steven for organizing this. Looking >>>>>>>>>>>>> > >> forward to meet you >>>>>>>>>>>>> > >> all next week! >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >> On Wed, Feb 4, 2026, 02:48 Steven Wu <[email protected] >>>>>>>>>>>>> > >> <mailto:[email protected]>> wrote: >>>>>>>>>>>>> > >> >>>>>>>>>>>>> > >>> We set up the dev calendar event with a new google meet >>>>>>>>>>>>> > >>> link. Please >>>>>>>>>>>>> > >>> ignore the link from Huaxin's original email. >>>>>>>>>>>>> > >>> >>>>>>>>>>>>> > >>> The dev calendar has the correct info (including the new >>>>>>>>>>>>> > >>> meeting link) >>>>>>>>>>>>> > >>> >>>>>>>>>>>>> > >>> Iceberg Index Support Sync >>>>>>>>>>>>> > >>> Wednesday, February 11 · 9:00 – 10:00am >>>>>>>>>>>>> > >>> Time zone: America/Los_Angeles >>>>>>>>>>>>> > >>> Google Meet joining info >>>>>>>>>>>>> > >>> Video call link: https://meet.google.com/nsp-ctyr-khk >>>>>>>>>>>>> > >>> >>>>>>>>>>>>> > >>> On Tue, Feb 3, 2026 at 5:08 PM huaxin gao <[email protected] >>>>>>>>>>>>> > >>> <mailto:[email protected]>> >>>>>>>>>>>>> > >>> wrote: >>>>>>>>>>>>> > >>> >>>>>>>>>>>>> > >>>> Sorry, I meant PST (not EST) :) >>>>>>>>>>>>> > >>>> Looking forward to the discussion! >>>>>>>>>>>>> > >>>> >>>>>>>>>>>>> > >>>> On Tue, Feb 3, 2026 at 4:58 PM Shawn Chang >>>>>>>>>>>>> > >>>> <[email protected] <mailto:[email protected]>> >>>>>>>>>>>>> > >>>> wrote: >>>>>>>>>>>>> > >>>> >>>>>>>>>>>>> > >>>>> Hi Huaxin, >>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>> > >>>>> Thanks for starting the sync! >>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>> > >>>>> The meeting seems to be 9-10AM PST on the dev events >>>>>>>>>>>>> > >>>>> calendar >>>>>>>>>>>>> > >>>>> <https://calendar.google.com/calendar/u/0?cid=MzkwNWQ0OTJmMWI0NTBiYTA3MTJmMmFlNmFmYTc2ZWI3NTdmMTNkODUyMjBjYzAzYWE0NTI3ODg1YWRjNTYyOUBncm91cC5jYWxlbmRhci5nb29nbGUuY29t>, >>>>>>>>>>>>> > >>>>> not EST. Maybe it's a typo? >>>>>>>>>>>>> > >>>>> Otherwise, looking forward to the discussion! >>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>> > >>>>> Best, >>>>>>>>>>>>> > >>>>> Shawn >>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>> > >>>>> On Tue, Feb 3, 2026 at 9:18 AM huaxin gao >>>>>>>>>>>>> > >>>>> <[email protected] <mailto:[email protected]>> >>>>>>>>>>>>> > >>>>> wrote: >>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>> > >>>>>> Hi all, >>>>>>>>>>>>> > >>>>>> I'd like to start a dedicated sync to discuss Iceberg >>>>>>>>>>>>> > >>>>>> Index support. >>>>>>>>>>>>> > >>>>>> Here is the existing discussion thread: >>>>>>>>>>>>> > >>>>>> https://lists.apache.org/thread/fzqk3jjf0xpj5m4cfqb3v4c65p0t04ty. >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> To ground the discussion, here are the two proposals: >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> - Peter's proposal >>>>>>>>>>>>> > >>>>>> <https://docs.google.com/document/d/1N6a2IOzC6Qsqv7NBqHKesees4N6WF49YUSIX2FrF7S0/edit?tab=t.0#heading=h.hs6r9d26w1y2> >>>>>>>>>>>>> > >>>>>> (overall >>>>>>>>>>>>> > >>>>>> index support) >>>>>>>>>>>>> > >>>>>> - My proposal >>>>>>>>>>>>> > >>>>>> <https://docs.google.com/document/d/1x-0KT43aTrt8u6EV7EgSietIFQSkGsocqwnBTHPebRU/edit?tab=t.0#heading=h.qouk73o4jxx7> >>>>>>>>>>>>> > >>>>>> (bloom filter skipping index) >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> Time slot: Every 3 weeks, Wednesdays at 9 AM to 10 AM >>>>>>>>>>>>> > >>>>>> EST, starting >>>>>>>>>>>>> > >>>>>> next Wednesday (2/11). After FileFormat sync finishes, >>>>>>>>>>>>> > >>>>>> we plan to use that >>>>>>>>>>>>> > >>>>>> slot and switch to every other Monday, 9 AM to 10 AM EST. >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> Meet link: https://meet.google.com/fjn-tyze-mko >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>>> Thanks, >>>>>>>>>>>>> > >>>>>> Huaxin >>>>>>>>>>>>> > >>>>>> >>>>>>>>>>>>> > >>>>> >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>
