I've also prepared a backgrounder at https://github.com/acruise/iceberg-indexing-ecosystem/blob/main/docs/proposals/puffin-blob-types-background.md which will hopefully provide a gentler, more informal learning curve. :)
-0xe1a On Mon, Nov 2, 2026 at 2:26 PM Alex Cruise <[email protected]> wrote: > Hey folks! > > I've been working on a proposal aimed at enabling an ecosystem of Puffin > blob types, with opinionated (but hopefully lightweight) metadata about > configuration, scope, and how future blobs interact with deletion and > compaction. > > For context, separate from any Iceberg specifics, I've been daydreaming > about an idea of "progressive indexing" for a few years: after a file is > written, perhaps weeks or months later, we may discover that read workloads > could be much faster and/or more efficient if we'd made different indexing > choices before. Unfortunately, it's hard to predict whether any such change > might impact ingest workloads, and rewriting data, while commonplace on > small files during early compaction eras, is a non-starter for files that > are large and old, so some read-heavy workloads are stuck with scanning > huge volumes of data that often has high-selectivity features lurking in > dusty corners. > > While building out an early, as yet unpublished version of the progressive > indexing idea, I realized that while I was initially focused on the > _timing_ of index builds (still important!) I hadn't spent enough time on > the _diversity_ of available indexes, and how to rein in the predictable > chaos if my earlier goals were achieved and such choices became broadly > available. > > So, in recent days, this proposal has evolved to focus on what I now > consider a prerequisite for the timing and diversity visions, which is: how > do we construct and operate an ecosystem that would support the desired > diversity, within guardrails that make the chaos less likely and more > contained, in which query accuracy can be safeguarded as long as > implementers play by the rules? > > I have looked at the parallel v4 proposal for secondary indexes, and I > believe my proposal is complementary, even though there are significant > overlaps in scope. In particular mine focuses mainly (but not exclusively) > on filters, or "negative" indices, in which an out-of-date index merely > costs efficiency (i.e. a file/chunk that does not contain rows that satisfy > the predicate is scanned), whereas the `scalar` index described in the v4 > proposal is a "positive" index, in which much more care is required to > prevent mutations from corrupting indexes, which would cause query > correctness problems. Section 12 of my proposal is a detailed comparison. > > This email is already long enough, so rather than go into further detail, > I'll ask you to have a look at > https://github.com/acruise/iceberg-indexing-ecosystem/blob/main/docs/proposals/puffin-blob-types-and-compaction.md > which is admittedly far too long, just like this email! > > The same doc is open for comments at > https://docs.google.com/document/d/18qlGNPfPy5iT_aRgxLVCqh5W1LuINR7x8Eu2zC6HMoU/edit?usp=sharing > and I'm on the Iceberg Slack now as @arrgh if you prefer that kind of > thing. :) > > Thanks! > > -0xe1a >
