Hey folks! I've been working on a proposal aimed at enabling an ecosystem of Puffin blob types, with opinionated (but hopefully lightweight) metadata about configuration, scope, and how future blobs interact with deletion and compaction.
For context, separate from any Iceberg specifics, I've been daydreaming about an idea of "progressive indexing" for a few years: after a file is written, perhaps weeks or months later, we may discover that read workloads could be much faster and/or more efficient if we'd made different indexing choices before. Unfortunately, it's hard to predict whether any such change might impact ingest workloads, and rewriting data, while commonplace on small files during early compaction eras, is a non-starter for files that are large and old, so some read-heavy workloads are stuck with scanning huge volumes of data that often has high-selectivity features lurking in dusty corners. While building out an early, as yet unpublished version of the progressive indexing idea, I realized that while I was initially focused on the _timing_ of index builds (still important!) I hadn't spent enough time on the _diversity_ of available indexes, and how to rein in the predictable chaos if my earlier goals were achieved and such choices became broadly available. So, in recent days, this proposal has evolved to focus on what I now consider a prerequisite for the timing and diversity visions, which is: how do we construct and operate an ecosystem that would support the desired diversity, within guardrails that make the chaos less likely and more contained, in which query accuracy can be safeguarded as long as implementers play by the rules? I have looked at the parallel v4 proposal for secondary indexes, and I believe my proposal is complementary, even though there are significant overlaps in scope. In particular mine focuses mainly (but not exclusively) on filters, or "negative" indices, in which an out-of-date index merely costs efficiency (i.e. a file/chunk that does not contain rows that satisfy the predicate is scanned), whereas the `scalar` index described in the v4 proposal is a "positive" index, in which much more care is required to prevent mutations from corrupting indexes, which would cause query correctness problems. Section 12 of my proposal is a detailed comparison. This email is already long enough, so rather than go into further detail, I'll ask you to have a look at https://github.com/acruise/iceberg-indexing-ecosystem/blob/main/docs/proposals/puffin-blob-types-and-compaction.md which is admittedly far too long, just like this email! The same doc is open for comments at https://docs.google.com/document/d/18qlGNPfPy5iT_aRgxLVCqh5W1LuINR7x8Eu2zC6HMoU/edit?usp=sharing and I'm on the Iceberg Slack now as @arrgh if you prefer that kind of thing. :) Thanks! -0xe1a
