Hi Dzeri, Thanks for sharing your view on this! A couple of comments:
*"we got the technical part down already"* As pointed out earlier, one difficulty might be that we don't really have a Java API currently that allows us to have multiple stat files for a snapshot. The API contract says: "replacing the previous statistics file for the snapshot if any exists" See: https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/UpdateStatistics.java#L38 *"secondary indexes" + "into an expanded statistics file spec"* 1) Might be a misunderstanding here: The secondary indexes aren't going to be part of statistics files. It's under discussion but probably there is going to be different wiring to the table metadata, definitely won't be through the TableMetadata.statistics field. 2) Identifying gaps and potential additions to the spec seems a way forward. The linked proposal PR also reflects this: there is an identified gap for secondary indexes, hence the proposal tries to adds it to the spec and make it officially supported in a cross-engine manner. "the spec changes are so simple to the point of being backwards-compatible" Would you mind sharing in a couple of sentences how the spec part would look like? I'm not entirely sure that this ownership of statistics objects is that easy to define in the spec, so I'm wondering what you have in mind. Thanks! Gabor dzeri96 <[email protected]> ezt írta (időpont: 2026. júl. 6., H, 17:05): > Hi Gabor, > > We're now getting into a philosophical discussion regarding the merit of > this change, but I haven't been a part of the iceberg community long > enough to be able to call on previous examples of decisions going one way > or the other. > > My argument is that people have started using statistics files in a way > that is not formally defined by the spec or JVM reference implementation, > but also not forbidden or particularly high-friction. > You can say: a) The iceberg spec wants to stay closed in this regard, we > consider this type of proprietary usage unsupported, or b) We will > facilitate extension of the spec in possibly unknown ways. > > Since Iceberg has no commercial interest (that I know of) to stay closed, > and the spec changes are so simple to the point of being > backwards-compatible, I think that option b) makes sense. Furthermore, the > open source implementations could benefit from a clear "iceberg statistics" > spec, instead of having everything bundled into "iceberg blob files", which > are already overloaded at their inception. > > One more thing: https://github.com/apache/iceberg/pull/16961 proposes a > new file type for secondary indexes. I think we're going to see more such > initiatives. They can be naturally channeled into an expanded statistics > file spec. > > All in all, I think we need to hear more opinions on the worthiness of > this change. At least we got the technical part down already. > > Cheers, > Dzeri > > > On Monday, July 6th, 2026 at 3:52 PM, Gábor Kaszab <[email protected]> > wrote: > > > Hi Dzeri, > > I think the easiest part of this is the code. However, I'm very hesitant > on the spec part, TBH. In general what you want to introduce is an > ownership of statistics files. With this, an engine computing stats will > rewrite the ones that they own, while carrying forward the ones that they > don't. While this seems easy, I'm not sure how such an ownership could be > implemented in the spec. Additionally, this is complexity we'd add to the > spec, but don't provide any benefits for open source engines that follow > the spec. > > > > > Is there anything I misunderstand? > > > > > Best Regards, > > Gabor > > > > > dzeri96 <[email protected]> ezt írta (időpont: 2026. júl. 6., H, 11:17): > > > > > > Hi Gabor, > > > > > > > I had actually overestimated the changes required for implementing > multiple statistics files per snapshot since I just looked at the > TableMetadata builder and assumed that the spec is equally strict. Your > findings make life easier for us. > > > > > > > But you are right, assuming that the first statistics file tied to the > current snapshot is the one we want is not reliable. This is where the > file-level "spec" field comes in. The reader/writer is doing filtering > anyway. We just add one more criteria to the process. > > > > > > > Granted, this requires a change of the iceberg specification, but > adding an extra field is one of the easiest ones, all things considered. > > > > > > > Obviously I might be biased since I'm pushing for one of my original > ideas, but I think this can be implemented with minimal impact, and even > code changes. > > > > > > > Cheers, > > > Dzeri > > > > > > > > > > > On Friday, July 3rd, 2026 at 1:39 PM, Gábor Kaszab < > [email protected]> wrote: > > > > > > > > One correction on the breaking change part:Actually, we'd somewhat > break the contract offered by the Java API here. It explicitly says that > setStatistics is supposed to replace existing stats for the snapshot. > > > > > > > > > > > > Additional thought: > > > > When running compute_table_stats procedure 100 times through Spark, > the expectation is that we end up with a single stat file for the given > snapshot, not 100 ones. Changing this logic is not that straightforward, > because how would we know which stat file to keep and which one to replace. > This makes implementing the suggested approach even more complicated. > > > > > > > > > > > > Gabor > > > > > > > > > > > > Gábor Kaszab <[email protected]> ezt írta (időpont: 2026. júl. > 3., P, 13:30): > > > > > > > > > > > > > Hi Dzeri, > > > > > I entertained the idea of having multiple statistics files for the > same snapshot ID and I did some reading on this to figure out what it would > take to implement. Here are my findings: > > > > > > > > > > > > > > Spec > > > > > I checked the spec first to see if there are any restrictions with > the physical representation and apparently there is a good amount of > flexibility on this front. In table metadata 'statistics' is a list of > table statistics, where table statistics in turn keeps the snapshot ID and > all the other metadata fields, e.g. path to Puffin file, etc. > > > > > There is no constraint that we can't have multiple entries in the > list with the same snapshot ID. > > > > > > > > > > > > > > Read path in reference implementation (Java) > > > > > We use these stats in SparkScan.estimateStatistics(snapshot) where > we find the first element in the list with the given snapshot ID. This > would continue to work fine. Issue might occur in ecosystems where > proprietary writers write proprietary stat objects into this list where the > expected regular stats by Spark might not be the first in the list. In this > case Spark won't use stats, might require a proprietary implementation > within Spark to find the one it needs. But that's the problem of those > environments I guess, open source should be fine :) > > > > > > > > > > > > > > Write path in reference implementation (Java) > > > > > TableMetadata as an in-memory representation keeps a > List<Statistics> similarly to the recommendation of the spec. > > > > > What prevents today from making this work is the Builder for > TableMetadata that is stricter than the spec and every time we add a new > statistics to a snapshot, it wipes out the existing ones. Loosening on this > part would do the trick to allow multiple stats files for the same snapshot > ID. > > > > > > > > > > > > > > The downsides of such design: > > > > > 1) It's more complicated to overwrite existing stats > > > > > Currently, you only have to set a new stat to the snapshot > (UpdateStatistics.setStatitics(statForSnapshotId)) to overwrite the > existing one, but with the change this would be a 2-step process: First > UpdateStatistics.removeStatistics(snapshotId).commit() and then > UpdateStatistics.setStatistics(statForSnapshotId).commit(). It's not even > possible to do it in one go currently (however, feasible to implement). > > > > > 2) Number of tracked files growing > > > > > 3) Is it a breaking change to alter the behaviour from "overwrite" > to "append" when adding stats? > > > > > We wouldn't break anything in the spec, though, just loosen the > strictness of a reference implementation. > > > > > > > > > > > > > > These are my takes, I'd like to hear what others think on this. > > > > > > > > > > > > > > Best Regards, > > > > > Gabor Kaszab > > > > > > > > > > > > > > dzeri96 <[email protected]> ezt írta (időpont: 2026. júl. 2., Cs, > 13:32): > > > > > > > > > > > > > > > Alright, both of you raised some valid points. > > > > > > > > > > > > > > > > Gabor drew my attention to the list of "supported" blob types in > the spec and it made me realize that the people who implemented deletion > vectors kind of had the same problem we're discussing. They had puffin > files which could not share the same lifecycle as regular statistics files > as it is now, so they stored them in manifest files. This is perfectly > reasonable, but it just takes some time to get the mental model right and > separate statistics files from the iceberg puffin spec. Even still, this > distinction stays blurry because the supported blob types are defined at a > lower level than they should be in my opinion. Reading Datasketches and > deletion vector data happens at two separate points of the read process. In > my opinion, it's unlikely that someone is writing a single instance of an > "iceberg puffin reader", so blob types should be defined at their > respective place of access. For example, the deletion vector type should be > defined together with manifest files. > > > > > > > > > > > > > > > > The reason why I'm saying this is not just to re-organize the > documentation, but to make a case that statistics files should be handled > more like large properties files with arbitrary information, than a fixed > spec that has to be interpretable by every reader. And what do we do with > custom properties currently? We carry them over, no matter the change. I'm > not the only person that understood statistics files like this. Dremio, for > example, had an article ( > https://www.dremio.com/blog/extending-apache-iceberg-best-practices-for-storing-and-discovering-custom-metadata/) > on this, and it coincides with my view. The way they solved our problem is > by adding a pointer to their latest puffin file in the table properties. I > don't like this because it doesn't work with time travel and breaks file > cleanup tasks. > > > > > > > > > > > > > > > > Going back to your feedback, I think a realistic approach to > achieving what I suggested is to indeed have multiple puffin files per > snapshot. That way, we solve the write amplification and the commit problem > (though I believe there still are edge cases, just as there are with custom > properties). The way we handle proprietary metadata is either by going > trough the blob types in order to find a supported file, or by adding a new > file-level field that saves the file's spec. One of these specs would be > the "iceberg statistics spec" with datasketches, and everything else is > left to the reader for interpretation. Versions would be handled by the > respective spec, just like iceberg does now. > > > > > > > > > > > > > > > > Overall, I think it's important to reaffirm that my goal is not > for everyone to understand each-other. It's just to prevent systems from > deleting each-others' metadata. An added benefit would be having > snapshot-specific custom properties. I think that's a pretty nice bonus. > > > > > > > > > > > > > > > > Let me know what you think, > > > > > > Dzeri > > > > > > > > > > > > > > > > On Wednesday, July 1st, 2026 at 1:01 PM, Gábor Kaszab < > [email protected]> wrote: > > > > > > > > > > > > > > > > > Hi Dzeri and Tamas, > > > > > > > Thank you for raising this and sharing your opinion! I'm not > entirely sure about the overall conclusions on the proposed way forward > here, though. Let me reflect to some of the details: > > > > > > > > > > > > > > Create new snapshot when computing statistics > > > > > > > This is different from the model we use now. New snapshot is > created when data changes within the table. With adding stats, the data is > intact but additional metadata is added, so following the model, we don't > create a new snapshot. Also, statistics files are attached to a particular > snapshot, wouldn't be that intuitive to say, in snapshot X we added stats > for snapshot Y. > > > > > > > Also, I'm not entirely sure I understand how this would help > with engine_A overwriting stats written by engine_B. Would there be > snapshot_X that contains stats for snapshot_Y written by engine_Z? Not > something we'd want. > > > > > > > > > > > > > > Allow for multiple statistics files to be bound to a snapshot > > > > > > > With this design, how would we match stat files with engines? > Would there be a special string ID that describes the engine? Would there > be a different ID for different versions of the engine? Would these IDs be > part of the Iceberg spec, or would we rely on each engine knowing its own > ID? How would readers know which stat file to read? Would Impala version > X.Y know if it can read stat files written by Spark version A.B? I'm not > sure there is a good way of implementing this. > > > > > > > Not convinced this is the way forward. > > > > > > > > > > > > > > Enforce carry-over of unknown blob data into new puffin files > > > > > > > From the reader's perspective this should be fine, they can > pick the blobs they understand by blob type. > > > > > > > From the writer's perspective I'd argue with "Simple to > implement". It's simple today, as whenever a writer creates stats for a > snapshot, it commits the computed stats for the snapshot unconditionally. > With the carry-over approach there are a couple of extra steps and > difficulties: > > > > > > > - If there are existing stats for the snapshot, they have to > be read > > > > > > > - After writing the merged stats to storage (the stats the > writer computed together with the blobs the writer doesn't know about) > conflict detection has to be performed before commit. Without this, in case > some other writer wrote some proprietary stuff we are expected to carry > over, we could lose that information. This requires not only conflict > checks, but conflict resolution, retries, etc. that complicates a process > that is pretty simple today. > > > > > > > > > > > > > > But most importantly, I'm very hesitant to introduce support > (potentially into the spec too) of proprietary stuff we don't understand. > > > > > > > - Iceberg is foremost a specification (with a number of > reference implementations) that is powerful for cross-engine compatibility. > This means, whatever is in the spec is expected to be understood by engines > following the spec. > > > > > > > - Adding proprietary stuff to stats files helps a subset of > proprietary engines only. The design allows putting whatever proprietary > stuff into Puffin files, but once done, it's up to the proprietary writers > to take care of it. > > > > > > > - Once proprietary stuff made it into the Puffin files, I > don't think the spec should mandate engines to carry them over. > > > > > > > > > > > > > > Ways forward: > > > > > > > 1) Use the proprietary writer to calculate stats > > > > > > > In this particular case I assume there are 2 writers, one that > writes proprietary stuff to Puffin, another that follows the spec and > calculates stats into the same Puffin. > > > > > > > In your description you mention you ran ANALYZE TABLE on your > table, but I don't think that it's valid for Iceberg tables. For Iceberg > tables, the compute_table_stats procedure is for creating the table-level > Puffin files. You either avoid using this through Spark, or you run this > first and after this you run your proprietary writer (I think this is what > you said you were doing) and then your Puffin is as you expect. > > > > > > > > > > > > > > 2) Standardize proprietary stuff > > > > > > > As I mentioned, Iceberg is powerful for cross-engine > compatibility. Your proprietary stuff doesn't help other engines, hence not > much point to add support for the spec/table format to keep them. However, > I think we can examine what exactly you'd like to store in the Puffin files > one-by-one and then discuss if the community shows support to add them to > the spec as officially supported blob types. WDYT? > > > > > > > > > > > > > > Best Regards, > > > > > > > Gabor > > > > > > > > > > > > > > Tamás Máté <[email protected]> ezt írta (időpont: 2026. jún. > 30., K, 19:41): > > > > > > > > > > > > > > > Hi Dzeri, > > > > > > > > > > > > > > > > Thanks for writing this up. I agree that the stats-only case > is the important one to separate from data-changing commits, and that > replacement is the tricky part. > > > > > > > > > > > > > > > > My mental model for the lifecycle is: > > > > > > > > > > > > > > > > 1. An engine analyzes an existing snapshot. > > > > > > > > 2. It writes a statistics file for that snapshot. > > > > > > > > 3. It commits a new table metadata version that references > the statistics file, without creating a new snapshot. > > > > > > > > 4. That metadata entry is carried forward until the snapshot > expires, or until something explicitly replaces or removes it. > > > > > > > > > > > > > > > > A writer that does not understand a blob type has no basis > to validate it. Because of that, I think dropping an unknown blob is safer > than carrying it forward into a newly written statistics file, where it may > become obsolete or misleading. I think we should be aggressive about > replacing statistics, and users should be aware that running a > stats-producing operation may replace the statistics for that snapshot and > drop statistics written by another engine. > > > > > > > > > > > > > > > > I am also concerned about multiple statistics files per > snapshot from a planning-latency perspective. Spark's current > column-statistics planning path reads NDV from statistics file metadata, > not by opening Puffin files ([SparkScan.estimateStatistics]( > https://github.com/apache/iceberg/blob/main/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java#L198-L245)). > If multiple files require Puffin reads during planning, that would > introduce a new planning-time I/O path. One thing to consider is whether > statistics loading should move to table load time instead. This is metadata > after all, and I would expect the relevant Puffin metadata to be small, > probably hundreds of KB at most. > > > > > > > > > > > > > > > > Overall, I think creating a snapshot when statistics are > written is the simplest model and makes the most sense to me. I would tie > statistics to snapshots so they can be expired with the snapshots they > belong to. Also, because a statistics file can already contain multiple > blobs, allowing multiple statistics files per snapshot feels similar to > allowing more blobs, but with extra file-level lifecycle and planning cost. > Problems could quickly arise if engine A drops stats X and Y, and then > engine B, which expects X, Y, and Z together, later finds only Z. > > > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > Best, > > > > > > > > Tamas > > > > > > > > > > > > > > > > On Thu, 25 Jun 2026 at 12:10, dzeri96 <[email protected]> > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > > > I've recently started a discussion on Slack and was > advised to post in the dev mailing list. > > > > > > > > > As puffin/statistics files are starting to catch on, we > are bound to come across situations where one writer wants to create a new > statistics file while some data which it might not understand is already > present in the current snapshot's statistics file. I've come across this > problem in real life, when I ran `ANALYZE TABLE` in iceberg-spark, which > created a new metadata file and replaced my proprietary index data with its > own. > > > > > > > > > You could argue that a single type of writer is expected > for a table, but on the other hand, the spirit of Iceberg is portability. > We can't know who's accessing the table and possibly corrupting its > (statistics-)data. > > > > > > > > > > > > > > > > > > Before I get into the proposed solutions, I think it's > important to distinguish two scenarios in which statistics files are being > written: data-changing and non-data-changing. > > > > > > > > > For data-changing scenarios, I think it's reasonable to > assume that old statistics files are no longer valid, and are therefore OK > to replace. In the rest of this email, I will focus on scenarios where > statistics are being generated and attached to the current snapshot via a > new metadata file, as these are the problematic ones. > > > > > > > > > > > > > > > > > > After a short discussion in Slack, we roughly see three > possible solutions. I think all of them require a change to the iceberg > spec, but with varying gravity: > > > > > > > > > > > > > > > > > > 1. Enforce carry-over of unknown blob data into new puffin > files. > > > > > > > > > Pros: > > > > > > > > > - Backwards-compatible reads, not only in terms of the > iceberg spec, but also in terms of statistics files semantics. > > > > > > > > > - Simple to implement because blob-level metadata is > already available. > > > > > > > > > - One reader could potentially understand statistics blobs > calculated by different writers. > > > > > > > > > Cons: > > > > > > > > > - Write amplification. > > > > > > > > > - Conflict resolution might require re-writing the whole > file again. > > > > > > > > > > > > > > > > > > 2. Allow for multiple statistics files to be bound to a > snapshot. > > > > > > > > > Pros: > > > > > > > > > - Avoids write amplification. > > > > > > > > > - Each writer cares only about its own statistics file. > > > > > > > > > - Finding relevant statistics files is easy thanks to > file-level metadata. > > > > > > > > > - One reader could understand statistics files written by > different writers. > > > > > > > > > Cons: > > > > > > > > > - Backwards-incompatible reads. > > > > > > > > > > > > > > > > > > 3. Create new snapshot when computing statistics. > > > > > > > > > Pros: > > > > > > > > > - Avoids write amplification. > > > > > > > > > - Each writer cares only about its own statistics files. > > > > > > > > > Cons: > > > > > > > > > - Requires readers to iterate over past snapshots in order > to find last valid entry written by a compatible writer. > > > > > > > > > > > > > > > > > > I've definitely left some pros and cons out, but you can > roughly map these cases to ways we handle existing file types (metadata, > manifest lists, manifests). I'm sure people who have spent time designing > the spec can more easily list out the possible pitfalls. In my humble > opinion, #3 might be the most straightforward, but #2 is what I initially > expected from the spec. We are doing #1 internally because it's the only > thing we can do in the current situation. > > > > > > > > > > > > > > > > > > Let me know what you think. > > > > > > > > > Cheers, > > > > > > > > > Dzeri > > > > > > > > >
