The most important thing, in my view, is that we retain InternalData (thank Ryan creating this abstraction). We should preserve the flexibility to change course in the future if needed.
For that reason, I have a slight preference for continuing to support multiple metadata file formats as well. Having more than one supported format helps validate that our APIs are truly general and not overly tailored to a single implementation. That said, I'm more concerned about data file formats than metadata file formats. Given the additional maintenance effort involved and the relatively smaller surface area, I would not oppose removing Avro as a metadata format if that is what the community, especially those doing the implementation work, prefers. Manu Zhang <[email protected]> ezt írta (időpont: 2026. aug. 13., Cs, 4:42): > > I want to clarify that the purpose of this thread is only about manifest > files. > Of course, Steven. I'm just echoing Russell's comments regarding the > possibility of a single file format. > > > I think that, for both metadata and data files, we should keep the > infrastructure open to experimenting with and if we decide so, adopting new > file formats. > Hi Peter, I don't think we are precluding that possibility, but we are > focusing on one manifests file format for the reference implementation. We > can maintain format flexibility through robust abstractions and interfaces, > similar to the great File Format API you contributed. > > > On Thu, Aug 13, 2026 at 3:19 AM Steven Wu <[email protected]> wrote: > >> > I recall a separate discussion in our community sync meetings about >> deprecating ORC as well >> >> I want to clarify that the purpose of this thread is only about manifest >> files. I like to keep the data file formats discussion separate (adding new >> or deprecating old). >> >> In v1-v3, manifest files are only in Avro format. As Anoop said, an >> upgraded v4 table can still contain pre-v4 Avro manifest files and read >> support for them is still needed. Here, we propose limiting v4 manifest >> files to Parquet format only, due to the benefits mentioned in this thread. >> >> On Wed, Aug 12, 2026 at 7:02 AM Péter Váry <[email protected]> >> wrote: >> >>> I think that, for both metadata and data files, we should keep the >>> infrastructure open to experimenting with and if we decide so, adopting new >>> file formats. File formats are a foundational layer of the ecosystem, and >>> tying ourselves to a single downstream project creates unnecessary risk. >>> >>> If that project is unwilling to accept features that are critical for >>> our use cases, or slow to adopt new technological advances, or even >>> abandoned, our ability to evolve would be constrained by decisions outside >>> our control. Maintaining format flexibility helps preserve innovation, >>> interoperability, and long-term resilience of Iceberg. >>> >>> Manu Zhang <[email protected]> ezt írta (időpont: 2026. aug. 12., >>> Sze, 5:20): >>> >>>> +1. I do not see any downside to making V4 manifests Parquet-only. >>>> Agree with Russell that we should converge on a single file format if >>>> possible (I recall a separate discussion in our community sync meetings >>>> about deprecating ORC as well). >>>> >>>> Thanks, >>>> Manu >>>> >>>> >>>> On Wed, Aug 12, 2026 at 3:30 AM Russell Spitzer < >>>> [email protected]> wrote: >>>> >>>>> I'm just pretty neutral here with a slight bias towards Parquet only. >>>>> I don't think supporting Avro is much more difficult, but that's mostly my >>>>> prejudice from the Java library. It's probably better for us to converge >>>>> on >>>>> a single file format if possible so I think Parquet only is a good step in >>>>> that direction. >>>>> >>>>> On Tue, Aug 11, 2026 at 1:42 PM Anoop Johnson <[email protected]> >>>>> wrote: >>>>> >>>>>> It would be great to make V4 manifest Parquet-only. This simplifies >>>>>> the spec and the implementation complexity and should improve read >>>>>> performance as Steven noted. Also we don't currently keep track of the >>>>>> root >>>>>> manifest format currently. So if we really want to support Avro root >>>>>> manifests, we need to start tracking the format. The alternative is to >>>>>> require writers to always produce the root manifest in Parquet. But then >>>>>> there is no point in supporting Avro leaf manifests if writers are >>>>>> required >>>>>> to produce root in Parquet anyway. >>>>>> >>>>>> Note that for v4 table upgrades we will support v3 leaf manifests >>>>>> which are in Avro - so the Parquet restriction applies to new V4 >>>>>> manifests >>>>>> only. >>>>>> >>>>>> Best, >>>>>> Anoop >>>>>> >>>>>> On Tue, Aug 11, 2026 at 11:29 AM Steven Wu <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> During today’s column update sync, we discussed whether to keep the >>>>>>> Avro format option for V4 manifest files. The initial inclination was to >>>>>>> keep it because it already exists, even though Avro won’t support >>>>>>> projection reads on manifest files (including column stats). >>>>>>> >>>>>>> With both Avro and Parquet available, integrations and engines would >>>>>>> need to choose. We expect most will pick Parquet for projection-read >>>>>>> support. That raises a question: should we instead require Parquet for >>>>>>> V4 >>>>>>> manifests to reduce the cognitive and decision burden on integrations? >>>>>>> >>>>>>> This also aligns with Iceberg’s usual priority of read >>>>>>> performance—especially query scan planning, where projecting column >>>>>>> stats >>>>>>> from manifests matters. >>>>>>> >>>>>>> In the sync, most people generally agreed with making V4 manifests >>>>>>> *Parquet-only*. I’m sending this to collect any feedback or >>>>>>> concerns from the broader community before we settle on that direction. >>>>>>> Thanks, >>>>>>> Steven >>>>>>> >>>>>>> P.S. I uploaded the recording to the YouTube channel here >>>>>>> <https://youtu.be/dDZKzz3PKIk?t=2925> (relevant discussion started >>>>>>> at 48:45). >>>>>>> >>>>>>
