Agreed on a new thread. I don't have strong feelings about the metadata files layer but I definitely want to keep things open in data files.
Manu Zhang <[email protected]> ezt írta (időpont: 2026. okt. 20., K, 4:02): > Agree with Steven that data file format change needs a new discussion > thread. Also, my preference is to converge on one file format in the Java > reference implementation while leaving the options open in the Spec. > > On Tue, Oct 20, 2026 at 4:43 AM Steven Wu <[email protected]> wrote: > >> Kevin, it is probably better to start a new thread for proposing a format >> change for data files. >> >> On Mon, Oct 19, 2026 at 1:25 PM Kevin Liu <[email protected]> wrote: >> >>> >>> Bumping this thread. Now that the AMT spec is merged [1], V4 manifests >>> are written in Parquet. Tables upgraded from earlier versions can keep >>> their existing Avro manifests, but any new or rewritten manifest has to be >>> a V4 (Parquet) manifest. >>> >>> I want to pick back up what Russell and Manu brought up earlier about >>> converging on a single file format. With V4 manifests now Parquet-only, I >>> think we should do the same for data files. >>> >>> >>> >>> *I'd like to propose dropping ORC and Avro as data file formats in V4.*I >>> looked into where ORC and Avro support is today, and I think there's a >>> pretty good case for it. There are gaps in V3 feature support, most of the >>> non-Java implementations don't support them, and there's not much happening >>> upstream in ORC or Avro on the features we need. >>> >>> What's the best next step here? Happy to share more details in this >>> thread or start a separate one. >>> >>> Best, >>> Kevin Liu >>> >>> [1] https://github.com/apache/iceberg/pull/16025 >>> >>> On Thu, Aug 13, 2026 at 1:09 AM Péter Váry <[email protected]> >>> wrote: >>> >>>> The most important thing, in my view, is that we retain InternalData >>>> (thank Ryan creating this abstraction). We should preserve the flexibility >>>> to change course in the future if needed. >>>> >>>> For that reason, I have a slight preference for continuing to support >>>> multiple metadata file formats as well. Having more than one supported >>>> format helps validate that our APIs are truly general and not overly >>>> tailored to a single implementation. >>>> >>>> That said, I'm more concerned about data file formats than metadata >>>> file formats. Given the additional maintenance effort involved and the >>>> relatively smaller surface area, I would not oppose removing Avro as a >>>> metadata format if that is what the community, especially those doing the >>>> implementation work, prefers. >>>> >>>> Manu Zhang <[email protected]> ezt írta (időpont: 2026. aug. >>>> 13., Cs, 4:42): >>>> >>>>> > I want to clarify that the purpose of this thread is only about >>>>> manifest files. >>>>> Of course, Steven. I'm just echoing Russell's comments regarding the >>>>> possibility of a single file format. >>>>> >>>>> > I think that, for both metadata and data files, we should keep the >>>>> infrastructure open to experimenting with and if we decide so, adopting >>>>> new >>>>> file formats. >>>>> Hi Peter, I don't think we are precluding that possibility, but we are >>>>> focusing on one manifests file format for the reference implementation. We >>>>> can maintain format flexibility through robust abstractions and >>>>> interfaces, >>>>> similar to the great File Format API you contributed. >>>>> >>>>> >>>>> On Thu, Aug 13, 2026 at 3:19 AM Steven Wu <[email protected]> >>>>> wrote: >>>>> >>>>>> > I recall a separate discussion in our community sync meetings >>>>>> about deprecating ORC as well >>>>>> >>>>>> I want to clarify that the purpose of this thread is only about >>>>>> manifest files. I like to keep the data file formats discussion separate >>>>>> (adding new or deprecating old). >>>>>> >>>>>> In v1-v3, manifest files are only in Avro format. As Anoop said, an >>>>>> upgraded v4 table can still contain pre-v4 Avro manifest files and read >>>>>> support for them is still needed. Here, we propose limiting v4 >>>>>> manifest files to Parquet format only, due to the benefits mentioned in >>>>>> this thread. >>>>>> >>>>>> On Wed, Aug 12, 2026 at 7:02 AM Péter Váry < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> I think that, for both metadata and data files, we should keep the >>>>>>> infrastructure open to experimenting with and if we decide so, adopting >>>>>>> new >>>>>>> file formats. File formats are a foundational layer of the ecosystem, >>>>>>> and >>>>>>> tying ourselves to a single downstream project creates unnecessary risk. >>>>>>> >>>>>>> If that project is unwilling to accept features that are critical >>>>>>> for our use cases, or slow to adopt new technological advances, or even >>>>>>> abandoned, our ability to evolve would be constrained by decisions >>>>>>> outside >>>>>>> our control. Maintaining format flexibility helps preserve innovation, >>>>>>> interoperability, and long-term resilience of Iceberg. >>>>>>> >>>>>>> Manu Zhang <[email protected]> ezt írta (időpont: 2026. aug. >>>>>>> 12., Sze, 5:20): >>>>>>> >>>>>>>> +1. I do not see any downside to making V4 manifests Parquet-only. >>>>>>>> Agree with Russell that we should converge on a single file format if >>>>>>>> possible (I recall a separate discussion in our community sync meetings >>>>>>>> about deprecating ORC as well). >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Manu >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Aug 12, 2026 at 3:30 AM Russell Spitzer < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> I'm just pretty neutral here with a slight bias towards Parquet >>>>>>>>> only. I don't think supporting Avro is much more difficult, but that's >>>>>>>>> mostly my prejudice from the Java library. It's probably better for >>>>>>>>> us to >>>>>>>>> converge on a single file format if possible so I think Parquet only >>>>>>>>> is a >>>>>>>>> good step in that direction. >>>>>>>>> >>>>>>>>> On Tue, Aug 11, 2026 at 1:42 PM Anoop Johnson <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> It would be great to make V4 manifest Parquet-only. This >>>>>>>>>> simplifies the spec and the implementation complexity and should >>>>>>>>>> improve >>>>>>>>>> read performance as Steven noted. Also we don't currently keep track >>>>>>>>>> of the >>>>>>>>>> root manifest format currently. So if we really want to support Avro >>>>>>>>>> root >>>>>>>>>> manifests, we need to start tracking the format. The alternative is >>>>>>>>>> to >>>>>>>>>> require writers to always produce the root manifest in Parquet. But >>>>>>>>>> then >>>>>>>>>> there is no point in supporting Avro leaf manifests if writers are >>>>>>>>>> required >>>>>>>>>> to produce root in Parquet anyway. >>>>>>>>>> >>>>>>>>>> Note that for v4 table upgrades we will support v3 leaf manifests >>>>>>>>>> which are in Avro - so the Parquet restriction applies to new V4 >>>>>>>>>> manifests >>>>>>>>>> only. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Anoop >>>>>>>>>> >>>>>>>>>> On Tue, Aug 11, 2026 at 11:29 AM Steven Wu <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> During today’s column update sync, we discussed whether to keep >>>>>>>>>>> the Avro format option for V4 manifest files. The initial >>>>>>>>>>> inclination was >>>>>>>>>>> to keep it because it already exists, even though Avro won’t support >>>>>>>>>>> projection reads on manifest files (including column stats). >>>>>>>>>>> >>>>>>>>>>> With both Avro and Parquet available, integrations and engines >>>>>>>>>>> would need to choose. We expect most will pick Parquet for >>>>>>>>>>> projection-read >>>>>>>>>>> support. That raises a question: should we instead require Parquet >>>>>>>>>>> for V4 >>>>>>>>>>> manifests to reduce the cognitive and decision burden on >>>>>>>>>>> integrations? >>>>>>>>>>> >>>>>>>>>>> This also aligns with Iceberg’s usual priority of read >>>>>>>>>>> performance—especially query scan planning, where projecting column >>>>>>>>>>> stats >>>>>>>>>>> from manifests matters. >>>>>>>>>>> >>>>>>>>>>> In the sync, most people generally agreed with making V4 >>>>>>>>>>> manifests *Parquet-only*. I’m sending this to collect any >>>>>>>>>>> feedback or concerns from the broader community before we settle on >>>>>>>>>>> that >>>>>>>>>>> direction. >>>>>>>>>>> Thanks, >>>>>>>>>>> Steven >>>>>>>>>>> >>>>>>>>>>> P.S. I uploaded the recording to the YouTube channel here >>>>>>>>>>> <https://youtu.be/dDZKzz3PKIk?t=2925> (relevant discussion >>>>>>>>>>> started at 48:45). >>>>>>>>>>> >>>>>>>>>>
