On Tue, Oct 20, 2026 at 5:35 PM Péter Váry <[email protected]> wrote: > > Agreed on a new thread. > I don't have strong feelings about the metadata files layer but I definitely > want to keep things open in data files.
I also think keeping data files in open formats is worth the effort, especially with popular options like Vortex, Lance, and Nimble. Let's wait for Kevin's new thread. I'm not sure whether he's proposing to move away from file formats altogether or just drop support for Avro and ORC. > > Manu Zhang <[email protected]> ezt írta (időpont: 2026. okt. 20., K, > 4:02): >> >> Agree with Steven that data file format change needs a new discussion >> thread. Also, my preference is to converge on one file format in the Java >> reference implementation while leaving the options open in the Spec. >> >> On Tue, Oct 20, 2026 at 4:43 AM Steven Wu <[email protected]> wrote: >>> >>> Kevin, it is probably better to start a new thread for proposing a format >>> change for data files. >>> >>> On Mon, Oct 19, 2026 at 1:25 PM Kevin Liu <[email protected]> wrote: >>>> >>>> >>>> Bumping this thread. Now that the AMT spec is merged [1], V4 manifests are >>>> written in Parquet. Tables upgraded from earlier versions can keep their >>>> existing Avro manifests, but any new or rewritten manifest has to be a V4 >>>> (Parquet) manifest. >>>> >>>> I want to pick back up what Russell and Manu brought up earlier about >>>> converging on a single file format. With V4 manifests now Parquet-only, I >>>> think we should do the same for data files. >>>> >>>> I'd like to propose dropping ORC and Avro as data file formats in V4. >>>> >>>> I looked into where ORC and Avro support is today, and I think there's a >>>> pretty good case for it. There are gaps in V3 feature support, most of the >>>> non-Java implementations don't support them, and there's not much >>>> happening upstream in ORC or Avro on the features we need. >>>> >>>> What's the best next step here? Happy to share more details in this thread >>>> or start a separate one. >>>> >>>> Best, >>>> Kevin Liu >>>> >>>> [1] https://github.com/apache/iceberg/pull/16025 >>>> >>>> On Thu, Aug 13, 2026 at 1:09 AM Péter Váry <[email protected]> >>>> wrote: >>>>> >>>>> The most important thing, in my view, is that we retain InternalData >>>>> (thank Ryan creating this abstraction). We should preserve the >>>>> flexibility to change course in the future if needed. >>>>> >>>>> For that reason, I have a slight preference for continuing to support >>>>> multiple metadata file formats as well. Having more than one supported >>>>> format helps validate that our APIs are truly general and not overly >>>>> tailored to a single implementation. >>>>> >>>>> That said, I'm more concerned about data file formats than metadata file >>>>> formats. Given the additional maintenance effort involved and the >>>>> relatively smaller surface area, I would not oppose removing Avro as a >>>>> metadata format if that is what the community, especially those doing the >>>>> implementation work, prefers. >>>>> >>>>> Manu Zhang <[email protected]> ezt írta (időpont: 2026. aug. 13., >>>>> Cs, 4:42): >>>>>> >>>>>> > I want to clarify that the purpose of this thread is only about >>>>>> > manifest files. >>>>>> Of course, Steven. I'm just echoing Russell's comments regarding the >>>>>> possibility of a single file format. >>>>>> >>>>>> > I think that, for both metadata and data files, we should keep the >>>>>> > infrastructure open to experimenting with and if we decide so, >>>>>> > adopting new file formats. >>>>>> Hi Peter, I don't think we are precluding that possibility, but we are >>>>>> focusing on one manifests file format for the reference implementation. >>>>>> We can maintain format flexibility through robust abstractions and >>>>>> interfaces, similar to the great File Format API you contributed. >>>>>> >>>>>> >>>>>> On Thu, Aug 13, 2026 at 3:19 AM Steven Wu <[email protected]> wrote: >>>>>>> >>>>>>> > I recall a separate discussion in our community sync meetings about >>>>>>> > deprecating ORC as well >>>>>>> >>>>>>> I want to clarify that the purpose of this thread is only about >>>>>>> manifest files. I like to keep the data file formats discussion >>>>>>> separate (adding new or deprecating old). >>>>>>> >>>>>>> In v1-v3, manifest files are only in Avro format. As Anoop said, an >>>>>>> upgraded v4 table can still contain pre-v4 Avro manifest files and read >>>>>>> support for them is still needed. Here, we propose limiting v4 manifest >>>>>>> files to Parquet format only, due to the benefits mentioned in this >>>>>>> thread. >>>>>>> >>>>>>> On Wed, Aug 12, 2026 at 7:02 AM Péter Váry >>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>> I think that, for both metadata and data files, we should keep the >>>>>>>> infrastructure open to experimenting with and if we decide so, >>>>>>>> adopting new file formats. File formats are a foundational layer of >>>>>>>> the ecosystem, and tying ourselves to a single downstream project >>>>>>>> creates unnecessary risk. >>>>>>>> >>>>>>>> If that project is unwilling to accept features that are critical for >>>>>>>> our use cases, or slow to adopt new technological advances, or even >>>>>>>> abandoned, our ability to evolve would be constrained by decisions >>>>>>>> outside our control. Maintaining format flexibility helps preserve >>>>>>>> innovation, interoperability, and long-term resilience of Iceberg. >>>>>>>> >>>>>>>> Manu Zhang <[email protected]> ezt írta (időpont: 2026. aug. >>>>>>>> 12., Sze, 5:20): >>>>>>>>> >>>>>>>>> +1. I do not see any downside to making V4 manifests Parquet-only. >>>>>>>>> Agree with Russell that we should converge on a single file format if >>>>>>>>> possible (I recall a separate discussion in our community sync >>>>>>>>> meetings about deprecating ORC as well). >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Manu >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Aug 12, 2026 at 3:30 AM Russell Spitzer >>>>>>>>> <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>> I'm just pretty neutral here with a slight bias towards Parquet >>>>>>>>>> only. I don't think supporting Avro is much more difficult, but >>>>>>>>>> that's mostly my prejudice from the Java library. It's probably >>>>>>>>>> better for us to converge on a single file format if possible so I >>>>>>>>>> think Parquet only is a good step in that direction. >>>>>>>>>> >>>>>>>>>> On Tue, Aug 11, 2026 at 1:42 PM Anoop Johnson <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> It would be great to make V4 manifest Parquet-only. This simplifies >>>>>>>>>>> the spec and the implementation complexity and should improve read >>>>>>>>>>> performance as Steven noted. Also we don't currently keep track of >>>>>>>>>>> the root manifest format currently. So if we really want to support >>>>>>>>>>> Avro root manifests, we need to start tracking the format. The >>>>>>>>>>> alternative is to require writers to always produce the root >>>>>>>>>>> manifest in Parquet. But then there is no point in supporting Avro >>>>>>>>>>> leaf manifests if writers are required to produce root in Parquet >>>>>>>>>>> anyway. >>>>>>>>>>> >>>>>>>>>>> Note that for v4 table upgrades we will support v3 leaf manifests >>>>>>>>>>> which are in Avro - so the Parquet restriction applies to new V4 >>>>>>>>>>> manifests only. >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Anoop >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 11, 2026 at 11:29 AM Steven Wu <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> During today’s column update sync, we discussed whether to keep >>>>>>>>>>>> the Avro format option for V4 manifest files. The initial >>>>>>>>>>>> inclination was to keep it because it already exists, even though >>>>>>>>>>>> Avro won’t support projection reads on manifest files (including >>>>>>>>>>>> column stats). >>>>>>>>>>>> >>>>>>>>>>>> With both Avro and Parquet available, integrations and engines >>>>>>>>>>>> would need to choose. We expect most will pick Parquet for >>>>>>>>>>>> projection-read support. That raises a question: should we instead >>>>>>>>>>>> require Parquet for V4 manifests to reduce the cognitive and >>>>>>>>>>>> decision burden on integrations? >>>>>>>>>>>> >>>>>>>>>>>> This also aligns with Iceberg’s usual priority of read >>>>>>>>>>>> performance—especially query scan planning, where projecting >>>>>>>>>>>> column stats from manifests matters. >>>>>>>>>>>> >>>>>>>>>>>> In the sync, most people generally agreed with making V4 manifests >>>>>>>>>>>> Parquet-only. I’m sending this to collect any feedback or concerns >>>>>>>>>>>> from the broader community before we settle on that direction. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Steven >>>>>>>>>>>> >>>>>>>>>>>> P.S. I uploaded the recording to the YouTube channel here >>>>>>>>>>>> (relevant discussion started at 48:45). -- Regards Junwang Zhao
