Thanks for bringing this up. I think there are two separate considerations here:
Whether the manifest encoding should be configurable per table. Which encoding Paimon should standardize on. Iceberg v1–v3 standardize manifests on Avro. Parquet manifests are being explored as part of the still-unadopted v4 adaptive metadata tree, which is a format-version decision tied to a broader metadata redesign. If Iceberg v4 eventually standardizes on Parquet, that still supports defining the manifest encoding at the table-format level rather than exposing it as an independent table option. For Paimon’s current access pattern, I believe Avro remains the more appropriate choice. Manifest processing commonly scans entries sequentially and materializes most of the metadata for each entry. A row-oriented format fits this pattern well, while the benefits of strong columnar compression and column pruning are less significant than they are for data files. Paimon’s ManifestAvroReader also supports projected reads and applies partition and bucket filters before decoding the complete data-file metadata. Therefore, the projection use cases we currently have do not require switching to Parquet. Avro is also the manifest encoding already supported by both Java Paimon and PyPaimon. For these reasons, standardizing manifests on Avro gives Paimon a simple and predictable cross-language contract, matches the dominant manifest access pattern, and avoids format-dependent compatibility risks. If future workloads demonstrate a clear benefit from another encoding, we can revisit it through a versioned format evolution with explicit compatibility and migration semantics. > 2026年8月13日 11:49，焦明烨 <[email protected]> 写道： > > It seems that Iceberg is choosing to keep Parquet only, rather than > Avro. Parquet format manifests may be benefited in projection reads. > >> >> Hi devs, >> >> I'd like to discuss removing the `manifest.format` option and >> standardizing Paimon manifest files on Avro. >> >> The related PR is: >> https://github.com/apache/paimon/pull/9154 >> >> The main reasons are: >> >> 1. Avro is the default and de facto manifest format. >> 2. PyPaimon currently supports Avro manifests only. >> 3. Allowing alternative formats introduces unnecessary compatibility >> risks across implementations. >> 4. We checked our managed production tables and found no usage of >> `manifest.format`. >> >> The proposal is to: >> >> - Remove `manifest.format` from `CoreOptions` and the documentation. >> - Always use Avro for manifest files. >> - Keep `manifest.compression` configurable. >> >> The compatibility concern is that tables whose manifests were >> actually written in another format, such as ORC, would no longer be >> readable after upgrading. We have not found such usage, but would >> like to confirm whether anyone in the community relies on non-Avro >> manifests. >> >> Please share your concerns or use cases if you depend on this option. >> >> Best, >> Junhao > > #/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ > This e-mail and its attachments contain confidential information from > XIAOMI, which is intended only for the person or entity whose address is > listed above. Any use of the information contained herein in any way > (including, but not limited to, total or partial disclosure, reproduction, or > dissemination) by persons other than the intended recipient(s) is prohibited. > If you receive this e-mail in error, please notify the sender by phone or > email immediately and delete it!******/#
