Hi all, Following up on the "should V4 manifests be Parquet-only?" thread [1], I'd like to continue the discussion here about supporting various data file formats in V4.
*Should we stop writing new ORC and Avro data files in V4?* Note that this would only affect new writes; existing ORC and Avro files in upgraded tables would still be readable. I've been thinking about this in two dimensions, with more details at the end of the email. Feature gaps: ORC and Avro don't fully support some of the features we've been adding, and our own format TCK already lists what each one is missing [2]. For example, neither supports variant shredding, and the current draft of the vector type spec doesn't support ORC. This also came up last year in #13060 [3] which proposed validating file format compatibility for V3 tables. Upstream activity: Both projects are maintained, but there's limited activity on the formats themselves. The last format changes we needed actually came from folks from the Iceberg community: Gang added geo types to ORC, and Fokko added nanosecond timestamps and fixed-size UUIDs to Avro. Otherwise, the features we need aren't being worked on, so I don't expect these gaps to close on their own. At a high level, the questions for the community are: 1. Should we stop writing new ORC and Avro data files in V4? 2. What does it mean for Iceberg to support ORC and Avro in V4? If we keep them, are we okay with users not being able to use some V4 features with them, as is already the case in V3? 3. If we stop writing them, what bar should a format meet to be writable again in a future version? 4. Are there any users of ORC or Avro data files today? What are your thoughts about using these in V4 versus Parquet? I'd love to hear your thoughts! Best, Kevin Liu [1] https://lists.apache.org/thread/fym02576n5sqy298fjxl0xmsv9z5rb7y [2] https://github.com/apache/iceberg/blob/6951360ec8a01644d3c24824ab635e06a82f8a7c/data/src/test/java/org/apache/iceberg/data/ReadFormatModelTests.java#L190-L212 [3] https://github.com/apache/iceberg/pull/13060 [4] https://github.com/apache/iceberg/issues/18386 [5] https://github.com/apache/iceberg/pull/18387 [6] https://iceberg.apache.org/spec/#appendix-a-format-specific-requirements [7] https://github.com/apache/iceberg/blob/6951360ec8a01644d3c24824ab635e06a82f8a7c/orc/src/main/java/org/apache/iceberg/orc/ORC.java#L114-L117 [8] https://github.com/apache/iceberg/pull/18416/files [9] https://github.com/apache/iceberg/blob/6951360ec8a01644d3c24824ab635e06a82f8a7c/core/src/main/java/org/apache/iceberg/avro/AvroMetrics.java#L35 [10] https://github.com/apache/iceberg/pull/17119 [11] https://lists.apache.org/[email protected]:lte=1y: [12] https://github.com/apache/orc-format/commits/main/specification [13] https://lists.apache.org/[email protected]:lte=1y: [14] https://lists.apache.org/thread/ds0gm18b8cghotv4xtsmgo2ybghht60m [15] https://github.com/apache/avro/commits/main/doc/content/en/docs/++version++/Specification/_index.md [16] https://lists.apache.org/[email protected]:lte=1y: [17] https://github.com/apache/parquet-format/commits/master [18] https://github.com/apache/parquet-format/pull/624 *More details* Feature gaps that block a V3 feature or a proposed V4 feature: - ORC - Default values (V3): our integration can't read them [2] - Geometry and geography types (V3): our integration can't read or write them [2] - Variant type (V3): writes currently fail in Spark [4], and the pending fix only turns the crash into a clear error [5] - Variant shredding (V3): not supported, per the spec [6], so we don't produce variant bounds for ORC files either - Table encryption (V3): our integration rejects the encrypted files that table encryption produces, so ORC can't be used in encrypted tables [7] - Vector type (proposed for V4): marked as not supported in the current draft of the vector type spec [8] - Avro - Variant shredding (V3): not supported, per the spec [6], so we don't produce variant bounds for Avro files either - Geo bounds (V3): Avro can store geo types, but the Avro data files we write have no column metrics (there's been a TODO for this since 2021 [9]), so they don't carry the bounding boxes used for pruning - #13060 noted in May 2025 that "Parquet is currently the only file format that supports all the required capabilities for v3 compatibility" [3]. It was approved but never merged, and Avro has since closed some of those gaps, such as geo types [10]. Upstream activity in the last 12 months: - ORC - Dev list: no technical discussions, only release votes and admin threads [11] - Format spec: unchanged since Gang added geo types in April 2025 [12] - Avro - Dev list: the technical threads were about the language SDKs, not the format [13]. The only two questions about the format itself got no replies, including one about standardizing a vector type that came out of our vector work [14] - Format spec: unchanged since December 2024. The last Iceberg-related changes were Fokko's nanosecond timestamps (Dec 2023) and fixed-size UUIDs (Jan 2024) [15] - Parquet - Dev list: roughly 100 technical threads [16] - Format spec: added a FILE type, ALP encoding (in preview), and NaN counts [17], and a vector type is being discussed [18]
