andygrove opened a new issue, #6955: URL: https://github.com/apache/datafusion-comet/issues/6955
### What is the problem the feature request solves? Some eligibility rules decline writes over settings that cannot change iceberg-java's output, so the fallback gains nothing: - **`write.parquet.shred-variants=true`** (`requireShredVariantsDisabled`). Iceberg 1.11 shreds only top-level `variant` columns (`ParquetFormatModel`: `shreddingEnabled && variantAnalyzer != null && hasVariantColumns(schema)`), and the column type rule already declines a write with a `variant` column at any depth. Spark 4.x resolves this property into every write, so a session that turns shredding on makes every Iceberg write fall back. - **An `io-impl` table property.** iceberg-java reads `io-impl` from catalog properties only (`HiveCatalog`, `HadoopCatalog`, `RESTSessionCatalog`, `GlueTableOperations` and the other catalogs), never from table properties. `requireRecognizedTableFileIO` already checks the `FileIO` the table actually uses. - **`parquet.*` keys iceberg-java's writer never reads.** For a Spark data write, iceberg-java builds `ParquetProperties` from Iceberg's own properties (`Parquet.WriteBuilder.build`). It reads `parquet.enable.dictionary` from the table and write properties, and from the Hadoop configuration only `parquet.columnindex.truncate.length` (`ParquetWriter`) and the codec settings (`ParquetCodecFactory`). So `parquet.block.size`, `parquet.page.size`, `parquet.writer.version`, `parquet.enable.summary-metadata`, the reader-side keys, and `parquet.enable.dictionary` in the Hadoop configuration cannot change a data file, yet `requireNoUnvettedParquetMrProperties` and `requireNoParquetHadoopConfOverrides` decline all of them. With `S3FileIO`, the writer's Hadoop configuration is a fresh `new Configuration()`, so session `spark.hadoop.parquet.*` settings never reach it. Checked against the Iceberg 1.5.2, 1.8.1, 1.10.0 and 1.11.0 sources. ### Describe the potential solution - Remove `requireShredVariantsDisabled`, and add a `CometIcebergWriteDetectionSuite` case showing that a `variant` column still falls back with shredding on. - Remove the `io-impl` property rule. - Replace the two `parquet.*` deny-alls with an explicit list of the parquet-mr keys iceberg-java's writer does not read, built from parquet-mr's `ParquetOutputFormat` and `ParquetInputFormat` constants or a vetted static list, and keep failing closed on keys outside the list. `parquet.enable.dictionary` (as a table property) and `parquet.columnindex.truncate.length` stay gated until #6954 translates them. Re-check the list against `Parquet.java` and `ParquetWriter.java` on every Iceberg version bump. ### Additional context Part of #5643 and the native write epic #5649. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
