snmvaughan opened a new pull request, #5410:
URL: https://github.com/apache/datafusion-comet/pull/5410
CometIcebergNativeWrite's `requireNoParquetHadoopConfOverrides` (added in
#5298) walks the entire session Hadoop Configuration and returns an
`Unsupported` fall-back reason for any key starting with `parquet.`. The intent
is sound (a `parquet.*` key in the Hadoop conf reaches iceberg-java's writer
but not Comet's native writer), but the predicate is too broad: it checks
reader-side `parquet.*` keys the same as writer-side ones.
`parquet.hadoop.vectored.io.enabled` is a reader-side vectored-IO knob:
- Declared in parquet-hadoop as
`org.apache.parquet.hadoop.ParquetInputFormat.HADOOP_VECTORED_IO_ENABLED` with
default `true` in parquet-hadoop 1.16+.
- Only consulted by parquet-mr's Hadoop reader path (via
`HadoopReadOptions.useHadoopVectoredIo`); iceberg-java's writer never reads it.
Any environment that seeds it into the session Hadoop configuration trips
this rule, and every otherwise-Compatible Iceberg V2 write is silently
mis-classified as Unsupported — the native writer is disabled without the user
having changed any writer-relevant setting.
## Which issue does this PR close?
Closes #.
## Rationale for this change
Avoid mis-classifying Compatible Iceberg V2 writes as Unsupported due to
well-known parquet reader settings.
## What changes are included in this PR?
Add an `IgnoredHadoopParquetConfKeys` set containing
`parquet.hadoop.vectored.io.enabled` and consult it inside the `parquet.*` walk
before emitting the fall-back reason. The broad `startsWith("parquet.")` gate
remains in place for every writer-relevant knob.
## How are these changes tested?
Add a regression test that pins
`parquet.hadoop.vectored.io.enabled=true` in the session Hadoop
configuration via `withSQLConf` and asserts Compatible. The existing negative
test ("fall-back: parquet.* key in the session Hadoop configuration", keyed on
`parquet.block.size`) continues to exercise the rejection path for real
writer-relevant `parquet.*` keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
