cornelcreanga opened a new issue, #7240:
URL: https://github.com/apache/datafusion-comet/issues/7240
### Describe the bug
With iceberg-spark-runtime 1.12.0 every native Iceberg write falls back,
logging only:
```
Comet cannot accelerate IcebergWriteExec because: executor-side Iceberg
reflection did not resolve:
NoSuchMethodException:
org.apache.iceberg.ManifestFiles.read(org.apache.iceberg.ManifestFile,org.apache.iceberg.io.FileIO)
```
Iceberg 1.12.0 removed the public 2-arg overload (javap on
iceberg-spark-runtime-3.5_2.12):
```
1.11.0: public static ManifestReader<DataFile> read(ManifestFile, FileIO)
public static ManifestReader<DataFile> read(ManifestFile, FileIO,
Map<Integer, PartitionSpec>)
1.12.0: public static ManifestReader<DataFile> read(ManifestFile, FileIO,
Map<Integer, PartitionSpec>)
```
Comet resolves the 2-arg form in two places in `IcebergReflection.scala`:
the executor probe
(`probeExecutorReflection`, line 1836 on main) and
`readDataFilesFromManifest` (line 2004). The
native Iceberg scan still works on 1.12.
### Steps to reproduce
Any `rewrite_data_files` or `INSERT` into an Iceberg table with
`spark.comet.iceberg.write.enabled=true`
and iceberg-spark-runtime 1.12.0 on the classpath. The plan has no
`CometIcebergWrite`, and the
WARN above is the only signal.
### Expected behavior
Native writes work on Iceberg 1.12. The 3-arg `read(manifest, io,
specsById)` is public on every
version Comet supports, and passing the table's `specs()` (or a single-entry
map for the stamped
`specId`) would work on 1.5.2 through 1.12.
### Additional context
Comet 1.1.0, Spark 3.5.6. Works on Iceberg 1.11.0. The Spark profiles in
`spark/pom.xml` pin
1.5.2 / 1.8.1 / 1.10.0 / 1.11.0, so no CI profile covers 1.12; adding one
would have caught this.
Related: #4871, #5308, #5649, #7040.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
