andygrove commented on code in PR #7520:
URL: https://github.com/apache/datafusion-comet/pull/7520#discussion_r4580018526


##########
.github/workflows/ci.yml:
##########
@@ -619,6 +657,22 @@ jobs:
       spark-full: '4.1.3'
       java: 17
 
+  iceberg_1_12:

Review Comment:
   You're right, it would not have caught it. Fixed in 6e991c518. 
`summarize-iceberg-writes.py` has a new `--require-native` flag that exits 
non-zero, after printing the summary, when none of the recorded writes ran on 
the native writer. That includes the case where no writes were recorded at all. 
The cross-shard summary in `iceberg-spark-shard-coverage` and the extensions 
job pass it, so the guard covers every Iceberg version, not just 1.12. Before 
relying on it, I ran the summarizer on last night's shard-1 reports. 1.9.1 had 
316 native writes (97.5%) and 1.10.0 had 340 (97.7%). The 1.10.0 extensions run 
had 2927 (37.5%). So no version is near zero. A shard's own summary still never 
fails, because a single shard can legitimately record no writes. Unit tests for 
the flag are in `test-summarize-iceberg-writes.py`.



##########
spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala:
##########
@@ -1949,18 +1935,38 @@ object IcebergReflection extends Logging {
    *
    * `specId` is stamped onto the synthesised `ManifestFile` because the 
manifest reader uses it
    * to pick the correct partition spec when materialising partition data on 
each `DataFile`.
-   */
-  def decodeManifestToDataFiles(bytes: Array[Byte], specId: Int): 
java.util.List[AnyRef] = {
+   * `spec` is the table's `PartitionSpec` for that id, handed to the reader 
through the
+   * `specsById` argument so it is not re-parsed from the manifest's metadata 
(Iceberg deprecated
+   * that fallback, and 1.12 removed the `read(ManifestFile, FileIO)` overload 
that implied it).
+   */
+  def decodeManifestToDataFiles(
+      bytes: Array[Byte],
+      specId: Int,
+      spec: AnyRef): java.util.List[AnyRef] = {
     // Iceberg's `ManifestReader.open` infers format from the file extension; 
the in-memory path
     // must end in `.avro` for the v2 data-manifest path to be picked. The 
"memory:" scheme just
     // namespaces the location so it can't collide with on-disk paths in the 
same FileIO.
     val location = s"memory:comet-manifest-${java.util.UUID.randomUUID()}.avro"
     val fileIO = newInMemoryFileIO(location, bytes)
     val inputFile = newInMemoryInputFile(location, bytes)
     val manifestFile = newDataManifestFile(inputFile, specId)
-    readDataFilesFromManifest(manifestFile, fileIO)
+    val specsById = 
java.util.Collections.singletonMap(Integer.valueOf(specId), spec)
+    readDataFilesFromManifest(manifestFile, fileIO, specsById)
   }
 
+  /**
+   * `ManifestFiles.read(ManifestFile, FileIO, Map<Integer, PartitionSpec>)`, 
the only overload
+   * that is public on every supported Iceberg release (1.5.2 through 1.12; 
1.12 dropped the
+   * two-argument one).
+   */
+  private def manifestReadMethodOf(fileIOClass: Class[_]): 
java.lang.reflect.Method =

Review Comment:
   Done in 6e991c518. `manifestReadMethodOf` now goes through the cached 
`getMethod(clazz, name, paramTypes*)`, and the comment says 1.8.1 through 1.12.



##########
spark/src/test/java/org/apache/comet/DetectionDelegatingFileIO.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet;
+
+import java.util.Map;
+import java.util.function.Function;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.util.SerializableSupplier;
+
+/**
+ * A FileIO that works normally (delegating to HadoopFileIO) but whose class 
is not on Comet's
+ * recognized-FileIO allowlist. Composition rather than inheritance is the 
point: a HadoopFileIO
+ * SUBCLASS passes the hierarchy check by design, while this class must be 
declined. Instantiated
+ * reflectively by Iceberg's `CatalogUtil.loadFileIO`, hence public with a 
no-arg constructor.
+ *
+ * <p>This is Java because {@code getConf} is abstract on some supported 
Iceberg versions (1.8.1)
+ * and a default method on others (1.12). Scala has no spelling that compiles 
against both, since it
+ * requires {@code override} for a concrete member and rejects it for an 
abstract one.
+ */
+public class DetectionDelegatingFileIO implements FileIO, HadoopConfigurable {
+  private final HadoopFileIO delegate = new HadoopFileIO();
+
+  @Override
+  public InputFile newInputFile(String path) {
+    return delegate.newInputFile(path);
+  }
+
+  @Override
+  public OutputFile newOutputFile(String path) {
+    return delegate.newOutputFile(path);
+  }
+
+  @Override
+  public void deleteFile(String path) {
+    delegate.deleteFile(path);
+  }
+
+  @Override
+  public void initialize(Map<String, String> properties) {
+    delegate.initialize(properties);
+  }
+
+  @Override
+  public void setConf(Configuration conf) {
+    delegate.setConf(conf);
+  }
+
+  public Configuration getConf() {

Review Comment:
   You're right, the Java wasn't needed. In 6e991c518 the class is back in 
`CometIcebergWriteDetectionSuite.scala` with `override def getConf`. I compiled 
that class on its own with scalac 2.12.18 and 2.13.18 against iceberg-core 
1.8.1, 1.10.0, 1.11.0 and 1.12.0, and every combination compiles. Without 
`override`, 1.12.0 fails with "needs `override' modifier" against 
`HadoopConfigurable`'s default `getConf`. The suite also test-compiles under 
the default, `spark-3.5` and `spark-4.2` profiles.



##########
docs/source/user-guide/latest/iceberg.md:
##########
@@ -36,7 +36,7 @@ then serialized to Comet's native execution engine (see
 [PR #2528](https://github.com/apache/datafusion-comet/pull/2528)).
 
 The example below uses Spark's package downloader to retrieve Comet 
$COMET_VERSION and Iceberg
-1.8.1, but Comet has been tested with Iceberg 1.5, 1.8, 1.9, 1.10, and 1.11. 
The native Iceberg
+1.8.1, but Comet has been tested with Iceberg 1.5, 1.8, 1.9, 1.10, 1.11, and 
1.12. The native Iceberg

Review Comment:
   Reworded in 6e991c518. The paragraph now says that Comet's own suites run 
against Iceberg 1.8, 1.10 and 1.11, and that Iceberg's Spark suites run with 
Comet against 1.9, 1.10, 1.11 and 1.12. It also says 1.12 is tested only 
through Iceberg's suites, with Spark 4.2. The sentence used to list 1.5 as 
well, which went stale with #7521, so I dropped it.



##########
dev/ci/compute-changes.py:
##########
@@ -385,6 +443,31 @@
         ".mvn/**",
         "mvnw",
     ],
+    "iceberg_1_12": [

Review Comment:
   Done in 6e991c518. `iceberg_1_9` holds the list, and `iceberg_1_10`, 
`iceberg_1_11` and `iceberg_1_12` are filled in after the dict, the same way as 
`spark_4_1_hive`. I dumped `FILTERS` and `POLICY` before and after the change, 
and the JSON is byte-identical. `check-ci-config.py` passes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to