comphead commented on code in PR #7520:
URL: https://github.com/apache/datafusion-comet/pull/7520#discussion_r4579934453


##########
.github/workflows/ci.yml:
##########
@@ -619,6 +657,22 @@ jobs:
       spark-full: '4.1.3'
       java: 17
 
+  iceberg_1_12:

Review Comment:
   The rationale is that no CI job covered 1.12, so nothing caught the silent 
fallback. Would this job have caught it? `dev/ci/summarize-iceberg-writes.py` 
never fails a job, and the upstream Iceberg tests do not assert which writer 
ran. I expect that reverting part 1 leaves `iceberg_1_12` green, with the 
native share only visible in the step summary. The Comet suites that would fail 
still run on 1.11 only, since the `spark-4.2` profile uses the 4.0 runtime. 
Could the job (or a Comet test) assert that at least one write ran natively, or 
that the executor reflection probe resolves against the 1.12 core jar? 
Otherwise the fix has no regression guard.



##########
spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala:
##########
@@ -1949,18 +1935,38 @@ object IcebergReflection extends Logging {
    *
    * `specId` is stamped onto the synthesised `ManifestFile` because the 
manifest reader uses it
    * to pick the correct partition spec when materialising partition data on 
each `DataFile`.
-   */
-  def decodeManifestToDataFiles(bytes: Array[Byte], specId: Int): 
java.util.List[AnyRef] = {
+   * `spec` is the table's `PartitionSpec` for that id, handed to the reader 
through the
+   * `specsById` argument so it is not re-parsed from the manifest's metadata 
(Iceberg deprecated
+   * that fallback, and 1.12 removed the `read(ManifestFile, FileIO)` overload 
that implied it).
+   */
+  def decodeManifestToDataFiles(
+      bytes: Array[Byte],
+      specId: Int,
+      spec: AnyRef): java.util.List[AnyRef] = {
     // Iceberg's `ManifestReader.open` infers format from the file extension; 
the in-memory path
     // must end in `.avro` for the v2 data-manifest path to be picked. The 
"memory:" scheme just
     // namespaces the location so it can't collide with on-disk paths in the 
same FileIO.
     val location = s"memory:comet-manifest-${java.util.UUID.randomUUID()}.avro"
     val fileIO = newInMemoryFileIO(location, bytes)
     val inputFile = newInMemoryInputFile(location, bytes)
     val manifestFile = newDataManifestFile(inputFile, specId)
-    readDataFilesFromManifest(manifestFile, fileIO)
+    val specsById = 
java.util.Collections.singletonMap(Integer.valueOf(specId), spec)
+    readDataFilesFromManifest(manifestFile, fileIO, specsById)
   }
 
+  /**
+   * `ManifestFiles.read(ManifestFile, FileIO, Map<Integer, PartitionSpec>)`, 
the only overload
+   * that is public on every supported Iceberg release (1.5.2 through 1.12; 
1.12 dropped the
+   * two-argument one).
+   */
+  private def manifestReadMethodOf(fileIOClass: Class[_]): 
java.lang.reflect.Method =

Review Comment:
   This resolves the class and the `Method` on every call, once per task in 
`decodeManifestToDataFiles` and once per plan in the probe. The same object 
already has a cached `getMethod(clazz, name, paramTypes*)` backed by 
`methodCache`. Would using it here keep this lookup consistent with the rest of 
the file and avoid the repeated `Class.getMethod` scan? The probe and the 
executor already share the helper, which is good, so only the lookup itself 
would change. The comment also says "1.5.2 through 1.12", but main no longer 
supports the 1.5.2 (Spark 3.4) profile after #7521.



##########
spark/src/test/java/org/apache/comet/DetectionDelegatingFileIO.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet;
+
+import java.util.Map;
+import java.util.function.Function;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.iceberg.hadoop.HadoopConfigurable;
+import org.apache.iceberg.hadoop.HadoopFileIO;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.InputFile;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.util.SerializableSupplier;
+
+/**
+ * A FileIO that works normally (delegating to HadoopFileIO) but whose class 
is not on Comet's
+ * recognized-FileIO allowlist. Composition rather than inheritance is the 
point: a HadoopFileIO
+ * SUBCLASS passes the hierarchy check by design, while this class must be 
declined. Instantiated
+ * reflectively by Iceberg's `CatalogUtil.loadFileIO`, hence public with a 
no-arg constructor.
+ *
+ * <p>This is Java because {@code getConf} is abstract on some supported 
Iceberg versions (1.8.1)
+ * and a default method on others (1.12). Scala has no spelling that compiles 
against both, since it
+ * requires {@code override} for a concrete member and rejects it for an 
abstract one.
+ */
+public class DetectionDelegatingFileIO implements FileIO, HadoopConfigurable {
+  private final HadoopFileIO delegate = new HadoopFileIO();
+
+  @Override
+  public InputFile newInputFile(String path) {
+    return delegate.newInputFile(path);
+  }
+
+  @Override
+  public OutputFile newOutputFile(String path) {
+    return delegate.newOutputFile(path);
+  }
+
+  @Override
+  public void deleteFile(String path) {
+    delegate.deleteFile(path);
+  }
+
+  @Override
+  public void initialize(Map<String, String> properties) {
+    delegate.initialize(properties);
+  }
+
+  @Override
+  public void setConf(Configuration conf) {
+    delegate.setConf(conf);
+  }
+
+  public Configuration getConf() {

Review Comment:
   The port keeps every override the Scala class had, and nothing else 
references the deleted Scala class. Is Java needed though? Scala only requires 
`override` when the inherited member is concrete (the 1.12 default `getConf`), 
and allows it when the member is abstract (1.8.1 through 1.11, via Hadoop's 
`Configurable`). I have not compiled it, but `override def getConf: 
Configuration = delegate.getConf` looks like it would satisfy both shapes and 
keep the class next to its only user. If the Java file is kept, the class 
comment's claim that Scala "rejects it for an abstract one" should be checked.



##########
dev/ci/compute-changes.py:
##########
@@ -385,6 +443,31 @@
         ".mvn/**",
         "mvnw",
     ],
+    "iceberg_1_12": [

Review Comment:
   This is a fourth copy of the same 24-entry path list (`iceberg_1_9`, 
`iceberg_1_10`, `iceberg_1_11`, now `iceberg_1_12`). Would it be worth defining 
the list once and referencing it, as `spark_4_1_hive` is populated after the 
dict, so a new path cannot be added to three of the four?



##########
docs/source/user-guide/latest/iceberg.md:
##########
@@ -36,7 +36,7 @@ then serialized to Comet's native execution engine (see
 [PR #2528](https://github.com/apache/datafusion-comet/pull/2528)).
 
 The example below uses Spark's package downloader to retrieve Comet 
$COMET_VERSION and Iceberg
-1.8.1, but Comet has been tested with Iceberg 1.5, 1.8, 1.9, 1.10, and 1.11. 
The native Iceberg
+1.8.1, but Comet has been tested with Iceberg 1.5, 1.8, 1.9, 1.10, 1.11, and 
1.12. The native Iceberg

Review Comment:
   "Comet has been tested with Iceberg ... 1.12" is stronger than the evidence 
in the PR. Comet's own Iceberg suites do not run on 1.12 in CI (they ran 
locally against a temporary pom edit, 204 of 206, with 
`RESTCatalogAdapter.java` not compiling on 1.12). What does run is Iceberg's 
own Spark 4.2 suite from source, nightly. Could the sentence say that, or list 
1.12 separately?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to