mbutrovich commented on code in PR #7491:
URL: https://github.com/apache/datafusion-comet/pull/7491#discussion_r4589182434


##########
native/proto/src/proto/operator.proto:
##########
@@ -410,7 +410,12 @@ message IcebergScanCommon {
   // Schema to read
   repeated SparkStructField required_schema = 3;
 
-  // Deduplication pools (must contain all entries for cross-partition 
deduplication)
+  // Deduplication pools. In the common block every task gets, the per-file 
pools
+  // (partition_data_pool, delete_files_pool, residual_pool, delete_file_pool,
+  // delete_file_path_pool) are empty: each partition's IcebergScan.common 
holds the entries its
+  // tasks use, and the JVM merges the two before the plan reaches native. The 
driver ships a
+  // partition's delete files and their paths apart from its message, shared 
with the other
+  // partitions that use them, and the executor JVM adds them to the 
partition's pools.

Review Comment:
   This says the driver ships a partition's delete files and their paths apart 
from its message. Residuals ship apart too: they live in 
`IcebergPartitionPlanData.residuals`, and `bytes` appends them to 
`residual_pool` on the executor ([`PartitionPlanData.scala` line 
103](https://github.com/apache/datafusion-comet/blob/b95fef2d45d0b3422557f070728843fe998bb77e/spark/src/main/scala/org/apache/spark/sql/comet/PartitionPlanData.scala#L103)).
 As written, a reader would expect `residual_pool` in the partition's 
`IcebergScan.common` on the wire. Could we name residuals here?
   
   ```suggestion
     // Deduplication pools. In the common block every task gets, the per-file 
pools
     // (partition_data_pool, delete_files_pool, residual_pool, 
delete_file_pool,
     // delete_file_path_pool) are empty: each partition's IcebergScan.common 
holds the entries its
     // tasks use, and the JVM merges the two before the plan reaches native. 
The driver ships a
     // partition's delete files, their paths, and its residuals apart from its 
message, shared with
     // the other partitions that use them, and the executor JVM adds them to 
the partition's pools.
   ```



##########
spark/src/test/scala/org/apache/spark/sql/comet/PlanDataInjectorSuite.scala:
##########
@@ -126,6 +128,247 @@ class PlanDataInjectorSuite extends AnyFunSuite {
     assert(result.getChildren(0).getPlanId == 2)
   }
 
+  /**
+   * One partition of a bucketed merge-on-read scan as the driver ships it: 
two tasks, and in its
+   * IcebergScan.common only the delete files, paths and partition value they 
use.
+   */
+  private def bucketPartition(bucket: Int): OperatorOuterClass.IcebergScan = {
+    val pools = OperatorOuterClass.IcebergScanCommon.newBuilder()
+    (0 until 2).foreach { d =>
+      
pools.addDeleteFilePathPool(f"s3://bucket/table/data/b$bucket%08d-$d-deletes.parquet")
+      pools.addDeleteFilePool(
+        OperatorOuterClass.IcebergDeleteFile
+          .newBuilder()
+          .setFilePathIdx(d)
+          .setContentType("POSITION_DELETES")
+          .setFileFormat("PARQUET")
+          .setRecordCount(10))
+    }
+    pools.addDeleteFilesPool(
+      OperatorOuterClass.DeleteFileList
+        .newBuilder()
+        .addDeleteFileIndices(0)
+        .addDeleteFileIndices(1))
+    pools.addPartitionDataPool(
+      OperatorOuterClass.PartitionData
+        .newBuilder()
+        .addValues(
+          OperatorOuterClass.PartitionValue
+            .newBuilder()
+            .setFieldId(1000)
+            
.setLiteral(OperatorOuterClass.IcebergLiteral.newBuilder().setIntVal(bucket))))
+    val scan = OperatorOuterClass.IcebergScan.newBuilder().setCommon(pools)
+    (0 until 2).foreach { f =>
+      scan.addFileScanTasks(
+        OperatorOuterClass.IcebergFileScanTask
+          .newBuilder()
+          .setDataFilePath(f"s3://bucket/table/data/b$bucket%08d-$f.parquet")
+          .setDeleteFilesIdx(0)
+          .setPartitionDataIdx(0))
+    }
+    scan.build()
+  }
+
+  test("an injected Iceberg plan holds the common block plus its own 
partition's pools") {
+    // The common block holds no per-file pools, so a task's plan carries only 
the delete files
+    // and partition values its own files use, however many other partitions 
the scan has.
+    val scanOp = icebergScanOp("s3://table/metadata/v1.json", scanHashCode = 7)
+    val key = IcebergPlanDataInjector.getKey(scanOp).get
+    val (commonBytes, _) =
+      icebergPlanData("s3://table/metadata/v1.json", 7, Seq("id", "v"), 
"unused.parquet")
+    val partition = bucketPartition(42)
+    val injected = PlanDataInjector
+      .injectPlanData(
+        parseBasePlan(scanOp.toByteArray),
+        Map(key -> commonBytes),
+        Map(key -> partition.toByteArray))
+      .getIcebergScan
+
+    val pools = injected.getCommon
+    assert(pools.getRequiredSchemaCount == 2, "the common block's fields must 
survive the merge")
+    assert(pools.getDeleteFilePoolCount == 2 && 
pools.getDeleteFilePathPoolCount == 2)
+    injected.getFileScanTasksList.asScala.foreach { task =>
+      val deletes = 
pools.getDeleteFilesPool(task.getDeleteFilesIdx).getDeleteFileIndicesList
+      val paths = deletes.asScala.map(d =>
+        pools.getDeleteFilePathPool(pools.getDeleteFilePool(d).getFilePathIdx))
+      assert(paths == Seq(0, 1).map(d => 
f"s3://bucket/table/data/b00000042-$d-deletes.parquet"))
+      assert(
+        pools
+          .getPartitionDataPool(task.getPartitionDataIdx)
+          .getValues(0)
+          .getLiteral
+          .getIntVal == 42)
+    }
+    val planSize =
+      
PlanDataInjector.serializeOperator(scanOp.toBuilder.setIcebergScan(injected).build()).length
+    assert(
+      planSize <= commonBytes.length + partition.getSerializedSize + 64,
+      s"the injected plan ($planSize bytes) should hold only the common block 
and this partition")
+  }
+
+  /**
+   * One partition whose tasks each use all of `deleteFiles` and `residual`, 
as the driver builds
+   * it: they are shared arrays and path strings, and the partition's own 
message indexes into
+   * them.
+   */
+  private def sharedDeletePartition(
+      dataFiles: Seq[String],
+      deleteFiles: Seq[(String, Array[Byte])],
+      residual: Option[Array[Byte]] = None): IcebergPartitionPlanData = {
+    val sets = OperatorOuterClass.DeleteFileList.newBuilder()
+    deleteFiles.indices.foreach(i => sets.addDeleteFileIndices(i))
+    val scan = OperatorOuterClass.IcebergScan
+      .newBuilder()
+      
.setCommon(OperatorOuterClass.IcebergScanCommon.newBuilder().addDeleteFilesPool(sets))
+    dataFiles.foreach { path =>
+      val task = OperatorOuterClass.IcebergFileScanTask
+        .newBuilder()
+        .setDataFilePath(path)
+        .setDeleteFilesIdx(0)
+      residual.foreach(_ => task.setResidualIdx(0))
+      scan.addFileScanTasks(task)
+    }
+    new IcebergPartitionPlanData(
+      scan.build().toByteArray,
+      deleteFiles.map(_._2).toArray,
+      deleteFiles.map(_._1).toArray,
+      residual.toArray)
+  }
+
+  private def deletionVector(dataFile: String, offset: Long): Array[Byte] =
+    OperatorOuterClass.IcebergDeleteFile
+      .newBuilder()
+      .setContentType("POSITION_DELETES")
+      .setFileFormat("PUFFIN")
+      .setReferencedDataFile(dataFile)
+      .setContentOffset(offset)
+      .setContentSizeInBytes(40)
+      .setRecordCount(3)
+      .build()
+      .toByteArray
+
+  test("partitions sharing delete files resolve each path through their own 
pool") {
+    // Two deletion vectors share one Puffin file, and an equality delete has 
its own. The
+    // partitions list them in different orders, so each sets its own path 
indices.
+    val puffin = "s3://bucket/table/data/dv.puffin"
+    val eqPath = "s3://bucket/table/data/eq-deletes.parquet"
+    val dvA = (puffin, deletionVector("a.parquet", 4))
+    val dvB = (puffin, deletionVector("b.parquet", 44))
+    val eq = (
+      eqPath,
+      OperatorOuterClass.IcebergDeleteFile
+        .newBuilder()
+        .setContentType("EQUALITY_DELETES")
+        .setFileFormat("PARQUET")
+        .addEqualityIds(1)
+        .setRecordCount(5)
+        .build()
+        .toByteArray)
+    val scanOp = icebergScanOp("s3://table/metadata/v1.json", scanHashCode = 7)
+    val key = IcebergPlanDataInjector.getKey(scanOp).get
+    val (commonBytes, _) =
+      icebergPlanData("s3://table/metadata/v1.json", 7, Seq("id"), 
"unused.parquet")
+
+    Seq(Seq(eq, dvA), Seq(dvB, eq, dvA)).foreach { deletes =>
+      val data = sharedDeletePartition(Seq("a.parquet"), deletes)
+      val injected = PlanDataInjector
+        .injectPlanData(
+          parseBasePlan(scanOp.toByteArray),
+          Map(key -> commonBytes),
+          Map(key -> data.bytes))
+        .getIcebergScan
+      val pools = injected.getCommon
+      assert(
+        pools.getDeleteFilePathPoolList.asScala.toSet == 
deletes.map(_._1).toSet &&
+          pools.getDeleteFilePathPoolCount == deletes.map(_._1).distinct.size,
+        s"each path should be pooled once: ${pools.getDeleteFilePathPoolList}")
+      val task = injected.getFileScanTasks(0)
+      val resolved = pools
+        .getDeleteFilesPool(task.getDeleteFilesIdx)
+        .getDeleteFileIndicesList
+        .asScala
+        .map { i =>
+          val file = pools.getDeleteFilePool(i)
+          (pools.getDeleteFilePathPool(file.getFilePathIdx), 
file.getContentOffset)
+        }
+      val expected = deletes.map { case (path, bytes) =>
+        (path, 
OperatorOuterClass.IcebergDeleteFile.parseFrom(bytes).getContentOffset)
+      }
+      assert(resolved == expected)
+    }
+  }
+
+  test("a coalesced task carries a delete file and residual its partitions 
share once") {
+    // A coalesce without a shuffle writes all of its input partitions into 
one task stream. A
+    // delete file or residual those partitions share is one array, so Java 
serialization writes
+    // it once.
+    val keyMetadata = com.google.protobuf.ByteString.copyFrom(new 
Array[Byte](8 * 1024))
+    val shared = (
+      "s3://bucket/table/data/eq-deletes.parquet",
+      OperatorOuterClass.IcebergDeleteFile
+        .newBuilder()
+        .setContentType("EQUALITY_DELETES")
+        .setFileFormat("PARQUET")
+        .addEqualityIds(1)
+        .setKeyMetadata(keyMetadata)
+        .build()
+        .toByteArray)
+    // A large IN list, the residual of a filter every file in the scan gets.
+    val inList = OperatorOuterClass.IcebergSetPredicate
+      .newBuilder()
+      .setColumn("id")
+      .setOp(OperatorOuterClass.IcebergPredicateOperator.In)
+    (0 until 2000).foreach { v =>
+      
inList.addValues(OperatorOuterClass.IcebergLiteral.newBuilder().setLongVal(v))

Review Comment:
   The strict Scala warnings job fails on this line with `implicit numeric 
widening`, because `v` is an `Int` and `setLongVal` takes a `long` ([job 
log](https://github.com/parthchandra/datafusion-comet/actions/runs/43762205581/job/132960394401)).
 That failure is what turns the fork run's `Required Checks` red.
   
   ```suggestion
         
inList.addValues(OperatorOuterClass.IcebergLiteral.newBuilder().setLongVal(v.toLong))
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to