laskoviymishka commented on code in PR #16956:
URL: https://github.com/apache/iceberg/pull/16956#discussion_r4618666392


##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/RowDataReader.java:
##########
@@ -67,6 +73,8 @@ class RowDataReader extends BaseRowReader<FileScanTask> 
implements PartitionRead
 
     numSplits = taskGroup.tasks().size();
     LOG.debug("Reading {} file split(s) for table {}", numSplits, 
table.name());
+
+    sharedEqDeletes = new EqualityDeletes(table.specs());

Review Comment:
   Could sharing sit behind a Spark SQL conf or read option, off by default? 
`GenericReader` already gates it on `shareEqDeletes` 
([GenericReader.java](https://github.com/apache/iceberg/blob/fec3a23f7cae0d862d8a10262200a6f7bfb3186b/data/src/main/java/org/apache/iceberg/data/GenericReader.java#L46-L55)),
 but here and in `BatchDataReader` the map is always built, and 
`SparkSQLProperties` / `SparkReadOptions` have no key for it. Off by default 
lets CDC users opt in once they've seen the memory profile on their own 
workload.



##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/RowDataReader.java:
##########
@@ -45,6 +46,11 @@ class RowDataReader extends BaseRowReader<FileScanTask> 
implements PartitionRead
 
   private final long numSplits;
 
+  // Shared across every data file in this task group so an equality delete 
file referenced by
+  // more than one of them is read and materialized only once. Not thread-safe 
because open()
+  // is called sequentially by the single task thread, so these are mutated by 
one thread.
+  private final EqualityDeletes sharedEqDeletes;

Review Comment:
   Could we get a peak-heap number for a task spanning several partitions, base 
vs head, before this merges? `EqualityDeletes` only grows 
([EqualityDeletes.java](https://github.com/apache/iceberg/blob/fec3a23f7cae0d862d8a10262200a6f7bfb3186b/data/src/main/java/org/apache/iceberg/data/EqualityDeletes.java#L33-L73)),
 so every partition's keys stay live until the task ends, while `main` drops 
each file's set once the next file opens 
([base](https://github.com/apache/iceberg/blob/411c38b58c5208dd8e972a394f0e409db32458ac/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/RowDataReader.java#L84-L99)).
 Split size caps compressed delete bytes, not heap, and for 
`rewrite_data_files` the split is at least the target file size 
([planner](https://github.com/apache/iceberg/blob/fec3a23f7cae0d862d8a10262200a6f7bfb3186b/core/src/main/java/org/apache/iceberg/actions/SizeBasedFileRewritePlanner.java#L212-L219)).



##########
data/src/test/java/org/apache/iceberg/data/TestGenericReaderDeletes.java:
##########
@@ -36,6 +41,19 @@
 public class TestGenericReaderDeletes extends DeleteReadTests {
   @TempDir private File tableDir;
 
+  @Parameter(index = 2)
+  private boolean shareEqDeletes;
+
+  @Parameters(name = "fileFormat = {0}, formatVersion = {1}, shareEqDeletes = 
{2}")
+  public static Object[][] parameters() {
+    List<Object[]> parameters = Lists.newArrayList();
+    for (Object[] params : DeleteReadTests.parameters()) {
+      parameters.add(new Object[] {params[0], params[1], false});
+    }
+    parameters.add(new Object[] {FileFormat.PARQUET, 3, true});

Review Comment:
   Could we add a `DeleteReadTests` case where a data file is appended after an 
equality delete in the same partition, and both are read in one task? Every 
`newRowDelta()` there comes after all appends 
([DeleteReadTests.java](https://github.com/apache/iceberg/blob/fec3a23f7cae0d862d8a10262200a6f7bfb3186b/data/src/test/java/org/apache/iceberg/data/DeleteReadTests.java#L343-L386)),
 so nothing checks that a key already in the shared map leaves the newer file 
alone. A spec-evolution case and a global eq delete would cover the rest of the 
map key.



##########
core/src/main/java/org/apache/iceberg/util/BinPacking.java:
##########
@@ -26,12 +26,54 @@
 import java.util.List;
 import java.util.NoSuchElementException;
 import java.util.function.Function;
+import java.util.function.Supplier;
 import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
 import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
 import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
 import org.apache.iceberg.relocated.com.google.common.collect.Lists;
 
 public class BinPacking {
+
+  /**
+   * Computes the weight an item contributes to a bin, with awareness of the 
bin's current contents.
+   * A new instance is created for each bin, allowing stateful deduplication 
of shared resources
+   * (e.g. equality delete files that apply to multiple data files in the same 
bin).
+   */
+  public interface BinWeighter<T> {

Review Comment:
   Could `BinWeighter`, `StatelessBinWeighter` and `ScanTaskBinWeighter` be 
package-private? All their users are in `org.apache.iceberg.util`, and 
`ScanTaskBinWeighter` is already public with a package-private constructor 
([TableScanUtil.java](https://github.com/apache/iceberg/blob/fec3a23f7cae0d862d8a10262200a6f7bfb3186b/core/src/main/java/org/apache/iceberg/util/TableScanUtil.java#L56-L62)),
 so nothing outside the package can build one. This applies wherever the 
weighter change ends up landing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to