laskoviymishka commented on code in PR #16956:
URL: https://github.com/apache/iceberg/pull/16956#discussion_r4618666392
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/RowDataReader.java:
##########
@@ -67,6 +73,8 @@ class RowDataReader extends BaseRowReader<FileScanTask>
implements PartitionRead
numSplits = taskGroup.tasks().size();
LOG.debug("Reading {} file split(s) for table {}", numSplits,
table.name());
+
+ sharedEqDeletes = new EqualityDeletes(table.specs());
Review Comment:
Could sharing sit behind a Spark SQL conf or read option, off by default?
`GenericReader` already gates it on `shareEqDeletes`
([GenericReader.java](https://github.com/apache/iceberg/blob/fec3a23f7cae0d862d8a10262200a6f7bfb3186b/data/src/main/java/org/apache/iceberg/data/GenericReader.java#L46-L55)),
but here and in `BatchDataReader` the map is always built, and
`SparkSQLProperties` / `SparkReadOptions` have no key for it. Off by default
lets CDC users opt in once they've seen the memory profile on their own
workload.
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/RowDataReader.java:
##########
@@ -45,6 +46,11 @@ class RowDataReader extends BaseRowReader<FileScanTask>
implements PartitionRead
private final long numSplits;
+ // Shared across every data file in this task group so an equality delete
file referenced by
+ // more than one of them is read and materialized only once. Not thread-safe
because open()
+ // is called sequentially by the single task thread, so these are mutated by
one thread.
+ private final EqualityDeletes sharedEqDeletes;
Review Comment:
Could we get a peak-heap number for a task spanning several partitions, base
vs head, before this merges? `EqualityDeletes` only grows
([EqualityDeletes.java](https://github.com/apache/iceberg/blob/fec3a23f7cae0d862d8a10262200a6f7bfb3186b/data/src/main/java/org/apache/iceberg/data/EqualityDeletes.java#L33-L73)),
so every partition's keys stay live until the task ends, while `main` drops
each file's set once the next file opens
([base](https://github.com/apache/iceberg/blob/411c38b58c5208dd8e972a394f0e409db32458ac/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/RowDataReader.java#L84-L99)).
Split size caps compressed delete bytes, not heap, and for
`rewrite_data_files` the split is at least the target file size
([planner](https://github.com/apache/iceberg/blob/fec3a23f7cae0d862d8a10262200a6f7bfb3186b/core/src/main/java/org/apache/iceberg/actions/SizeBasedFileRewritePlanner.java#L212-L219)).
##########
data/src/test/java/org/apache/iceberg/data/TestGenericReaderDeletes.java:
##########
@@ -36,6 +41,19 @@
public class TestGenericReaderDeletes extends DeleteReadTests {
@TempDir private File tableDir;
+ @Parameter(index = 2)
+ private boolean shareEqDeletes;
+
+ @Parameters(name = "fileFormat = {0}, formatVersion = {1}, shareEqDeletes =
{2}")
+ public static Object[][] parameters() {
+ List<Object[]> parameters = Lists.newArrayList();
+ for (Object[] params : DeleteReadTests.parameters()) {
+ parameters.add(new Object[] {params[0], params[1], false});
+ }
+ parameters.add(new Object[] {FileFormat.PARQUET, 3, true});
Review Comment:
Could we add a `DeleteReadTests` case where a data file is appended after an
equality delete in the same partition, and both are read in one task? Every
`newRowDelta()` there comes after all appends
([DeleteReadTests.java](https://github.com/apache/iceberg/blob/fec3a23f7cae0d862d8a10262200a6f7bfb3186b/data/src/test/java/org/apache/iceberg/data/DeleteReadTests.java#L343-L386)),
so nothing checks that a key already in the shared map leaves the newer file
alone. A spec-evolution case and a global eq delete would cover the rest of the
map key.
##########
core/src/main/java/org/apache/iceberg/util/BinPacking.java:
##########
@@ -26,12 +26,54 @@
import java.util.List;
import java.util.NoSuchElementException;
import java.util.function.Function;
+import java.util.function.Supplier;
import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
import org.apache.iceberg.relocated.com.google.common.collect.Lists;
public class BinPacking {
+
+ /**
+ * Computes the weight an item contributes to a bin, with awareness of the
bin's current contents.
+ * A new instance is created for each bin, allowing stateful deduplication
of shared resources
+ * (e.g. equality delete files that apply to multiple data files in the same
bin).
+ */
+ public interface BinWeighter<T> {
Review Comment:
Could `BinWeighter`, `StatelessBinWeighter` and `ScanTaskBinWeighter` be
package-private? All their users are in `org.apache.iceberg.util`, and
`ScanTaskBinWeighter` is already public with a package-private constructor
([TableScanUtil.java](https://github.com/apache/iceberg/blob/fec3a23f7cae0d862d8a10262200a6f7bfb3186b/core/src/main/java/org/apache/iceberg/util/TableScanUtil.java#L56-L62)),
so nothing outside the package can build one. This applies wherever the
weighter change ends up landing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]