rdblue commented on a change in pull request #1888:
URL: https://github.com/apache/iceberg/pull/1888#discussion_r540596966
##########
File path: core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java
##########
@@ -75,6 +81,135 @@ public WriteResult complete() throws IOException {
.build();
}
+ /**
+ * Base equality delta writer to write both insert records and
equality-deletes.
+ */
+ protected abstract class BaseEqualityDeltaWriter implements Closeable {
+ private final StructProjection structProjection;
+ private RollingFileWriter dataWriter;
+ private RollingEqDeleteWriter eqDeleteWriter;
+ private SortedPosDeleteWriter<T> posDeleteWriter;
+ private Map<StructLike, PathOffset> insertedRowMap;
+
+ public BaseEqualityDeltaWriter(PartitionKey partition, Schema schema,
Schema deleteSchema) {
+ Preconditions.checkNotNull(schema, "Iceberg table schema cannot be
null.");
+ Preconditions.checkNotNull(deleteSchema, "Equality-delete schema cannot
be null.");
+ this.structProjection = StructProjection.create(schema, deleteSchema);
+
+ this.dataWriter = new RollingFileWriter(partition);
+
+ this.eqDeleteWriter = new RollingEqDeleteWriter(partition);
+ this.insertedRowMap = StructLikeMap.create(deleteSchema.asStruct());
+
+ this.posDeleteWriter = new SortedPosDeleteWriter<>(appenderFactory,
fileFactory, format, partition);
+ }
+
+ /**
+ * Make the generic data could be read as a {@link StructLike}.
+ */
+ protected abstract StructLike asStructLike(T data);
+
+ public void write(T row) throws IOException {
+ PathOffset pathOffset = PathOffset.of(dataWriter.currentPath(),
dataWriter.currentRows());
+
+ StructLike copiedKey = structProjection.copy().wrap(asStructLike(row));
+ // Adding a pos-delete to replace the old path-offset.
+ PathOffset previous = insertedRowMap.put(copiedKey, pathOffset);
Review comment:
Why did you decide to delegate to the implementation instead of creating
a copy here? I think it would be fairly easy to copy with a StructLike class:
```java
public static class StructCopy implements StructLike {
public static StructLike copy(StructLike struct) {
return new StructCopy(struct);
}
private final Object[] values;
private StructCopy(StructLike toCopy) {
this.values = new Object[toCopy.size()];
for (int i = 0; i < values.length; i += 1) {
Object value = toCopy.get(i, Object.class);
if (value instanceof StructLike) {
values[i] = copy((StructLike) value);
} else {
values[i] = value;
}
}
}
@Override
public int size() {
return values.length;
}
@Override
public <T> T get(int pos, Class<T> javaClass) {
return javaClass.cast(values[pos]);
}
@Override
public <T> void set(int pos, T value) {
throw new UnsupportedOperationException("Struct copy cannot be
modified");
}
}
```
That doesn't handle lists or maps, but the projection doesn't handle lists
or maps either, so it would be okay to use that in the StructLikeMap for all of
the cases we currently support.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
