openinx commented on a change in pull request #1888:
URL: https://github.com/apache/iceberg/pull/1888#discussion_r539084806
##########
File path: core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java
##########
@@ -75,6 +79,113 @@ public WriteResult complete() throws IOException {
.build();
}
+ /**
+ * Base delta writer to write both insert records and equality-deletes.
+ */
+ protected abstract class BaseDeltaWriter implements Closeable {
+ private RollingFileWriter dataWriter;
+ private RollingEqDeleteWriter eqDeleteWriter;
+ private SortedPosDeleteWriter<T> posDeleteWriter;
+ private StructLikeMap<PathOffset> insertedRowMap;
+
+ public BaseDeltaWriter(PartitionKey partition, Schema eqDeleteSchema) {
+ Preconditions.checkNotNull(eqDeleteSchema, "equality-delete schema could
not be null.");
+
+ this.dataWriter = new RollingFileWriter(partition);
+
+ this.eqDeleteWriter = new RollingEqDeleteWriter(partition);
+ this.insertedRowMap = StructLikeMap.create(eqDeleteSchema.asStruct());
+
+ this.posDeleteWriter = new SortedPosDeleteWriter<>(appenderFactory,
fileFactory, format, partition);
+ }
+
+ /**
+ * Make the generic data could be read as a {@link StructLike}.
+ */
+ protected abstract StructLike asStructLike(T data);
+
+ protected abstract StructLike asCopiedKey(T row);
+
+ public void write(T row) throws IOException {
+ PathOffset pathOffset = PathOffset.of(dataWriter.currentPath(),
dataWriter.currentRows());
+
+ StructLike copiedKey = asCopiedKey(row);
+ // Adding a pos-delete to replace the old filePos.
+ PathOffset previous = insertedRowMap.put(copiedKey, pathOffset);
+ if (previous != null) {
+ // TODO attach the previous row if has a positional-delete row schema
in appender factory.
+ posDeleteWriter.delete(previous.path, previous.rowOffset, null);
+ }
+
+ dataWriter.write(row);
+ }
+
+ /**
+ * Delete the rows with the given key.
+ *
+ * @param key is the projected values which could be write to eq-delete
file directly.
+ */
+ public void delete(T key) throws IOException {
Review comment:
Thought it again, if the table has the `(a,b,c,d)` columns, and the
equality fields are `(b,c)`, then I think it's enough to provide two ways to
write the equality-delete file:
1. Only write the `(b,c)` column values into equality-delete file. That
is enough to maintain the correct deletion semantics if people don't expect to
do any real incremental pulling.
2. Write the `(a,b,c,d)` column values into equality-delete file. That's
suitable if people want to consume the incremental inserts and deletes for
future streaming analysis or data pipeline etc.
Except the above cases, I can't think of other scenarios that require a
custom equality schema.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
