openinx commented on a change in pull request #1974:
URL: https://github.com/apache/iceberg/pull/1974#discussion_r549921095
##########
File path: flink/src/main/java/org/apache/iceberg/flink/sink/FlinkSink.java
##########
@@ -169,6 +172,17 @@ public Builder writeParallelism(int newWriteParallelism) {
return this;
}
+ /**
+ * Configuring the equality field columns for iceberg table that accept
CDC or UPSERT events.
+ *
+ * @param columns defines the iceberg table's key.
+ * @return {@link Builder} to connect the iceberg table.
+ */
+ public Builder equalityFieldColumns(List<String> columns) {
Review comment:
> Are you saying that if uniqueness is enforced, each insert becomes an
upsert. But if uniqueness is not enforced, then the sink would assume that
whatever is emitting records will correctly delete before inserting?
Yes. If someone are exporting relational database's change log events to
apache iceberg table and they could guarantee the exactly-once semantics (For
example, the
[flink-cdc-connector](https://github.com/ververica/flink-cdc-connectors) could
guarantee that), then the uniqueness is always correct when we just write the
INSERT/DELETE/UPDATE_BEFORE/UPDATE_AFTER to iceberg. While in some other
cases, for example flink aggregate job to refresh the metrics count value, we
will write the same key several times without deleting first, then we should
regard all the INSERT as UPSERT.
> even if uniqueness is not enforced, tables will quickly require compaction
to rewrite the equality deletes.
That was planned in the second phase, include:
1. Use bloom filter to reduce lots of useless deletes;
2. Minor compaction to convert parts of equality deletes to pos-deletes
3. Major compaction to eliminate all the deletes.
4. Make the whole read path & write path more stable. For example, cache
policy reduce duplicated delete files loading when merging on read in the same
tasks; Spill to disk if the `insertedRowMap` is exceeding the task's memory
threshold, etc. I will evaluate the read & write & compaction paths in a large
dataset, making this to be a stable solution for production.
It's good to have a document to collect all those things for reviewing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
