aokolnychyi commented on a change in pull request #1738:
URL: https://github.com/apache/iceberg/pull/1738#discussion_r521670864
##########
File path: spark3/src/main/java/org/apache/iceberg/spark/source/SparkTable.java
##########
@@ -155,6 +163,39 @@ public WriteBuilder newWriteBuilder(LogicalWriteInfo info)
{
return new SparkWriteBuilder(sparkSession(), icebergTable, info);
}
+ @Override
+ public MergeBuilder newMergeBuilder(LogicalWriteInfo info) {
+ String mode = icebergTable.properties().getOrDefault(WRITE_ROW_LEVEL_MODE,
WRITE_ROW_LEVEL_MODE_DEFAULT);
+ ValidationException.check(mode.equals("copy-on-write"), "Unsupported row
operations mode: %s", mode);
+ return new SparkMergeBuilder(sparkSession(), icebergTable, info);
+ }
+
+ @Override
+ public boolean canDeleteWhere(Filter[] filters) {
+ // TODO: multiple partition specs
+ Set<Integer> partitionFieldSourceIds =
PartitionUtil.sourceFieldIds(table().spec());
+ Schema schema = table().schema();
+
+ for (Filter filter : filters) {
+ // return false if we have a predicate on a non-partition column or if
we cannot translate the filter
+ if (isDataFilter(filter, schema, partitionFieldSourceIds) ||
SparkFilters.convert(filter) == null) {
+ return false;
+ }
+ }
+
+ return true;
+ }
+
+ private boolean isDataFilter(Filter filter, Schema schema, Set<Integer>
partitionFieldSourceIds) {
+ // TODO: handle dots correctly via v2references
+ Set<String> filterRefs = Sets.newHashSet(filter.references());
+ return filterRefs.stream().anyMatch(ref -> {
+ Types.NestedField field = schema.findField(ref);
+ ValidationException.check(field != null, "Cannot find field %s in
schema", ref);
+ return !partitionFieldSourceIds.contains(field.fieldId());
Review comment:
You are right, if a table is partitioned by `day(ts)`, it is not safe to
execute `ts < timestamp`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
