aokolnychyi commented on a change in pull request #3661:
URL: https://github.com/apache/iceberg/pull/3661#discussion_r767949371
##########
File path:
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java
##########
@@ -163,8 +167,25 @@ public DistributionMode distributionMode() {
return DistributionMode.fromName(modeName);
}
- public DistributionMode deleteDistributionMode() {
- return
rowLevelCommandDistributionMode(TableProperties.DELETE_DISTRIBUTION_MODE);
+ public DistributionMode copyOnWriteDeleteDistributionMode() {
+ String deleteModeName = confParser.stringConf()
+ .option(SparkWriteOptions.DISTRIBUTION_MODE)
+ .tableProperty(TableProperties.DELETE_DISTRIBUTION_MODE)
+ .parseOptional();
+
+ if (deleteModeName != null) {
+ // range distribution only makes sense if the sort order is set
+ DistributionMode deleteMode = DistributionMode.fromName(deleteModeName);
+ if (deleteMode == RANGE && table.sortOrder().isUnsorted()) {
+ return HASH;
+ } else {
+ return deleteMode;
+ }
+ } else {
+ // use hash distribution if write distribution is range or hash
Review comment:
Added a comment below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
