rdblue commented on a change in pull request #3461:
URL: https://github.com/apache/iceberg/pull/3461#discussion_r744305596
##########
File path:
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/SparkWriteOptions.java
##########
@@ -53,4 +53,11 @@ private SparkWriteOptions() {
// Controls whether to allow writing timestamps without zone info
public static final String HANDLE_TIMESTAMP_WITHOUT_TIMEZONE =
"handle-timestamp-without-timezone";
+
+ // Overrides the default distribution mode for a write operation
+ public static final String DISTRIBUTION_MODE = "distribution-mode";
+
+ // Controls whether to ignore the table sort order during a write operation
+ public static final String IGNORE_SORT_ORDER = "ignore-sort-order";
Review comment:
What does this mean? Treat the table as unsorted? What about
distribution, does that require overriding: `distribution-mode=none`? Looks
like the current implementation of `SparkWriteConf.distributionMode()` does not
check `ignoreSortOrder`.
It makes sense to me to have an option like this, so that someone writing a
job can turn off automatic sort order and partitioning and can customize by
adding a sort to the job instead. My main concern is that we are consistent and
it may be easier to turn off required distribution and ordering together rather
than separately. For example, if you set `ignore-sort-order=true` but the
table's distribution mode is still `range`, then
`SparkDistributionAndOrderingUtil` will still build a global sort using the
table's sort order:
https://github.com/apache/iceberg/pull/3461/files#diff-acf71a9ace396ea8a334c93bbf55629f493eb830124ef5e6a41a5a6a4e200af6R53
Maybe this should be `ignore-table-distribution-and-sort` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
