aokolnychyi commented on code in PR #7443:
URL: https://github.com/apache/iceberg/pull/7443#discussion_r1179310231
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingDataRewriter.java:
##########
@@ -135,4 +144,36 @@ private double compressionFactor(Map<String, String>
options) {
value > 0, "'%s' is set to %s but must be > 0", COMPRESSION_FACTOR,
value);
return value;
}
+
+ private static class OrderedWrite implements RequiresDistributionAndOrdering
{
+ private final OrderedDistribution distribution;
+ private final SortOrder[] ordering;
+ private final int numShufflePartitions;
+
+ OrderedWrite(SortOrder[] ordering, int numShufflePartitions) {
+ this.distribution = Distributions.ordered(ordering);
+ this.ordering = ordering;
+ this.numShufflePartitions = numShufflePartitions;
+ }
+
+ @Override
+ public Distribution requiredDistribution() {
+ return distribution;
+ }
+
+ @Override
+ public boolean distributionStrictlyRequired() {
+ return true;
+ }
+
+ @Override
+ public int requiredNumPartitions() {
+ return numShufflePartitions;
Review Comment:
This is used to request a particular number of partitions instead of setting
a SQL conf.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
