stevenzwu commented on a change in pull request #2064:
URL: https://github.com/apache/iceberg/pull/2064#discussion_r559714619
##########
File path: core/src/main/java/org/apache/iceberg/TableProperties.java
##########
@@ -138,6 +138,9 @@ private TableProperties() {
public static final String ENGINE_HIVE_ENABLED = "engine.hive.enabled";
public static final boolean ENGINE_HIVE_ENABLED_DEFAULT = false;
+ public static final String WRITE_SHUFFLE_BY_PARTITION =
"write.shuffle-by.partition";
Review comment:
Hashing over very large cardinality (like `user_id`). I also like to
caution about `keyBy` over partition key which has relatively small cardinality
(like `bucket(user_id)`). It can get pretty uneven distribution.
```
{0=19, 1=62, 2=169, 3=282, 4=364, 5=318, 6=317, 7=207, 8=131, 9=60, 10=37,
11=15, 12=14, 13=4, 14=1}
```
Here is the sample code.
```java
@Test
public void testKeyBy() {
final int maxParallelism = 4000;
final int numberOfTasks = 2000;
final int numberOfBuckets = 10000;
final Integer[] assignment = new Integer[2000];
for (int i = 0; i < numberOfTasks; ++i) {
assignment[i] = 0;
}
for (int i = 0; i < numberOfBuckets; ++i) {
final String key = "date=20210119/bucket=" + i;
final int assignedTask =
KeyGroupRangeAssignment.assignKeyToParallelOperator(
key, maxParallelism, numberOfTasks);
assignment[assignedTask] += 1;
}
final Map<Integer, Long> assignmentStats =
Lists.newArrayList(assignment).stream()
.collect(Collectors.groupingBy(Function.identity(),
Collectors.counting()));
System.out.println(assignmentStats);
}
```
We have to write a custom partitioner that directly map bucket id to the
subtask
```java
static class BucketPartitioner implements Partitioner<Integer> {
@Override
public int partition(Integer key, int numPartitions) {
return key % numPartitions;
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
