shenh062326 commented on pull request #1838:
URL: https://github.com/apache/hudi/pull/1838#issuecomment-660575757
Add test in TestUpsertPartitioner, set partition newInsertBucket0.weight =
0.3, newInsertBucket1.weight = 0.7.
```
@Test
public void testGetPartitioner2() throws Exception {
String testPartitionPath1 = "2016/09/26";
HoodieWriteConfig config = makeHoodieClientConfigBuilder()
.withCompactionConfig(HoodieCompactionConfig.newBuilder().compactionSmallFileSize(0)
.insertSplitSize(100).autoTuneInsertSplits(false).build())
.withStorageConfig(HoodieStorageConfig.newBuilder().limitFileSize(1000 *
1024).build()).build();
HoodieClientTestUtils.fakeCommitFile(basePath, "001");
metaClient = HoodieTableMetaClient.reload(metaClient);
HoodieCopyOnWriteTable table = (HoodieCopyOnWriteTable)
HoodieTable.create(metaClient, config, hadoopConf);
HoodieTestDataGenerator dataGenerator1 = new HoodieTestDataGenerator(new
String[] {testPartitionPath1});
List<HoodieRecord> insertRecords1 =
dataGenerator1.generateInserts("001", 200);
List<HoodieRecord> records1 = new ArrayList<>();
records1.addAll(insertRecords1);
WorkloadProfile profile = new WorkloadProfile(jsc.parallelize(records1));
UpsertPartitioner partitioner = new UpsertPartitioner(profile, jsc,
table, config);
List<InsertBucket> insertBuckets =
partitioner.getInsertBuckets(testPartitionPath1);
InsertBucket newInsertBucket0 = new InsertBucket();
newInsertBucket0.bucketNumber = 0;
newInsertBucket0.weight = 0.3;
insertBuckets.remove(0);
insertBuckets.add(0, newInsertBucket0);
InsertBucket newInsertBucket1 = new InsertBucket();
newInsertBucket1.bucketNumber = 1;
newInsertBucket1.weight = 0.7;
insertBuckets.remove(1);
insertBuckets.add(1, newInsertBucket1);
Map<Integer, Integer> partition2numRecords = new HashMap<Integer,
Integer>();
for (HoodieRecord hoodieRecord: insertRecords1) {
int partition = partitioner.getPartition(new Tuple2<>(
hoodieRecord.getKey(),
Option.ofNullable(hoodieRecord.getCurrentLocation())));
if (!partition2numRecords.containsKey(partition)) {
partition2numRecords.put(partition, 0);
}
int num = partition2numRecords.get(partition);
partition2numRecords.put(partition, num + 1);
}
System.out.println(partition2numRecords);
}
```
Test it five times, the results show that the number of records inserted to
a bucket approximately matches the bucket weight.
```
{0=66, 1=134}
{0=63, 1=137}
{0=64, 1=136}
{0=67, 1=133}
{0=68, 1=132}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
