yihua commented on code in PR #12874:
URL: https://github.com/apache/hudi/pull/12874#discussion_r1966455425
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/common/TestHoodieSparkEngineContext.java:
##########
@@ -67,4 +78,49 @@ void testSetJobStatus() {
// Assert jobGroupId does not change
assertEquals(jobGroupId,
context.getJavaSparkContext().getLocalProperty("spark.jobGroup.id"));
}
+
+ @Test
+ void testMapPartitionsToPairAndReduceByKey() {
+ int numPartitions = 6;
+ HoodieData<Integer> rddData = context.parallelize(
+ IntStream.rangeClosed(0, 99).boxed().collect(Collectors.toList()),
numPartitions);
+
+ //
+ /* output from map to pair.
+ 1 = 0, 1
+ 2 = 1, 5
+ 3 = 1, 5
+ 4 = 2, 10
+ 5 = 2, 10
+ 6 = 3, 15
+ 7 = 3, 15
+ 8 = 4, 20
+
+ And then we do reduce by key, where we sum up the values.
+ */
+ Stream<ImmutablePair<Integer, Integer>> result =
context.mapPartitionsToPairAndReduceByKey(rddData.collectAsList().stream(),
+ new SerializablePairFlatMapTestFunc(), new
SerializableReduceTestFunc(), 6);
+ result.forEach(entry -> {
+ assertEquals(entry.getKey() * 10, entry.getValue());
+ });
Review Comment:
Also assert the number of items in `result`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
