danny0405 commented on code in PR #13603:
URL: https://github.com/apache/hudi/pull/13603#discussion_r2234547869
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1083,24 +1084,32 @@ public static Pair<Set<String>, Set<String>>
computeRevivedAndDeletedKeys(Set<St
* @return
*/
@VisibleForTesting
- public static HoodieData<HoodieRecord> reduceByKeys(HoodieData<HoodieRecord>
recordIndexRecords, int parallelism) {
- return recordIndexRecords.mapToPair(
- (SerializablePairFunction<HoodieRecord, HoodieKey, HoodieRecord>)
t -> Pair.of(t.getKey(), t))
- .reduceByKey((SerializableBiFunction<HoodieRecord, HoodieRecord,
HoodieRecord>) (record1, record2) -> {
- boolean isRecord1Deleted = record1.getData() instanceof
EmptyHoodieRecordPayload;
- boolean isRecord2Deleted = record2.getData() instanceof
EmptyHoodieRecordPayload;
- if (isRecord1Deleted && !isRecord2Deleted) {
- return record2;
- } else if (!isRecord1Deleted && isRecord2Deleted) {
- return record1;
- } else if (isRecord1Deleted && isRecord2Deleted) {
- // let's delete just 1 of them
- return record1;
- } else {
- throw new HoodieIOException("Two HoodieRecord updates to RLI is
seen for same record key " + record2.getRecordKey() + ", record 1 : "
- + record1.getData().toString() + ", record 2 : " +
record2.getData().toString());
- }
- }, parallelism).values();
+ public static HoodieData<HoodieRecord> reduceByKeys(HoodieData<HoodieRecord>
recordIndexRecords, int parallelism, boolean isPartitionedRLI) {
+ HoodiePairData<HoodieKey, HoodieRecord> recordIndexRecordsPair;
+ if (isPartitionedRLI) {
+ recordIndexRecordsPair = recordIndexRecords.mapToPair(r -> {
+ String recordPartitionPath = r.getData() instanceof
EmptyHoodieRecordPayloadWithPartition
Review Comment:
this is so hacky, the record from MDT should have a `HoodieKey` that
represents it's actual MDT partition, here it should be `record_index`, I feel
like we should use the record key + data table partition path as the composite
record keys under partitionedRLI to make the same keys from different DT
partition distinct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
