the-other-tim-brown commented on code in PR #772:
URL: https://github.com/apache/incubator-xtable/pull/772#discussion_r3492461891
##########
xtable-hudi-support/xtable-hudi-support-extensions/pom.xml:
##########
@@ -26,7 +26,7 @@
<version>0.2.0-SNAPSHOT</version>
</parent>
-
<artifactId>xtable-hudi-support-extensions_${scala.binary.version}</artifactId>
+ <artifactId>xtable-hudi-support-extensions_2.12</artifactId>
Review Comment:
I think it makes sense to keep the extension dependent on the scala version
##########
xtable-core/src/main/java/org/apache/xtable/hudi/HudiTableManager.java:
##########
@@ -102,11 +109,15 @@ HoodieTableMetaClient initializeHudiTable(String
tableDataPath, InternalTable ta
boolean hiveStylePartitioningEnabled =
DataLayoutStrategy.HIVE_STYLE_PARTITION == table.getLayoutStrategy();
try {
- return HoodieTableMetaClient.withPropertyBuilder()
+ return HoodieTableMetaClient.newTableBuilder()
.setCommitTimezone(HoodieTimelineTimeZone.UTC)
.setHiveStylePartitioningEnable(hiveStylePartitioningEnabled)
.setTableType(HoodieTableType.COPY_ON_WRITE)
+ // Pin new tables to table version 6 for now. Table version 9 (Hudi
1.x) support will be
Review Comment:
link issue here please
##########
xtable-hudi-support/xtable-hudi-support-extensions/pom.xml:
##########
@@ -38,7 +38,7 @@
<dependency>
<groupId>org.apache.xtable</groupId>
- <artifactId>xtable-core_${scala.binary.version}</artifactId>
+ <artifactId>xtable-core_2.12</artifactId>
Review Comment:
Same here
##########
xtable-core/src/test/java/org/apache/xtable/hudi/TestBaseFileUpdatesExtractor.java:
##########
@@ -386,16 +404,25 @@ void
extractSnapshotChanges_existingNonPartitionedTargetTable() {
// validate added files
List<WriteStatus> expectedWriteStatuses =
Collections.singletonList(
- getExpectedWriteStatus(newFileName1, "",
getExpectedColumnStats(newFileName1)));
+ getExpectedWriteStatus(
+ newFileName1, "", getExpectedColumnStats(newFileName1,
HoodieIndexVersion.V2)));
assertWriteStatusesEquivalent(expectedWriteStatuses,
replaceMetadata.getWriteStatuses());
}
private static void assertWriteStatusesEquivalent(
List<WriteStatus> expected, List<WriteStatus> actual) {
- // write status does not implement equals, so we compare their toString
results
- assertEquals(
-
expected.stream().map(WriteStatus::toString).collect(Collectors.toSet()),
-
actual.stream().map(WriteStatus::toString).collect(Collectors.toSet()));
+ // BLOCKER (hudi 1.2.0 upgrade): this comparison relied on
WriteStatus#toString, which in Hudi
+ // 1.2.0 (a) embeds per-instance identity hashes (an internal
Random/IndexStats) making exact
+ // string equality impossible, and (b) now serializes numInserts and the
full recordsStats map,
+ // which surfaces long-standing gaps in the hand-rolled expectations
(numInserts is never set on
+ // the expected stat; array elements are named ".element" here vs
production's ".array"; and
+ // decimal/double/float columns are missing). Temporarily disabled so the
rest of the suite can
+ // run and reveal any other regressions. MUST be replaced, before this PR
merges, with a
+ // semantic field-by-field comparison of
fileId/partitionPath/path/numWrites/numInserts/
+ // totalWriteBytes/fileSizeInBytes plus recordsStats
(columnName/min/max/null/value/totalSize).
Review Comment:
Can you add this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]