the-other-tim-brown commented on code in PR #772:
URL: https://github.com/apache/incubator-xtable/pull/772#discussion_r3364660305
##########
pom.xml:
##########
@@ -271,6 +271,12 @@
<version>${hudi.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-utilities_2.12</artifactId>
Review Comment:
we should make this take in the scala version so it is consistent with the
build time scala version
##########
xtable-core/src/test/java/org/apache/xtable/TestSparkHudiTable.java:
##########
@@ -164,10 +164,7 @@ public List<HoodieRecord<HoodieAvroPayload>>
insertRecordsWithCommitAlreadyStart
String commitInstant,
boolean checkForNoErrors) {
JavaRDD<HoodieRecord<HoodieAvroPayload>> writeRecords =
jsc.parallelize(inserts, 1);
- List<WriteStatus> result = writeClient.bulkInsert(writeRecords,
commitInstant).collect();
- if (checkForNoErrors) {
- assertNoWriteErrors(result);
- }
+ assert writeClient.commit(commitInstant,
writeClient.bulkInsert(writeRecords, commitInstant));
Review Comment:
Nitpick: instead of using plain `assert` in these files, can we use the
junit platform assertions for consistency?
##########
xtable-core/src/main/java/org/apache/xtable/hudi/BaseFileUpdatesExtractor.java:
##########
@@ -74,6 +84,7 @@ public class BaseFileUpdatesExtractor {
* @param commit The current commit started by the Hudi client
* @return The information needed to create a "replace" commit for the Hudi
table
*/
+ @SneakyThrows
Review Comment:
Should we do this? catch and rethrow with some proper context for the user?
##########
xtable-hudi-support/xtable-hudi-support-extensions/src/main/java/org/apache/xtable/hudi/extensions/AddFieldIdsClientInitCallback.java:
##########
@@ -63,17 +62,16 @@ public void call(BaseHoodieClient hoodieClient) {
try {
Option<Schema> currentSchema = Option.empty();
try {
- Configuration hadoopConfiguration =
hoodieClient.getEngineContext().getHadoopConf().get();
- String tableBasePath = config.getBasePath();
- FileSystem fs = FSUtils.getFs(tableBasePath, hadoopConfiguration);
- if (FSUtils.isTableExists(config.getBasePath(), fs)) {
- HoodieTableMetaClient metaClient =
- HoodieTableMetaClient.builder()
- .setConf(hadoopConfiguration)
- .setBasePath(tableBasePath)
- .build();
+ HoodieTableMetaClient metaClient =
Review Comment:
Previously this meta client was only build if the table existed. Does this
handle the missing table gracefully now?
##########
xtable-service/pom.xml:
##########
@@ -208,6 +208,7 @@
<dependency>
<groupId>io.delta</groupId>
<artifactId>delta-core_${scala.binary.version}</artifactId>
+ <version>${delta.version}</version>
Review Comment:
is this required? it should be defined in the parent
##########
xtable-core/src/test/java/org/apache/xtable/TestAbstractHudiTable.java:
##########
@@ -433,10 +443,9 @@ protected HoodieWriteConfig generateWriteConfig(Schema
schema, TypedProperties k
HoodieMetadataConfig metadataConfig =
HoodieMetadataConfig.newBuilder()
.enable(true)
- // enable col stats only on un-partitioned data due to bug in Hudi
- // https://issues.apache.org/jira/browse/HUDI-6954
- .withMetadataIndexColumnStats(
- !keyGenProperties.getString(PARTITIONPATH_FIELD_NAME.key(),
"").isEmpty())
+ // TODO: Hudi 1.1 MDT col-stats generation fails for array and map
types.
Review Comment:
can we enable the stats when the array/maps are not present?
##########
xtable-hudi-support/xtable-hudi-support-utils/src/test/java/org/apache/xtable/hudi/idtracking/TestIdTracker.java:
##########
@@ -341,28 +498,28 @@ private static IdTracking
getExpectTrackingForComplexSchemaEvolvedNoMetaFields()
"double_nested",
13,
Collections.singletonList(new
IdMapping("double_nested_int", 14))),
- new IdMapping("level", 20))))),
+ new IdMapping("level", 18))))),
new IdMapping(
"primitive_map_field",
6,
Arrays.asList(new IdMapping("key", 15), new IdMapping("value",
16))),
new IdMapping(
"array_field",
- 17,
+ 19,
Collections.singletonList(
new IdMapping(
"element",
- 21,
+ 20,
Arrays.asList(
- new IdMapping("nested_int", 22),
+ new IdMapping("nested_int", 21),
new IdMapping(
"double_nested",
- 23,
- Collections.singletonList(new
IdMapping("double_nested_int", 25))),
- new IdMapping("level", 24))))),
+ 22,
Review Comment:
why are the values updating?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]