the-other-tim-brown commented on code in PR #772:
URL: https://github.com/apache/incubator-xtable/pull/772#discussion_r3364660305


##########
pom.xml:
##########
@@ -271,6 +271,12 @@
                 <version>${hudi.version}</version>
                 <scope>provided</scope>
             </dependency>
+            <dependency>
+                <groupId>org.apache.hudi</groupId>
+                <artifactId>hudi-utilities_2.12</artifactId>

Review Comment:
   we should make this take in the scala version so it is consistent with the 
build time scala version



##########
xtable-core/src/test/java/org/apache/xtable/TestSparkHudiTable.java:
##########
@@ -164,10 +164,7 @@ public List<HoodieRecord<HoodieAvroPayload>> 
insertRecordsWithCommitAlreadyStart
       String commitInstant,
       boolean checkForNoErrors) {
     JavaRDD<HoodieRecord<HoodieAvroPayload>> writeRecords = 
jsc.parallelize(inserts, 1);
-    List<WriteStatus> result = writeClient.bulkInsert(writeRecords, 
commitInstant).collect();
-    if (checkForNoErrors) {
-      assertNoWriteErrors(result);
-    }
+    assert writeClient.commit(commitInstant, 
writeClient.bulkInsert(writeRecords, commitInstant));

Review Comment:
   Nitpick: instead of using plain `assert` in these files, can we use the 
junit platform assertions for consistency?



##########
xtable-core/src/main/java/org/apache/xtable/hudi/BaseFileUpdatesExtractor.java:
##########
@@ -74,6 +84,7 @@ public class BaseFileUpdatesExtractor {
    * @param commit The current commit started by the Hudi client
    * @return The information needed to create a "replace" commit for the Hudi 
table
    */
+  @SneakyThrows

Review Comment:
   Should we do this? catch and rethrow with some proper context for the user?



##########
xtable-hudi-support/xtable-hudi-support-extensions/src/main/java/org/apache/xtable/hudi/extensions/AddFieldIdsClientInitCallback.java:
##########
@@ -63,17 +62,16 @@ public void call(BaseHoodieClient hoodieClient) {
       try {
         Option<Schema> currentSchema = Option.empty();
         try {
-          Configuration hadoopConfiguration = 
hoodieClient.getEngineContext().getHadoopConf().get();
-          String tableBasePath = config.getBasePath();
-          FileSystem fs = FSUtils.getFs(tableBasePath, hadoopConfiguration);
-          if (FSUtils.isTableExists(config.getBasePath(), fs)) {
-            HoodieTableMetaClient metaClient =
-                HoodieTableMetaClient.builder()
-                    .setConf(hadoopConfiguration)
-                    .setBasePath(tableBasePath)
-                    .build();
+          HoodieTableMetaClient metaClient =

Review Comment:
   Previously this meta client was only build if the table existed. Does this 
handle the missing table gracefully now?



##########
xtable-service/pom.xml:
##########
@@ -208,6 +208,7 @@
         <dependency>
             <groupId>io.delta</groupId>
             <artifactId>delta-core_${scala.binary.version}</artifactId>
+            <version>${delta.version}</version>

Review Comment:
   is this required? it should be defined in the parent



##########
xtable-core/src/test/java/org/apache/xtable/TestAbstractHudiTable.java:
##########
@@ -433,10 +443,9 @@ protected HoodieWriteConfig generateWriteConfig(Schema 
schema, TypedProperties k
     HoodieMetadataConfig metadataConfig =
         HoodieMetadataConfig.newBuilder()
             .enable(true)
-            // enable col stats only on un-partitioned data due to bug in Hudi
-            // https://issues.apache.org/jira/browse/HUDI-6954
-            .withMetadataIndexColumnStats(
-                !keyGenProperties.getString(PARTITIONPATH_FIELD_NAME.key(), 
"").isEmpty())
+            // TODO: Hudi 1.1 MDT col-stats generation fails for array and map 
types.

Review Comment:
   can we enable the stats when the array/maps are not present?



##########
xtable-hudi-support/xtable-hudi-support-utils/src/test/java/org/apache/xtable/hudi/idtracking/TestIdTracker.java:
##########
@@ -341,28 +498,28 @@ private static IdTracking 
getExpectTrackingForComplexSchemaEvolvedNoMetaFields()
                                 "double_nested",
                                 13,
                                 Collections.singletonList(new 
IdMapping("double_nested_int", 14))),
-                            new IdMapping("level", 20))))),
+                            new IdMapping("level", 18))))),
             new IdMapping(
                 "primitive_map_field",
                 6,
                 Arrays.asList(new IdMapping("key", 15), new IdMapping("value", 
16))),
             new IdMapping(
                 "array_field",
-                17,
+                19,
                 Collections.singletonList(
                     new IdMapping(
                         "element",
-                        21,
+                        20,
                         Arrays.asList(
-                            new IdMapping("nested_int", 22),
+                            new IdMapping("nested_int", 21),
                             new IdMapping(
                                 "double_nested",
-                                23,
-                                Collections.singletonList(new 
IdMapping("double_nested_int", 25))),
-                            new IdMapping("level", 24))))),
+                                22,

Review Comment:
   why are the values updating?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to