vinishjail97 commented on code in PR #772:
URL: https://github.com/apache/incubator-xtable/pull/772#discussion_r3494552617
##########
xtable-core/src/main/java/org/apache/xtable/hudi/HudiConversionTarget.java:
##########
@@ -607,7 +635,15 @@ private HoodieWriteConfig getWriteConfig(
HoodieMetadataConfig.newBuilder()
.enable(true)
.withProperties(properties)
- .withMetadataIndexColumnStats(true)
+ // Hudi 1.2.0 couples the partition-stats index to the
column-stats index. For
+ // partitioned tables the
+ // partition-stats generation path rebuilds a file-system
view over the committed
+ // external
+ // parquet files and groups them by fileId; XTable's
externally-registered files
+ // have non-Hudi names
+ // whose fileId cannot be parsed once the "_hudiext" marker
is stripped, leading
+ // to failures.
Review Comment:
Thank you — addressed in 42bba3ad. I've reflowed the comment into clean
lines and pointed it at a dedicated tracking issue (#832) that explains why the
column-stats index is currently gated to un-partitioned tables.
##########
xtable-core/src/main/java/org/apache/xtable/hudi/HudiTableManager.java:
##########
@@ -54,6 +56,10 @@ public class HudiTableManager {
"org.apache.hudi.keygen.TimestampBasedKeyGenerator";
private static final String COMPLEX_KEY_GENERATOR =
"org.apache.hudi.keygen.ComplexKeyGenerator";
private static final String SIMPLE_KEY_GENERATOR =
"org.apache.hudi.keygen.SimpleKeyGenerator";
+ // Hudi 1.x spark query defaults to "default" database and spark read query
picks up delta
+ // instead, 0.x doesn't have the same problem.
+ // TODO: https://app.clickup.com/t/18029943/ENG-23339
+ private static final String DEFAULT_DATABASE_NAME = "default_hudi";
Review Comment:
Good catch — removed in 42bba3ad. The `DEFAULT_DATABASE_NAME` constant and
the `setDatabaseName(...)` call are both gone, so newly created tables fall
back to Hudi's default database, matching the pre-upgrade behaviour on `main`.
I re-ran `TestHudiTableManager`, `TestHudiConversionTarget` and
`ITHudiConversionTarget` after the change and they all pass.
##########
xtable-hudi-support/xtable-hudi-support-extensions/pom.xml:
##########
@@ -26,7 +26,7 @@
<version>0.2.0-SNAPSHOT</version>
</parent>
-
<artifactId>xtable-hudi-support-extensions_${scala.binary.version}</artifactId>
+ <artifactId>xtable-hudi-support-extensions_2.12</artifactId>
Review Comment:
Agreed — reverted in 42bba3ad. The module artifactId is back to
`xtable-hudi-support-extensions_${scala.binary.version}` so it stays tied to
the build-time Scala version.
##########
xtable-hudi-support/xtable-hudi-support-extensions/pom.xml:
##########
@@ -38,7 +38,7 @@
<dependency>
<groupId>org.apache.xtable</groupId>
- <artifactId>xtable-core_${scala.binary.version}</artifactId>
+ <artifactId>xtable-core_2.12</artifactId>
Review Comment:
Done in 42bba3ad — the `xtable-core` dependencies are back to
`xtable-core_${scala.binary.version}` as well (both the compile dependency and
the test-jar entry).
##########
xtable-core/src/main/java/org/apache/xtable/hudi/HudiTableManager.java:
##########
@@ -102,11 +109,15 @@ HoodieTableMetaClient initializeHudiTable(String
tableDataPath, InternalTable ta
boolean hiveStylePartitioningEnabled =
DataLayoutStrategy.HIVE_STYLE_PARTITION == table.getLayoutStrategy();
try {
- return HoodieTableMetaClient.withPropertyBuilder()
+ return HoodieTableMetaClient.newTableBuilder()
.setCommitTimezone(HoodieTimelineTimeZone.UTC)
.setHiveStylePartitioningEnable(hiveStylePartitioningEnabled)
.setTableType(HoodieTableType.COPY_ON_WRITE)
+ // Pin new tables to table version 6 for now. Table version 9 (Hudi
1.x) support will be
Review Comment:
Added in 42bba3ad — the v6-pin comment now links the umbrella issue #762
(Hudi version upgrade), under which the table-version-9 follow-up work is
tracked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]