vinishjail97 commented on code in PR #835:
URL: https://github.com/apache/incubator-xtable/pull/835#discussion_r4159173668
##########
xtable-core/src/main/java/org/apache/xtable/hudi/HudiConversionTarget.java:
##########
@@ -648,14 +723,16 @@ private HoodieWriteConfig getWriteConfig(
HoodieMetadataConfig.newBuilder()
.enable(true)
.withProperties(properties)
- // Hudi 1.2.0 couples the partition-stats index to the
column-stats index. For
- // partitioned tables, the partition-stats generation path
rebuilds a file-system
- // view over the committed external parquet files and groups
them by fileId.
- // XTable's externally-registered files have non-Hudi names
whose fileId cannot be
- // parsed once the "_hudiext" marker is stripped, which
leads to failures. So
- // column stats are only enabled for un-partitioned tables
for now. Tracked in
- // https://github.com/apache/incubator-xtable/issues/832.
-
.withMetadataIndexColumnStats(!metaClient.getTableConfig().isTablePartitioned())
+ // Build the column-stats index for all tables. The
partition-stats index is
+ // disabled independently: its generation path rebuilds a
file-system view over
+ // the committed external parquet files and groups them by
fileId, but XTable's
+ // externally-registered files have non-Hudi names whose
fileId cannot be parsed
+ // once the "_hudiext" marker is stripped, which leads to
failures on partitioned
+ // tables. Disabling partition stats (while keeping column
stats) requires the
+ // independent toggle added in
https://github.com/apache/hudi/pull/19111. Tracked
+ // in https://github.com/apache/incubator-xtable/issues/832.
+ .withMetadataIndexColumnStats(true)
Review Comment:
The fix is in a stacked PR, so this diff stays focused:
https://github.com/vinishjail97/onetable/pull/2. When the column-stats index is
missing, the target requires one snapshot sync, which registers every file
again with its stats. The PR also fixes `markInstantsAsCleaned` for the older
file slices.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
