vinishjail97 commented on code in PR #772:
URL: https://github.com/apache/incubator-xtable/pull/772#discussion_r3478041255
##########
xtable-hudi-support/xtable-hudi-support-extensions/src/main/java/org/apache/xtable/hudi/extensions/AddFieldIdsClientInitCallback.java:
##########
@@ -63,17 +62,16 @@ public void call(BaseHoodieClient hoodieClient) {
try {
Option<Schema> currentSchema = Option.empty();
try {
- Configuration hadoopConfiguration =
hoodieClient.getEngineContext().getHadoopConf().get();
- String tableBasePath = config.getBasePath();
- FileSystem fs = FSUtils.getFs(tableBasePath, hadoopConfiguration);
- if (FSUtils.isTableExists(config.getBasePath(), fs)) {
- HoodieTableMetaClient metaClient =
- HoodieTableMetaClient.builder()
- .setConf(hadoopConfiguration)
- .setBasePath(tableBasePath)
- .build();
+ HoodieTableMetaClient metaClient =
Review Comment:
Yes, the missing-table case is handled gracefully. The
`HoodieTableMetaClient.builder()...build()` plus schema resolution are wrapped
in a `try/catch`, so for a not-yet-created table the `TableNotFoundException`
thrown by `build()` is caught, logged at WARN, and `currentSchema` stays
`Option.empty()`. The callback then proceeds with no field-id schema — same
effective outcome as the old `FSUtils.isTableExists(...)` guard. The only
behavioral difference is a benign WARN log on the very first sync of a
brand-new table.
##########
xtable-core/src/test/java/org/apache/xtable/TestSparkHudiTable.java:
##########
@@ -164,10 +164,7 @@ public List<HoodieRecord<HoodieAvroPayload>>
insertRecordsWithCommitAlreadyStart
String commitInstant,
boolean checkForNoErrors) {
JavaRDD<HoodieRecord<HoodieAvroPayload>> writeRecords =
jsc.parallelize(inserts, 1);
- List<WriteStatus> result = writeClient.bulkInsert(writeRecords,
commitInstant).collect();
- if (checkForNoErrors) {
- assertNoWriteErrors(result);
- }
+ assert writeClient.commit(commitInstant,
writeClient.bulkInsert(writeRecords, commitInstant));
Review Comment:
Done — these now use `Assertions.assertTrue(...)` (e.g. `TestSparkHudiTable`
lines 168, 179, 190, 213, 231). No plain `assert` keyword remains in these test
files.
##########
xtable-core/src/test/java/org/apache/xtable/TestAbstractHudiTable.java:
##########
@@ -433,10 +443,9 @@ protected HoodieWriteConfig generateWriteConfig(Schema
schema, TypedProperties k
HoodieMetadataConfig metadataConfig =
HoodieMetadataConfig.newBuilder()
.enable(true)
- // enable col stats only on un-partitioned data due to bug in Hudi
- // https://issues.apache.org/jira/browse/HUDI-6954
- .withMetadataIndexColumnStats(
- !keyGenProperties.getString(PARTITIONPATH_FIELD_NAME.key(),
"").isEmpty())
+ // TODO: Hudi 1.1 MDT col-stats generation fails for array and map
types.
Review Comment:
MDT col-stats can't be enabled in Hudi 1.2.0 even when arrays/maps are
absent, because it couples the partition-stats index to the column-stats index.
For partitioned tables the partition-stats generation path rebuilds a
file-system view over the committed external parquet files and groups them by
fileId; XTable's externally-registered files have non-Hudi names whose fileId
cannot be parsed once the `_hudiext` marker is stripped, which leads to
failures. So enabling col-stats here isn't viable yet — tracked in #773. (The
commented-out `schemaContainsArrayOrMap` / TODO line is stale and will be
cleaned up.)
##########
xtable-core/src/main/java/org/apache/xtable/hudi/BaseFileUpdatesExtractor.java:
##########
@@ -74,6 +84,7 @@ public class BaseFileUpdatesExtractor {
* @param commit The current commit started by the Hudi client
* @return The information needed to create a "replace" commit for the Hudi
table
*/
+ @SneakyThrows
Review Comment:
Done — there's no `@SneakyThrows` left in `BaseFileUpdatesExtractor`.
Extraction/IO failures are caught and rethrown with context as `ReadException`,
e.g. `throw new ReadException("Failed to extract snapshot changes for Hudi
table at " + tableBasePath, ex)`.
##########
pom.xml:
##########
@@ -271,6 +271,12 @@
<version>${hudi.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-utilities_2.12</artifactId>
Review Comment:
Done — this is declared as `hudi-utilities_${scala.binary.version}` in the
root pom's dependencyManagement, consistent with the build-time scala version
(same convention as the other `_${scala.binary.version}` artifacts).
##########
xtable-service/pom.xml:
##########
@@ -208,6 +208,7 @@
<dependency>
<groupId>io.delta</groupId>
<artifactId>delta-core_${scala.binary.version}</artifactId>
+ <version>${delta.version}</version>
Review Comment:
Done — `delta-core` in `xtable-service/pom.xml` no longer pins a
`<version>`; it inherits `${delta.version}` from the parent's
dependencyManagement.
##########
xtable-hudi-support/xtable-hudi-support-utils/src/test/java/org/apache/xtable/hudi/idtracking/TestIdTracker.java:
##########
@@ -341,28 +498,28 @@ private static IdTracking
getExpectTrackingForComplexSchemaEvolvedNoMetaFields()
"double_nested",
13,
Collections.singletonList(new
IdMapping("double_nested_int", 14))),
- new IdMapping("level", 20))))),
+ new IdMapping("level", 18))))),
new IdMapping(
"primitive_map_field",
6,
Arrays.asList(new IdMapping("key", 15), new IdMapping("value",
16))),
new IdMapping(
"array_field",
- 17,
+ 19,
Collections.singletonList(
new IdMapping(
"element",
- 21,
+ 20,
Arrays.asList(
- new IdMapping("nested_int", 22),
+ new IdMapping("nested_int", 21),
new IdMapping(
"double_nested",
- 23,
- Collections.singletonList(new
IdMapping("double_nested_int", 25))),
- new IdMapping("level", 24))))),
+ 22,
Review Comment:
The expected ids shift because the evolved-schema path now goes through the
new `updateIdMappings(...)`: existing top-level columns keep their original
ids, while newly-added top-level columns (e.g. `array_field`,
`primitive_array_field`) get the next sequential id, and nested ids are
(re)generated per-column subtree rather than in one global pre-order pass over
the whole schema. That changes the spacing/order of the nested ids (e.g.
`array_field` 17→19, its `element` 21→20) while keeping ids stable within each
column's subtree.
##########
xtable-hudi-support/xtable-hudi-support-utils/src/main/java/org/apache/xtable/hudi/idtracking/IdTracker.java:
##########
@@ -122,12 +124,63 @@ public IdTracking getIdTracking(
AtomicInteger currentId = new AtomicInteger(existingState.getLastIdUsed());
// add meta fields to the schema in order to ensure they will be assigned
IDs
Schema schemaForIdMapping =
- includeMetaFields ? HoodieAvroUtils.addMetadataFields(schema) : schema;
- List<IdMapping> newMappings =
- generateIdMappings(schemaForIdMapping, currentId,
existingState.getIdMappings());
- return new IdTracking(newMappings, currentId.get());
+ includeMetaFields
+ ?
HoodieSchemaUtils.addMetadataFields(HoodieSchema.fromAvroSchema(schema))
+ .toAvroSchema()
+ : schema;
+ if (currentId.intValue() != 0) {
Review Comment:
Not a pre-existing bug — it's an adaptation for the Hudi 1.2 schema path.
`getIdTracking` now branches: when there's prior id-tracking state (`currentId
!= 0`, i.e. schema evolution) it calls `updateIdMappings` to preserve existing
top-level ids and only assign ids to newly-added columns; the fresh-schema case
(`currentId == 0`) still calls `generateIdMappings`. The old single-path
`generateIdMappings` over `HoodieAvroUtils.addMetadataFields` regenerated
everything, which didn't preserve ids across evolution.
##########
xtable-hudi-support/xtable-hudi-support-utils/src/main/java/org/apache/xtable/hudi/idtracking/IdTracker.java:
##########
@@ -122,12 +124,63 @@ public IdTracking getIdTracking(
AtomicInteger currentId = new AtomicInteger(existingState.getLastIdUsed());
// add meta fields to the schema in order to ensure they will be assigned
IDs
Schema schemaForIdMapping =
- includeMetaFields ? HoodieAvroUtils.addMetadataFields(schema) : schema;
- List<IdMapping> newMappings =
- generateIdMappings(schemaForIdMapping, currentId,
existingState.getIdMappings());
- return new IdTracking(newMappings, currentId.get());
+ includeMetaFields
+ ?
HoodieSchemaUtils.addMetadataFields(HoodieSchema.fromAvroSchema(schema))
+ .toAvroSchema()
+ : schema;
+ if (currentId.intValue() != 0) {
+ // if the schema has ID tracking, update the existing mappings.
+ List<IdMapping> updatedMappings =
+ updateIdMappings(schemaForIdMapping, currentId,
existingState.getIdMappings());
+ return new IdTracking(updatedMappings, currentId.get());
+ } else {
+ // if the schema does not have ID tracking, generate new mappings.
+ List<IdMapping> newMappings =
+ generateIdMappings(schemaForIdMapping, currentId,
existingState.getIdMappings());
+ return new IdTracking(newMappings, currentId.get());
+ }
}
+ /**
+ * Updates the IdMappings in the provided schema. For all newly added
columns, we process column
+ * by column for new id assignment.
+ *
+ * <p>Different from generateIdMappings which traverse the entire schema
tree, this method
+ * traverse individual columns and update the id mappings.
Review Comment:
The javadoc now reflects this accurately — it states that `updateIdMappings`
preserves the existing top-level mappings and *delegates to
`generateIdMappings` per column* to (re)generate the nested mappings within
each column's subtree. So the per-column traversal is called out explicitly
rather than implying the method never traverses.
##########
xtable-core/src/main/java/org/apache/hudi/stats/XTableValueMetadata.java:
##########
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.stats;
+
+import static
org.apache.xtable.model.schema.InternalSchema.MetadataKey.TIMESTAMP_PRECISION;
+import static
org.apache.xtable.model.schema.InternalSchema.MetadataValue.MICROS;
+
+import java.lang.reflect.Constructor;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+
+import org.apache.hudi.metadata.HoodieIndexVersion;
+
+import org.apache.xtable.model.schema.InternalSchema;
+import org.apache.xtable.model.stat.ColumnStat;
+
+/**
+ * Utility class for creating and converting Hudi {@link ValueMetadata}
instances from XTable's
+ * internal schema representation.
+ *
+ * <p>This class bridges XTable's {@link InternalSchema} types to Hudi's
{@link ValueType} and
+ * {@link ValueMetadata} used for column statistics. It handles the conversion
of various data types
+ * including timestamps, decimals, and dates.
+ *
+ * <p>Note: This class uses reflection to create {@link ValueMetadata}
instances because XTable
+ * classes may be loaded by a different classloader than Hudi classes in Spark
environments, making
+ * direct constructor access illegal.
+ */
+public class XTableValueMetadata {
+
+ /**
+ * Creates a {@link ValueMetadata} instance from a {@link ColumnStat} for
the specified Hudi index
+ * version.
+ *
+ * @param columnStat the column statistics containing schema information
+ * @param indexVersion the Hudi index version to use for metadata creation
+ * @return the appropriate {@link ValueMetadata} for the column's data type
+ * @throws IllegalArgumentException if columnStat is null (for V2+ index),
or if decimal metadata
+ * is missing required precision/scale
+ * @throws IllegalStateException if an unsupported internal type is
encountered
+ */
+ public static ValueMetadata getValueMetadata(
+ ColumnStat columnStat, HoodieIndexVersion indexVersion) {
+ if (indexVersion.lowerThan(HoodieIndexVersion.V2)) {
+ return ValueMetadata.V1EmptyMetadata.get();
+ }
+ if (columnStat == null) {
+ throw new IllegalArgumentException("ColumnStat cannot be null");
+ }
+ InternalSchema internalSchema = columnStat.getField().getSchema();
+ ValueType valueType = fromInternalSchema(internalSchema);
+ if (valueType == ValueType.V1) {
+ throw new IllegalStateException(
+ "InternalType V1 should not be returned from fromInternalSchema");
+ } else if (valueType == ValueType.DECIMAL) {
+ if (internalSchema.getMetadata() == null) {
+ throw new IllegalArgumentException("Decimal metadata is null");
+ } else if (!internalSchema
+ .getMetadata()
+ .containsKey(InternalSchema.MetadataKey.DECIMAL_SCALE)) {
+ throw new IllegalArgumentException("Decimal scale is null");
+ } else if (!internalSchema
+ .getMetadata()
+ .containsKey(InternalSchema.MetadataKey.DECIMAL_PRECISION)) {
+ throw new IllegalArgumentException("Decimal precision is null");
+ }
+ int scale = (int)
internalSchema.getMetadata().get(InternalSchema.MetadataKey.DECIMAL_SCALE);
+ int precision =
+ (int)
internalSchema.getMetadata().get(InternalSchema.MetadataKey.DECIMAL_PRECISION);
+ return ValueMetadata.DecimalMetadata.create(precision, scale);
+ } else {
+ return createValueMetadata(valueType);
+ }
+ }
+
+ /**
+ * Maps an XTable {@link InternalSchema} to the corresponding Hudi {@link
ValueType}.
+ *
+ * @param internalSchema the internal schema to convert
+ * @return the corresponding Hudi value type
+ * @throws UnsupportedOperationException if the internal data type is not
supported
+ */
+ static ValueType fromInternalSchema(InternalSchema internalSchema) {
+ switch (internalSchema.getDataType()) {
+ case NULL:
+ return ValueType.NULL;
+ case BOOLEAN:
+ return ValueType.BOOLEAN;
+ case INT:
+ return ValueType.INT;
+ case LONG:
+ return ValueType.LONG;
+ case FLOAT:
+ return ValueType.FLOAT;
+ case DOUBLE:
+ return ValueType.DOUBLE;
+ case STRING:
Review Comment:
Added — `ENUM` is handled in `fromInternalSchema` (it falls through with
`STRING`, since enum values are stored as their string symbols in column
stats), and there's a unit test for it
(`testFromInternalSchemaEnumMapsToString` in `TestXTableValueMetadata`).
##########
xtable-core/src/main/java/org/apache/xtable/hudi/HudiConversionSource.java:
##########
@@ -237,10 +239,21 @@ private HoodieTimeline getCompletedCommits() {
}
private CommitsPair getCompletedAndPendingCommitsAfterInstant(HoodieInstant
commitInstant) {
+ // Get all instants after the provided commitInstant and instants that are
yet to be completed.
+ // We use completionTime (not requestedTime) for filtering completed
commits because we need
+ // commits that actually finished after the last sync, not just ones that
were initiated after.
+ // This handles out-of-order completion scenarios where a later-requested
commit may complete
+ // before an earlier-requested one.
List<HoodieInstant> allInstants =
metaClient
.getActiveTimeline()
- .findInstantsAfter(commitInstant.getTimestamp())
+ .filter(
+ hoodieInstant ->
+ !hoodieInstant.isCompleted()
+ || InstantComparison.compareTimestamps(
Review Comment:
We've pulled the completion-time switch out of this PR. To keep backwards
compatibility, this PR pins the write table version to 6 (old timeline view)
and selects/orders instants by requested (instant) time. The completion-time
handling and its parameterized v6/v9 tests will land with table-version-9
support in the follow-up PR, per the plan discussed above.
##########
xtable-core/src/main/java/org/apache/xtable/hudi/HudiConversionSource.java:
##########
@@ -249,17 +262,23 @@ private CommitsPair
getCompletedAndPendingCommitsAfterInstant(HoodieInstant comm
if (completedInstants.isEmpty()) {
return CommitsPair.builder().completedCommits(completedInstants).build();
}
- // remove from pending instants that are larger than the last completed
instant.
+ // Remove pending instants that were requested after the last completed
instant finished.
+ // We compare requestedTime of pending instants against completionTime of
the last completed
+ // instant because pending instants don't have a completionTime yet. This
captures pending
+ // commits that were initiated before or during the last completed
commit's execution.
+ HoodieInstant lastCompletedInstant =
completedInstants.get(completedInstants.size() - 1);
Review Comment:
For table version 6 the active timeline (and `findInstantsAfter`) is ordered
by requested (instant) time, and that's the ordering `completedInstants`
inherits here — so `completedInstants.get(size-1)` is the last by requested
time, which is correct for the v6 semantics. Completion-time ordering and a
test covering out-of-order completion come with the table-version-9 work in the
follow-up PR; this PR deliberately stays on the v6 requested-time path.
##########
xtable-core/src/main/java/org/apache/hudi/stats/XTableValueMetadata.java:
##########
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.stats;
+
+import static
org.apache.xtable.model.schema.InternalSchema.MetadataKey.TIMESTAMP_PRECISION;
+import static
org.apache.xtable.model.schema.InternalSchema.MetadataValue.MICROS;
+
+import java.lang.reflect.Constructor;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.LocalDateTime;
+
+import org.apache.hudi.metadata.HoodieIndexVersion;
+
+import org.apache.xtable.model.schema.InternalSchema;
+import org.apache.xtable.model.stat.ColumnStat;
+
+public class XTableValueMetadata {
Review Comment:
Added — class-level and method-level javadoc are in `XTableValueMetadata`
now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]