vinishjail97 commented on code in PR #801:
URL: https://github.com/apache/incubator-xtable/pull/801#discussion_r3229655583


##########
xtable-core/src/main/java/org/apache/xtable/delta/DeltaStatsUtils.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.delta;
+
+import java.io.IOException;
+import java.util.ArrayDeque;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Queue;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import lombok.AllArgsConstructor;
+import lombok.Value;
+import lombok.extern.log4j.Log4j2;
+
+import org.apache.commons.lang3.StringUtils;
+
+import com.fasterxml.jackson.annotation.JsonAnySetter;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.annotations.VisibleForTesting;
+
+import org.apache.xtable.collectors.CustomCollectors;
+import org.apache.xtable.model.exception.ParseException;
+import org.apache.xtable.model.schema.InternalField;
+import org.apache.xtable.model.schema.InternalSchema;
+import org.apache.xtable.model.schema.InternalType;
+import org.apache.xtable.model.stat.ColumnStat;
+import org.apache.xtable.model.stat.FileStats;
+import org.apache.xtable.model.stat.Range;
+
+/**
+ * Shared utility for Delta stats conversion between XTable internal format 
and Delta format. Used
+ * by both Delta Standalone and Delta Kernel implementations.
+ */
+@Log4j2
+public class DeltaStatsUtils {
+  private static final Set<InternalType> FIELD_TYPES_WITH_STATS_SUPPORT =
+      new HashSet<>(
+          Arrays.asList(
+              InternalType.BOOLEAN,
+              InternalType.DATE,
+              InternalType.DECIMAL,
+              InternalType.DOUBLE,
+              InternalType.INT,
+              InternalType.LONG,
+              InternalType.FLOAT,
+              InternalType.STRING,
+              InternalType.TIMESTAMP,
+              InternalType.TIMESTAMP_NTZ));
+
+  private static final ObjectMapper MAPPER = new ObjectMapper();
+
+  /* this data structure collects type names of all unrecognized Delta Lake 
stats. For instance
+  data file stats in presence of delete vectors would contain 'tightBounds' 
stat which is
+  currently not handled by XTable */
+  private static final Set<String> unsupportedStats = 
ConcurrentHashMap.newKeySet();

Review Comment:
   Thread-safety is fixed (`ConcurrentHashMap.newKeySet()` 👍), but this is 
still **global mutable static state** shared across all tables and across all 
tests in the same JVM. Concretely:
   - A test that asserts a key is *not* in `unsupportedStats` can pass or fail 
depending on which tests ran first.
   - Two tables with different schemas in the same process will pollute each 
other's unsupported-key set.
   
   Preferred: make this an instance field on `DeltaStatsUtils` (or whichever 
class owns parsing) and pass it through. If keeping static for caching reasons, 
at minimum expose a `@VisibleForTesting clear()` and call it from `@BeforeEach`.



##########
xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionTarget.java:
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.kernel;
+
+import static io.delta.kernel.utils.CloseableIterable.inMemoryIterable;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import lombok.Getter;
+import lombok.extern.log4j.Log4j2;
+
+import org.apache.hadoop.conf.Configuration;
+
+import scala.collection.JavaConverters;
+import scala.collection.Seq;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import io.delta.kernel.Operation;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.Transaction;
+import io.delta.kernel.TransactionBuilder;
+import io.delta.kernel.TransactionCommitResult;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.hook.PostCommitHook;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.Metadata;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.actions.RowBackedAction;
+import io.delta.kernel.internal.actions.SingleAction;
+import io.delta.kernel.types.StructField;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterable;
+import io.delta.kernel.utils.CloseableIterator;
+
+import org.apache.xtable.conversion.TargetTable;
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.exception.UpdateException;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.metadata.TableSyncMetadata;
+import org.apache.xtable.model.schema.InternalPartitionField;
+import org.apache.xtable.model.schema.InternalSchema;
+import org.apache.xtable.model.storage.InternalFilesDiff;
+import org.apache.xtable.model.storage.PartitionFileGroup;
+import org.apache.xtable.model.storage.TableFormat;
+import org.apache.xtable.spi.sync.ConversionTarget;
+
+/**
+ * Implementation of {@link ConversionTarget} for Delta Lake using the Delta 
Kernel API.
+ *
+ * <p>This implementation uses Delta Kernel (io.delta.kernel) instead of Delta 
Standalone for write
+ * operations, providing better compatibility with cloud storage (S3, GCS, 
Azure Blob Storage, HDFS)
+ * and improved support for Delta Lake 3.x features.
+ *
+ * <p><strong>Initialization:</strong> This class supports two initialization 
patterns:
+ *
+ * <ul>
+ *   <li><strong>Factory/ServiceLoader Pattern:</strong> Use the no-arg 
constructor followed by
+ *       {@link #init(TargetTable, Configuration)}. This is used by {@link
+ *       org.apache.xtable.conversion.ConversionTargetFactory}.
+ *   <li><strong>Direct Testing Pattern:</strong> Use the constructor with 
{@link TargetTable} and
+ *       {@link Engine} parameters for direct instantiation with custom 
dependencies in tests.
+ * </ul>
+ *
+ * <p><strong>Important:</strong> Do not mix initialization patterns. If you 
use the parameterized
+ * constructor, do not call {@link #init(TargetTable, Configuration)} 
afterward, as it will
+ * overwrite the custom Engine.
+ *
+ * <p><strong>Exception Handling:</strong> This implementation only catches 
{@link
+ * io.delta.kernel.exceptions.TableNotFoundException} when checking for table 
existence, allowing
+ * other exceptions (network errors, permission issues, corrupted metadata) to 
propagate rather than
+ * being silently masked. This ensures real errors are visible and fail fast.
+ *
+ * <p><strong>Known Limitations:</strong>
+ *
+ * <ul>
+ *   <li><strong>Commit Tags:</strong> Delta Kernel 4.0.0 does not support 
commit tags in commitInfo
+ *       (e.g., XTABLE_METADATA tags). This affects source-to-target commit 
identifier mapping.
+ *       Tracked in: https://github.com/apache/incubator-xtable/issues/819
+ *   <li><strong>Schema Evolution:</strong> Schema changes are handled through 
Delta Kernel's
+ *       transaction API, which may have different semantics compared to Delta 
Standalone.
+ *   <li><strong>Internal API Usage:</strong> This implementation casts to 
internal classes
+ *       (SnapshotImpl, TableImpl) to access metadata and commit history, as 
Delta Kernel 4.0.0
+ *       lacks public APIs for these operations. These casts are brittle and 
may break on version
+ *       upgrades. Public API alternatives should be used when available.
+ * </ul>
+ *
+ * <p><strong>Implementation Choice:</strong> Delta Kernel API was chosen over 
Delta Standalone to:
+ *
+ * <ul>
+ *   <li>Support newer Delta Lake features and protocol versions
+ *   <li>Align with the Delta Lake community's direction (Kernel is the 
recommended API)
+ *   <li>Reduce dependency on Spark-specific implementations
+ * </ul>
+ *
+ * @see ConversionTarget
+ * @see io.delta.kernel.Table
+ * @see io.delta.kernel.Transaction
+ */
+@Log4j2
+public class DeltaKernelConversionTarget implements ConversionTarget {
+  private static final String DELTA_LOG_RETENTION_DURATION = 
"delta.logRetentionDuration";
+
+  private DeltaKernelSchemaExtractor schemaExtractor;
+  private DeltaKernelPartitionExtractor partitionExtractor;
+  private DeltaKernelDataFileUpdatesExtractor dataKernelFileUpdatesExtractor;
+
+  private String basePath;
+  private long logRetentionInHours;
+  private DeltaKernelConversionTarget.TransactionState transactionState;
+  private Engine engine;
+
+  /**
+   * No-arg constructor for ServiceLoader instantiation. Must call {@link 
#init(TargetTable,
+   * Configuration)} before use.
+   */
+  public DeltaKernelConversionTarget() {}
+
+  /**
+   * Creates a fully initialized DeltaKernelConversionTarget with custom 
Engine. Typically used in
+   * tests. Do not call {@link #init(TargetTable, Configuration)} after this.
+   *
+   * @param targetTable the target table configuration
+   * @param engine custom Delta Kernel engine instance
+   */
+  public DeltaKernelConversionTarget(TargetTable targetTable, Engine engine) {
+    this(
+        targetTable.getBasePath(),
+        targetTable.getMetadataRetention().toHours(),
+        engine,
+        DeltaKernelSchemaExtractor.getInstance(),
+        DeltaKernelPartitionExtractor.getInstance(),
+        DeltaKernelDataFileUpdatesExtractor.builder()
+            .engine(engine)
+            .basePath(targetTable.getBasePath())
+            // Column statistics are not needed for conversion operations
+            .includeColumnStats(false)
+            .build());
+  }
+
+  @VisibleForTesting
+  DeltaKernelConversionTarget(
+      String tableDataPath,
+      long logRetentionInHours,
+      Engine engine,
+      DeltaKernelSchemaExtractor schemaExtractor,
+      DeltaKernelPartitionExtractor partitionExtractor,
+      DeltaKernelDataFileUpdatesExtractor dataKernelFileUpdatesExtractor) {
+    initInternal(
+        tableDataPath,
+        logRetentionInHours,
+        engine,
+        schemaExtractor,
+        partitionExtractor,
+        dataKernelFileUpdatesExtractor);
+  }
+
+  /**
+   * Private initialization helper to avoid code duplication between 
constructor and init() paths.
+   */
+  private void initInternal(
+      String tableDataPath,
+      long logRetentionInHours,
+      Engine engine,
+      DeltaKernelSchemaExtractor schemaExtractor,
+      DeltaKernelPartitionExtractor partitionExtractor,
+      DeltaKernelDataFileUpdatesExtractor dataKernelFileUpdatesExtractor) {
+    this.basePath = tableDataPath;
+    this.schemaExtractor = schemaExtractor;
+    this.partitionExtractor = partitionExtractor;
+    this.dataKernelFileUpdatesExtractor = dataKernelFileUpdatesExtractor;
+    this.engine = engine;
+    this.logRetentionInHours = logRetentionInHours;
+  }
+
+  @Override
+  public void init(TargetTable targetTable, Configuration configuration) {
+    Engine engine = DefaultEngine.create(configuration);

Review Comment:
   **Blocking (Tim +1'd this twice).** `init()` still unconditionally creates a 
new `DefaultEngine.create(configuration)` and overwrites any engine passed via 
the parameterized constructor. The class-level Javadoc now warns users not to 
mix initialization patterns, but documenting the footgun isn't the same as 
removing it — a misconfigured caller will silently lose their custom engine 
with no error.
   
   Suggested fix: either (a) make `init()` a no-op (or throw 
`IllegalStateException`) when `this.engine != null`, or (b) skip the 
`DefaultEngine.create` if an engine was already injected.
   
   ```java
   @Override
   public void init(TargetTable targetTable, Configuration configuration) {
     if (this.engine != null) {
       throw new IllegalStateException(
           "init() called on already-initialized DeltaKernelConversionTarget. "
               + "Use either the no-arg constructor + init(), or the 
parameterized constructor — not both.");
     }
     Engine engine = DefaultEngine.create(configuration);
     initInternal(...);
   }
   ```



##########
xtable-core/src/test/java/org/apache/xtable/kernel/TestDeltaKernelReadWriteIntegration.java:
##########
@@ -0,0 +1,488 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.kernel;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.time.Duration;
+import java.time.Instant;
+import java.time.temporal.ChronoUnit;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.Set;
+import java.util.UUID;
+
+import org.apache.hadoop.conf.Configuration;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.internal.SnapshotImpl;
+
+import org.apache.xtable.conversion.TargetTable;
+import org.apache.xtable.model.InternalSnapshot;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.schema.InternalField;
+import org.apache.xtable.model.schema.InternalPartitionField;
+import org.apache.xtable.model.schema.InternalSchema;
+import org.apache.xtable.model.schema.InternalType;
+import org.apache.xtable.model.schema.PartitionTransformType;
+import org.apache.xtable.model.stat.PartitionValue;
+import org.apache.xtable.model.stat.Range;
+import org.apache.xtable.model.storage.DataLayoutStrategy;
+import org.apache.xtable.model.storage.FileFormat;
+import org.apache.xtable.model.storage.InternalDataFile;
+import org.apache.xtable.model.storage.PartitionFileGroup;
+import org.apache.xtable.model.storage.TableFormat;
+import org.apache.xtable.spi.sync.TableFormatSync;
+
+/**
+ * Comprehensive end-to-end integration test for Delta Kernel read and write 
operations.
+ *
+ * <p>This test validates: 1. Writing data to Delta tables using 
DeltaKernelConversionTarget 2.
+ * Reading data from Delta tables using DeltaKernelConversionSource 3. 
Round-trip data integrity
+ * (write → read → validate) 4. Partitioned tables 5. Incremental updates 
(add/remove files) 6. Time
+ * travel (version-based reads) 7. Empty table handling
+ */
+public class TestDeltaKernelReadWriteIntegration {
+  private static final Random RANDOM = new Random();
+  private static final Instant LAST_COMMIT_TIME = Instant.now();
+
+  @TempDir public Path tempDir;
+  private Engine engine;
+
+  @BeforeEach
+  public void setup() {
+    Configuration hadoopConf = new Configuration();
+    engine = DefaultEngine.create(hadoopConf);
+  }
+
+  /**
+   * Test 1: Basic Write and Read Validates that data written to Delta can be 
read back correctly.
+   */
+  @Test
+  public void testBasicWriteAndRead() throws Exception {
+    String tableName = "test_basic_" + UUID.randomUUID();
+    Path basePath = tempDir.resolve(tableName);
+    Files.createDirectories(basePath);
+
+    // === WRITE PHASE ===
+    InternalSchema schema = createSimpleSchema();
+    DeltaKernelConversionTarget writer = createWriter(tableName, basePath);
+
+    // Create test data files
+    InternalDataFile file1 = createDataFile(1, Collections.emptyList(), 
basePath);
+    InternalDataFile file2 = createDataFile(2, Collections.emptyList(), 
basePath);
+
+    // Write data to Delta table
+    InternalTable writeTable = createInternalTable(tableName, basePath, 
schema, null);
+    InternalSnapshot snapshot = buildSnapshot(writeTable, "0", file1, file2);
+    
TableFormatSync.getInstance().syncSnapshot(Collections.singletonList(writer), 
snapshot);
+
+    // Verify Delta table was created
+    assertTrue(Files.exists(basePath.resolve("_delta_log")), "Delta log 
directory should exist");
+
+    // === READ PHASE ===
+    DeltaKernelConversionSource reader = createReader(tableName, basePath);
+
+    // Read current table metadata
+    InternalTable readTable = reader.getCurrentTable();
+    assertNotNull(readTable, "Should be able to read table");
+    assertEquals(tableName, readTable.getName());
+
+    // Normalize paths for comparison (handle file:// prefix differences)
+    String expectedPath = basePath.toString();
+    String actualPath = readTable.getBasePath().replace("file://", 
"").replace("file:", "");
+    assertTrue(
+        actualPath.endsWith(expectedPath) || actualPath.equals(expectedPath),
+        "Base path should match. Expected: " + expectedPath + ", Actual: " + 
actualPath);
+
+    // Verify schema
+    InternalSchema readSchema = readTable.getReadSchema();
+    assertNotNull(readSchema);
+    assertEquals(schema.getFields().size(), readSchema.getFields().size());

Review Comment:
   This still only compares `getFields().size()`, not content. Schema 
corruption that preserves field count (renamed field, swapped type, lost 
nullability, dropped metadata) goes undetected. Please use 
`assertEquals(schema, readSchema)` for structural equality. Same pattern on 
line 355.



##########
xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionTarget.java:
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.kernel;
+
+import static io.delta.kernel.utils.CloseableIterable.inMemoryIterable;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import lombok.Getter;
+import lombok.extern.log4j.Log4j2;
+
+import org.apache.hadoop.conf.Configuration;
+
+import scala.collection.JavaConverters;
+import scala.collection.Seq;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import io.delta.kernel.Operation;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.Transaction;
+import io.delta.kernel.TransactionBuilder;
+import io.delta.kernel.TransactionCommitResult;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.hook.PostCommitHook;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.Metadata;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.actions.RowBackedAction;
+import io.delta.kernel.internal.actions.SingleAction;
+import io.delta.kernel.types.StructField;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterable;
+import io.delta.kernel.utils.CloseableIterator;
+
+import org.apache.xtable.conversion.TargetTable;
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.exception.UpdateException;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.metadata.TableSyncMetadata;
+import org.apache.xtable.model.schema.InternalPartitionField;
+import org.apache.xtable.model.schema.InternalSchema;
+import org.apache.xtable.model.storage.InternalFilesDiff;
+import org.apache.xtable.model.storage.PartitionFileGroup;
+import org.apache.xtable.model.storage.TableFormat;
+import org.apache.xtable.spi.sync.ConversionTarget;
+
+/**
+ * Implementation of {@link ConversionTarget} for Delta Lake using the Delta 
Kernel API.
+ *
+ * <p>This implementation uses Delta Kernel (io.delta.kernel) instead of Delta 
Standalone for write
+ * operations, providing better compatibility with cloud storage (S3, GCS, 
Azure Blob Storage, HDFS)
+ * and improved support for Delta Lake 3.x features.
+ *
+ * <p><strong>Initialization:</strong> This class supports two initialization 
patterns:
+ *
+ * <ul>
+ *   <li><strong>Factory/ServiceLoader Pattern:</strong> Use the no-arg 
constructor followed by
+ *       {@link #init(TargetTable, Configuration)}. This is used by {@link
+ *       org.apache.xtable.conversion.ConversionTargetFactory}.
+ *   <li><strong>Direct Testing Pattern:</strong> Use the constructor with 
{@link TargetTable} and
+ *       {@link Engine} parameters for direct instantiation with custom 
dependencies in tests.
+ * </ul>
+ *
+ * <p><strong>Important:</strong> Do not mix initialization patterns. If you 
use the parameterized
+ * constructor, do not call {@link #init(TargetTable, Configuration)} 
afterward, as it will
+ * overwrite the custom Engine.
+ *
+ * <p><strong>Exception Handling:</strong> This implementation only catches 
{@link
+ * io.delta.kernel.exceptions.TableNotFoundException} when checking for table 
existence, allowing
+ * other exceptions (network errors, permission issues, corrupted metadata) to 
propagate rather than
+ * being silently masked. This ensures real errors are visible and fail fast.
+ *
+ * <p><strong>Known Limitations:</strong>
+ *
+ * <ul>
+ *   <li><strong>Commit Tags:</strong> Delta Kernel 4.0.0 does not support 
commit tags in commitInfo
+ *       (e.g., XTABLE_METADATA tags). This affects source-to-target commit 
identifier mapping.
+ *       Tracked in: https://github.com/apache/incubator-xtable/issues/819
+ *   <li><strong>Schema Evolution:</strong> Schema changes are handled through 
Delta Kernel's
+ *       transaction API, which may have different semantics compared to Delta 
Standalone.
+ *   <li><strong>Internal API Usage:</strong> This implementation casts to 
internal classes
+ *       (SnapshotImpl, TableImpl) to access metadata and commit history, as 
Delta Kernel 4.0.0
+ *       lacks public APIs for these operations. These casts are brittle and 
may break on version
+ *       upgrades. Public API alternatives should be used when available.
+ * </ul>
+ *
+ * <p><strong>Implementation Choice:</strong> Delta Kernel API was chosen over 
Delta Standalone to:
+ *
+ * <ul>
+ *   <li>Support newer Delta Lake features and protocol versions
+ *   <li>Align with the Delta Lake community's direction (Kernel is the 
recommended API)
+ *   <li>Reduce dependency on Spark-specific implementations
+ * </ul>
+ *
+ * @see ConversionTarget
+ * @see io.delta.kernel.Table
+ * @see io.delta.kernel.Transaction
+ */
+@Log4j2
+public class DeltaKernelConversionTarget implements ConversionTarget {
+  private static final String DELTA_LOG_RETENTION_DURATION = 
"delta.logRetentionDuration";
+
+  private DeltaKernelSchemaExtractor schemaExtractor;
+  private DeltaKernelPartitionExtractor partitionExtractor;
+  private DeltaKernelDataFileUpdatesExtractor dataKernelFileUpdatesExtractor;
+
+  private String basePath;
+  private long logRetentionInHours;
+  private DeltaKernelConversionTarget.TransactionState transactionState;
+  private Engine engine;
+
+  /**
+   * No-arg constructor for ServiceLoader instantiation. Must call {@link 
#init(TargetTable,
+   * Configuration)} before use.
+   */
+  public DeltaKernelConversionTarget() {}
+
+  /**
+   * Creates a fully initialized DeltaKernelConversionTarget with custom 
Engine. Typically used in
+   * tests. Do not call {@link #init(TargetTable, Configuration)} after this.
+   *
+   * @param targetTable the target table configuration
+   * @param engine custom Delta Kernel engine instance
+   */
+  public DeltaKernelConversionTarget(TargetTable targetTable, Engine engine) {
+    this(
+        targetTable.getBasePath(),
+        targetTable.getMetadataRetention().toHours(),
+        engine,
+        DeltaKernelSchemaExtractor.getInstance(),
+        DeltaKernelPartitionExtractor.getInstance(),
+        DeltaKernelDataFileUpdatesExtractor.builder()
+            .engine(engine)
+            .basePath(targetTable.getBasePath())
+            // Column statistics are not needed for conversion operations
+            .includeColumnStats(false)
+            .build());
+  }
+
+  @VisibleForTesting
+  DeltaKernelConversionTarget(
+      String tableDataPath,
+      long logRetentionInHours,
+      Engine engine,
+      DeltaKernelSchemaExtractor schemaExtractor,
+      DeltaKernelPartitionExtractor partitionExtractor,
+      DeltaKernelDataFileUpdatesExtractor dataKernelFileUpdatesExtractor) {
+    initInternal(
+        tableDataPath,
+        logRetentionInHours,
+        engine,
+        schemaExtractor,
+        partitionExtractor,
+        dataKernelFileUpdatesExtractor);
+  }
+
+  /**
+   * Private initialization helper to avoid code duplication between 
constructor and init() paths.
+   */
+  private void initInternal(
+      String tableDataPath,
+      long logRetentionInHours,
+      Engine engine,
+      DeltaKernelSchemaExtractor schemaExtractor,
+      DeltaKernelPartitionExtractor partitionExtractor,
+      DeltaKernelDataFileUpdatesExtractor dataKernelFileUpdatesExtractor) {
+    this.basePath = tableDataPath;
+    this.schemaExtractor = schemaExtractor;
+    this.partitionExtractor = partitionExtractor;
+    this.dataKernelFileUpdatesExtractor = dataKernelFileUpdatesExtractor;
+    this.engine = engine;
+    this.logRetentionInHours = logRetentionInHours;
+  }
+
+  @Override
+  public void init(TargetTable targetTable, Configuration configuration) {
+    Engine engine = DefaultEngine.create(configuration);
+
+    initInternal(
+        targetTable.getBasePath(),
+        targetTable.getMetadataRetention().toHours(),
+        engine,
+        DeltaKernelSchemaExtractor.getInstance(),
+        DeltaKernelPartitionExtractor.getInstance(),
+        DeltaKernelDataFileUpdatesExtractor.builder()
+            .engine(engine)
+            .basePath(targetTable.getBasePath())
+            .includeColumnStats(true)
+            .build());
+  }
+
+  @Override
+  public void beginSync(InternalTable table) {
+    this.transactionState =
+        new DeltaKernelConversionTarget.TransactionState(engine, 
logRetentionInHours);
+  }
+
+  @Override
+  public void syncSchema(InternalSchema schema) {
+    transactionState.setLatestSchema(schema);
+  }
+
+  @Override
+  public void syncPartitionSpec(List<InternalPartitionField> partitionSpec) {
+    if (partitionSpec != null) {
+      Map<String, StructField> spec =
+          partitionExtractor.convertToDeltaPartitionFormat(partitionSpec);
+      for (Map.Entry<String, StructField> partitionEntry : spec.entrySet()) {
+        String partitionColumnName = partitionEntry.getKey();
+        StructField partitionField = partitionEntry.getValue();
+
+        transactionState.addPartitionColumn(partitionColumnName);
+        if (partitionField != null
+            && transactionState.getLatestSchema().fields().stream()
+                .noneMatch(field -> 
field.getName().equals(partitionField.getName()))) {
+          // add generated columns to schema.
+          transactionState.addColumn(partitionField);
+        }
+      }
+    }
+  }
+
+  @Override
+  public void syncMetadata(TableSyncMetadata metadata) {
+    transactionState.setMetadata(metadata);
+  }
+
+  @Override
+  public void syncFilesForSnapshot(List<PartitionFileGroup> 
partitionedDataFiles) {
+    Table table = Table.forPath(engine, basePath);
+    // Pass cached snapshot to avoid reloading it
+    transactionState.setActions(
+        dataKernelFileUpdatesExtractor.applySnapshot(
+            table,
+            partitionedDataFiles,
+            transactionState.getLatestSchemaInternal(),
+            transactionState.getCachedSnapshot()));
+  }
+
+  @Override
+  public void syncFilesForDiff(InternalFilesDiff internalFilesDiff) {
+    // Use cached schema directly (already extracted during TransactionState 
initialization)
+    // For new tables, latestSchema is null and applyDiff will handle it 
appropriately
+    transactionState.setActions(
+        dataKernelFileUpdatesExtractor.applyDiff(
+            internalFilesDiff,
+            transactionState.getLatestSchemaInternal(),
+            basePath,
+            transactionState.getLatestSchema()));
+  }
+
+  @Override
+  public void completeSync() {
+    transactionState.commitTransaction();
+    transactionState = null;
+  }
+
+  @Override
+  public Optional<TableSyncMetadata> getTableMetadata() {
+    Table table = Table.forPath(engine, basePath);
+    Snapshot snapshot = table.getLatestSnapshot(engine);
+
+    // WORKAROUND: Cast to SnapshotImpl (internal class) to access metadata 
configuration.
+    // Delta Kernel 4.0.0 does not provide a public API to access table 
metadata/configuration.
+    // This cast is brittle and may break on Kernel version upgrades.
+    Metadata metadata = ((SnapshotImpl) snapshot).getMetadata();
+
+    // Get configuration from metadata
+    Map<String, String> configuration = metadata.getConfiguration();
+    String metadataJson = configuration.get(TableSyncMetadata.XTABLE_METADATA);
+
+    return TableSyncMetadata.fromJson(metadataJson);
+  }
+
+  @Override
+  public String getTableFormat() {
+    return TableFormat.DELTA;
+  }
+
+  @Override
+  public Optional<String> getTargetCommitIdentifier(String sourceIdentifier) {
+    // Delta Kernel 4.0.0 does not support commit tags in commitInfo, which 
are required for
+    // source-to-target commit identifier mapping. This limitation is 
documented in:
+    // https://github.com/delta-io/delta/issues/6167
+    // XTable tracking issue: 
https://github.com/apache/incubator-xtable/issues/819
+    //
+    // Unlike DeltaConversionTarget (which uses Delta Standalone with commit 
tag support),
+    // DeltaKernelConversionTarget cannot retrieve commit tags from Delta 
Kernel's API.
+    // Rather than silently scanning all commits (O(n) performance cost) and 
always returning
+    // empty, we explicitly throw an exception to indicate this feature is 
unsupported.
+    //
+    // When Delta Kernel adds commit tag support, this method can be 
reimplemented to:
+    // 1. Scan commit history using tableImpl.getChanges(engine, 0, 
currentVersion, actionSet)
+    // 2. Extract tags from CommitInfo.tags MapValue
+    // 3. Parse XTABLE_METADATA from tags and match sourceIdentifier
+    throw new NotSupportedException(
+        "Source-to-target commit identifier mapping is not supported in 
DeltaKernelConversionTarget. "
+            + "Delta Kernel 4.0.0 does not support commit tags in commitInfo. "
+            + "See: https://github.com/delta-io/delta/issues/6167";);
+  }
+
+  private class TransactionState {
+    private final Engine engine;
+    private final long retentionInHours;
+    private final List<String> partitionColumns;
+    @Getter private StructType latestSchema;
+    @Getter private InternalSchema latestSchemaInternal;
+    private TableSyncMetadata metadata;
+    private List<RowBackedAction> actions;
+
+    // Cache the snapshot to avoid multiple loads during a single sync cycle
+    // This snapshot is used for: schema extraction, table existence check, 
and file diff operations
+    private Snapshot cachedSnapshot;
+    private boolean tableExists;
+
+    private TransactionState(Engine engine, long retentionInHours) {
+      this.engine = engine;
+      this.partitionColumns = new ArrayList<>();
+      this.retentionInHours = retentionInHours;
+
+      try {
+        Table table = Table.forPath(engine, basePath);
+        this.cachedSnapshot = table.getLatestSnapshot(engine);
+        this.latestSchema = cachedSnapshot.getSchema();
+        this.tableExists = true;
+      } catch (TableNotFoundException e) {
+        // Expected: table doesn't exist yet on first sync
+        this.latestSchema = null;
+        this.cachedSnapshot = null;
+        this.tableExists = false;
+      }
+      // Let other exceptions propagate (network issues, permissions, 
corrupted metadata, etc.)
+    }
+
+    /**
+     * Adds a partition column name to the list. Package-private to allow 
access from outer class.
+     */
+    void addPartitionColumn(String columnName) {
+      partitionColumns.add(columnName);
+    }
+
+    /**
+     * Gets the cached snapshot. Returns null if no snapshot was cached (new 
table). Package-private
+     * to allow access from outer class.
+     */
+    Snapshot getCachedSnapshot() {
+      return cachedSnapshot;
+    }
+
+    void setMetadata(TableSyncMetadata metadata) {
+      this.metadata = metadata;
+    }
+
+    /**
+     * Sets the actions to be committed. Converts from Scala Seq to Java List 
for internal storage.
+     */
+    void setActions(Seq<RowBackedAction> scalaActions) {
+      this.actions = JavaConverters.seqAsJavaList(scalaActions);
+    }
+
+    private void addColumn(StructField field) {
+      latestSchema = latestSchema.add(field);
+      latestSchemaInternal = schemaExtractor.toInternalSchema(latestSchema);
+    }
+
+    private void setLatestSchema(InternalSchema schema) {
+      this.latestSchemaInternal = schema;
+      this.latestSchema = schemaExtractor.fromInternalSchema(schema);
+    }
+
+    private void commitTransaction() {
+      // Use cached table existence check instead of loading snapshot again
+      Operation operation = tableExists ? Operation.WRITE : 
Operation.CREATE_TABLE;
+
+      Table table = Table.forPath(engine, basePath);
+      TransactionBuilder txnBuilder =
+          table.createTransactionBuilder(engine, "XTable Delta Sync", 
operation);
+
+      // LIMITATION: Schema evolution for existing tables is NOT supported in 
Delta Kernel 4.0.0.
+      // The withSchema() method only works during CREATE_TABLE operations. 
For existing tables:
+      // - AddFile/RemoveFile actions are created using the old schema from 
existing snapshot
+      // - If source schema has evolved (columns added/removed/type changed), 
the Delta table
+      //   will have mismatched metadata and data, causing query failures or 
incorrect results
+      // This is a known Delta Kernel limitation: 
https://github.com/delta-io/delta/issues/4305
+      if (!tableExists) {
+        txnBuilder = txnBuilder.withSchema(engine, latestSchema);
+
+        if (!partitionColumns.isEmpty()) {
+          txnBuilder = txnBuilder.withPartitionColumns(engine, 
partitionColumns);
+        }
+      }
+
+      Map<String, String> tableProperties = getConfigurationsForDeltaSync();
+      txnBuilder = txnBuilder.withTableProperties(engine, tableProperties);
+
+      Transaction txn = txnBuilder.build(engine);
+      List<Row> allActionRows = new ArrayList<>();
+
+      // Iterate through actions (Java List) and convert to Row format
+      for (RowBackedAction action : actions) {
+
+        if (action instanceof AddFile) {
+          AddFile addFile = (AddFile) action;
+          Row wrappedRow = 
SingleAction.createAddFileSingleAction(addFile.toRow());
+          allActionRows.add(wrappedRow);
+        } else if (action instanceof RemoveFile) {
+          RemoveFile removeFile = (RemoveFile) action;
+          Row wrappedRow = 
SingleAction.createRemoveFileSingleAction(removeFile.toRow());
+          allActionRows.add(wrappedRow);
+        }
+      }
+
+      CloseableIterator<Row> allActionsIterator =
+          new CloseableIterator<Row>() {

Review Comment:
   Nit (non-blocking): the anonymous `CloseableIterator<Row>` (lines 439–455) 
is ~15 lines of boilerplate for what is conceptually `List → 
CloseableIterator`. Delta Kernel exposes utilities for this; if there's a 
`Utils.toCloseableIterator(...)` or similar, using it would drop this block to 
one line.



##########
xtable-core/src/main/java/org/apache/xtable/kernel/DeltaKernelConversionTarget.java:
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+package org.apache.xtable.kernel;
+
+import static io.delta.kernel.utils.CloseableIterable.inMemoryIterable;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+import lombok.Getter;
+import lombok.extern.log4j.Log4j2;
+
+import org.apache.hadoop.conf.Configuration;
+
+import scala.collection.JavaConverters;
+import scala.collection.Seq;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import io.delta.kernel.Operation;
+import io.delta.kernel.Snapshot;
+import io.delta.kernel.Table;
+import io.delta.kernel.Transaction;
+import io.delta.kernel.TransactionBuilder;
+import io.delta.kernel.TransactionCommitResult;
+import io.delta.kernel.data.Row;
+import io.delta.kernel.defaults.engine.DefaultEngine;
+import io.delta.kernel.engine.Engine;
+import io.delta.kernel.exceptions.TableNotFoundException;
+import io.delta.kernel.hook.PostCommitHook;
+import io.delta.kernel.internal.SnapshotImpl;
+import io.delta.kernel.internal.actions.AddFile;
+import io.delta.kernel.internal.actions.Metadata;
+import io.delta.kernel.internal.actions.RemoveFile;
+import io.delta.kernel.internal.actions.RowBackedAction;
+import io.delta.kernel.internal.actions.SingleAction;
+import io.delta.kernel.types.StructField;
+import io.delta.kernel.types.StructType;
+import io.delta.kernel.utils.CloseableIterable;
+import io.delta.kernel.utils.CloseableIterator;
+
+import org.apache.xtable.conversion.TargetTable;
+import org.apache.xtable.exception.NotSupportedException;
+import org.apache.xtable.exception.UpdateException;
+import org.apache.xtable.model.InternalTable;
+import org.apache.xtable.model.metadata.TableSyncMetadata;
+import org.apache.xtable.model.schema.InternalPartitionField;
+import org.apache.xtable.model.schema.InternalSchema;
+import org.apache.xtable.model.storage.InternalFilesDiff;
+import org.apache.xtable.model.storage.PartitionFileGroup;
+import org.apache.xtable.model.storage.TableFormat;
+import org.apache.xtable.spi.sync.ConversionTarget;
+
+/**
+ * Implementation of {@link ConversionTarget} for Delta Lake using the Delta 
Kernel API.
+ *
+ * <p>This implementation uses Delta Kernel (io.delta.kernel) instead of Delta 
Standalone for write
+ * operations, providing better compatibility with cloud storage (S3, GCS, 
Azure Blob Storage, HDFS)
+ * and improved support for Delta Lake 3.x features.
+ *
+ * <p><strong>Initialization:</strong> This class supports two initialization 
patterns:
+ *
+ * <ul>
+ *   <li><strong>Factory/ServiceLoader Pattern:</strong> Use the no-arg 
constructor followed by
+ *       {@link #init(TargetTable, Configuration)}. This is used by {@link
+ *       org.apache.xtable.conversion.ConversionTargetFactory}.
+ *   <li><strong>Direct Testing Pattern:</strong> Use the constructor with 
{@link TargetTable} and
+ *       {@link Engine} parameters for direct instantiation with custom 
dependencies in tests.
+ * </ul>
+ *
+ * <p><strong>Important:</strong> Do not mix initialization patterns. If you 
use the parameterized
+ * constructor, do not call {@link #init(TargetTable, Configuration)} 
afterward, as it will
+ * overwrite the custom Engine.
+ *
+ * <p><strong>Exception Handling:</strong> This implementation only catches 
{@link
+ * io.delta.kernel.exceptions.TableNotFoundException} when checking for table 
existence, allowing
+ * other exceptions (network errors, permission issues, corrupted metadata) to 
propagate rather than
+ * being silently masked. This ensures real errors are visible and fail fast.
+ *
+ * <p><strong>Known Limitations:</strong>
+ *
+ * <ul>
+ *   <li><strong>Commit Tags:</strong> Delta Kernel 4.0.0 does not support 
commit tags in commitInfo
+ *       (e.g., XTABLE_METADATA tags). This affects source-to-target commit 
identifier mapping.
+ *       Tracked in: https://github.com/apache/incubator-xtable/issues/819
+ *   <li><strong>Schema Evolution:</strong> Schema changes are handled through 
Delta Kernel's
+ *       transaction API, which may have different semantics compared to Delta 
Standalone.
+ *   <li><strong>Internal API Usage:</strong> This implementation casts to 
internal classes
+ *       (SnapshotImpl, TableImpl) to access metadata and commit history, as 
Delta Kernel 4.0.0
+ *       lacks public APIs for these operations. These casts are brittle and 
may break on version
+ *       upgrades. Public API alternatives should be used when available.
+ * </ul>
+ *
+ * <p><strong>Implementation Choice:</strong> Delta Kernel API was chosen over 
Delta Standalone to:
+ *
+ * <ul>
+ *   <li>Support newer Delta Lake features and protocol versions
+ *   <li>Align with the Delta Lake community's direction (Kernel is the 
recommended API)
+ *   <li>Reduce dependency on Spark-specific implementations
+ * </ul>
+ *
+ * @see ConversionTarget
+ * @see io.delta.kernel.Table
+ * @see io.delta.kernel.Transaction
+ */
+@Log4j2
+public class DeltaKernelConversionTarget implements ConversionTarget {
+  private static final String DELTA_LOG_RETENTION_DURATION = 
"delta.logRetentionDuration";
+
+  private DeltaKernelSchemaExtractor schemaExtractor;
+  private DeltaKernelPartitionExtractor partitionExtractor;
+  private DeltaKernelDataFileUpdatesExtractor dataKernelFileUpdatesExtractor;
+
+  private String basePath;
+  private long logRetentionInHours;
+  private DeltaKernelConversionTarget.TransactionState transactionState;
+  private Engine engine;
+
+  /**
+   * No-arg constructor for ServiceLoader instantiation. Must call {@link 
#init(TargetTable,
+   * Configuration)} before use.
+   */
+  public DeltaKernelConversionTarget() {}
+
+  /**
+   * Creates a fully initialized DeltaKernelConversionTarget with custom 
Engine. Typically used in
+   * tests. Do not call {@link #init(TargetTable, Configuration)} after this.
+   *
+   * @param targetTable the target table configuration
+   * @param engine custom Delta Kernel engine instance
+   */
+  public DeltaKernelConversionTarget(TargetTable targetTable, Engine engine) {
+    this(
+        targetTable.getBasePath(),
+        targetTable.getMetadataRetention().toHours(),
+        engine,
+        DeltaKernelSchemaExtractor.getInstance(),
+        DeltaKernelPartitionExtractor.getInstance(),
+        DeltaKernelDataFileUpdatesExtractor.builder()
+            .engine(engine)
+            .basePath(targetTable.getBasePath())
+            // Column statistics are not needed for conversion operations

Review Comment:
   Nit: the comment says *what* (`Column statistics are not needed for 
conversion operations`) but not *why*. Tim's original ask was to capture that 
`false` here is because the extractor built in the parameterized-constructor 
path is used to **read the existing set of files** during snapshot syncs — i.e. 
it's the read-side extractor, not the write-side one. The factory `init()` path 
uses `true` (line 214) for the write side. Calling that out inline would save a 
future reader the trip through both constructors.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


Reply via email to