vinishjail97 opened a new issue, #886: URL: https://github.com/apache/incubator-xtable/issues/886
### Feature Request / Improvement The Delta Kernel source and target are on `main` (#729, #801), but they are not exercised by the shared conversion harness. Today they are only covered by their own suites: `ITDeltaKernelConversionSource`, `TestDeltaKernelReadWriteIntegration`, `TestDeltaKernelSync`, plus the extractor, partition, schema and stats unit tests. `ITConversionController` is where every other source and target combination is validated for dataset equivalence across sync modes and partitioning, and the Kernel path does not appear in it at all. This came up on the dev@ thread for the Spark-free runtime, where the point was made that the dedicated suites are not the right bar and the shared harness is: https://lists.apache.org/thread/bx1xyh0ml30r60od1hdvhndyd16b375j Closing this gap is the precondition for making Kernel the default Delta path, so it needs doing regardless of how the packaging discussion lands. **Current state** `generateTestParametersForFormatsSyncModesAndPartitioning()` iterates source formats `HUDI, DELTA, ICEBERG, PAIMON`, crossed with `SyncMode.values()` and partitioned against unpartitioned. `getConversionSourceProvider(String sourceTableFormat)` switches on the format name and returns `DeltaConversionSourceProvider` for `DELTA`. There is no Kernel branch, and `DeltaKernelConversionSourceProvider` is not imported. On the target side, the implementation is chosen by `xtable.delta.target.use_kernel` in the target table's additional properties, which `DeltaConversionTargetConfig` reads and `ConversionTargetFactory` dispatches on. It defaults to `false`, so every existing case in this harness exercises Delta Standalone only. **Proposed scope** 1. Cover Delta as a **source** through Kernel, so `DeltaKernelConversionSourceProvider` runs the same equivalence checks that `DeltaConversionSourceProvider` runs today, across both sync modes and both partitioning cases. 2. Cover Delta as a **target** through Kernel, by setting `xtable.delta.target.use_kernel=true` in the target table properties for the Kernel variant, so both writers are validated. 3. Keep the Standalone cases as they are. The goal is both implementations passing the same assertions, not replacing one with the other. 4. Avoid doubling the whole matrix if it costs too much wall-clock time. A separate parameter, or a targeted subset of the existing cases, is fine as long as both sync modes and both partitioning cases are covered for each of source and target. **Out of scope, tracked separately** - Flipping the default of `xtable.delta.target.use_kernel`. - Re-verifying the recently reported Delta conversion issues against the Kernel path (#711, #779, #813, #860, #586, #641). Worth doing before the default changes, but it is not test-harness work. - Deletion vector parity for Kernel (#713). ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
