rangareddy opened a new issue, #921:
URL: https://github.com/apache/incubator-xtable/issues/921
`website/docs/how-to-spark-runtime.md` tells readers that the Delta
implementation is chosen for
them on Spark 3.5:
> Delta is the only engine whose implementation depends on the Spark
version, and the jar chooses
> the right one automatically
> | 3.5.x and newer | ✅ | Delta Kernel, selected automatically |
> Delta Standalone doesn't run on Spark 3.5, so on 3.5 and newer a Delta
source or target is
> routed through Delta Kernel with no flag needed.
That holds for the CLI entry point only. In `XTableSparkSync.main`, on the
`0.4.0-incubating` tag:
```java
boolean deltaInvolved =
TableFormat.DELTA.equals(sourceFormat) ||
targetFormats.contains(TableFormat.DELTA);
boolean useDeltaKernel = useDeltaKernelFlag;
if (!useDeltaKernel && deltaInvolved && isSparkAtLeast35(spark.version())) {
useDeltaKernel = true;
...
}
```
The in-job API has no such logic. `XTableSyncService.sync` reads the flag
straight off the spec:
```java
ConversionSourceProvider<?> sourceProvider =
sourceProviderFor(spec.getSourceFormat(), spec.isUseDeltaKernel());
```
and `TableSyncSpec.useDeltaKernel` is a plain `boolean` that defaults to
`false`. Nothing in
`xtable-spark-runtime` inspects the Spark version outside `XTableSparkSync`;
grepping the module
on the tag returns `isSparkAtLeast35` as the only call site.
### Impact
The page leads with the embedded API ("This is what the jar is for") and its
example syncs to
`DELTA`. A reader who copies that example onto Spark 3.5 gets no Kernel
auto-selection, falls
through to `DeltaConversionSourceProvider`, and hits the `delta-core` path
that the same page says
does not run on 3.5. The one sentence that does mention
`.useDeltaKernel(true)` frames it as a
Spark 3.4 opt-in, which points the reader away from the case where it is
actually required.
### Suggested fix
Separate the two entry points in the "Spark version support" section:
`XTableSparkSync`
auto-enables Kernel when Delta is involved and Spark is 3.5 or later, while
`XTableSyncService`
requires `.useDeltaKernel(true)` explicitly on Spark 3.5.
Split out of #920 so that issue stays scoped to the review comments on #913.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
