vinishjail97 commented on code in PR #843: URL: https://github.com/apache/incubator-xtable/pull/843#discussion_r3617458751
########## rfc/rfc-3/rfc-3.md: ########## @@ -0,0 +1,234 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> +# RFC-3: xtable-spark-runtime - in-job metadata sync via a thin Spark bundle + +## Proposers + +- @vinishjail97 + +## Approvers +- Anyone from the XTable community can approve/add feedback. + +## Status + +GH Feature Request: https://github.com/apache/incubator-xtable/issues/836 + +> Please keep the status updated in `rfc/README.md`. + +## Abstract + +XTable's modules are published to Maven, so a user can depend on `xtable-core` and assemble their own +runtime today. But there is no maintained, thin, drop-in artifact — everyone re-solves the same +shading/classpath problem, or uses `xtable-utilities`, an unshaded ~1 GB fat jar that is not practical +to add to a Spark job and requires running a separate `RunSync` process with YAML config files. + +XTable conversion is metadata-only and lightweight, so the common case — "I already write this table +with Spark, keep it in sync in other formats" — should be a one-dependency, config-only addition to an +existing pipeline. This RFC proposes `xtable-spark-runtime`: a thin, relocated Spark bundle that +registers a driver-side listener and, after each successful write to a source table, runs an +incremental `ConversionController.sync(...)` for the configured targets. It complements — does not +replace — the standalone CLI (`RunSync`). + +## Background + +Everything needed to run a sync already exists in the engine; the gap is purely packaging and a trigger: + +- The entire sync entry surface is Hadoop-`Configuration`-only: `new ConversionController(conf).sync(config, sourceProvider)`. + The caller supplies only the **source** `ConversionSourceProvider`; target providers are created + internally by `ConversionTargetFactory` (`ConversionController.java`). Source providers instantiate + directly (`new HudiConversionSourceProvider()` / `IcebergConversionSourceProvider` / + `DeltaConversionSourceProvider`) and take `.init(conf)`. +- `SyncMode.INCREMENTAL` is self-healing: `ConversionController` auto-falls back to a full snapshot + per-target when there is no prior sync metadata or incremental is not safe (`isIncrementalSyncSufficient`). + First sync is effectively full; subsequent syncs are incremental. +- The sync watermark (last-synced instant, pending commits) is persisted in the **target's** + `TableSyncMetadata`. There is no additional state to keep on the client side. +- `xtable-hudi-support-extensions` is a pure-Java precedent for a Spark-adjacent module whose + `XTableSyncTool` already builds `SourceTable`/`TargetTable` and calls `ConversionController.sync(...)`; + it declares Spark deps as `provided` and uses a Scala-suffixed artifactId. +- Delta conversion currently runs Delta-on-Spark in-process (`delta-core`); `DeltaConversionUtils.buildSparkSession` + uses `SparkSession.builder()...getOrCreate()` with no `master`, so it reuses the host job's active + session rather than starting a second `SparkContext`. (A Spark-free Delta **Kernel** path exists and, + once it becomes the default, removes the per-Spark-version coupling — tracked separately.) + +## Implementation + +A new **pure-Java** module `xtable-spark-runtime_${scala.binary.version}` (package `org.apache.xtable.spark`). + +### What ships in v1 + +This PR delivers the **thin bundle plus the standalone `spark-submit` entry point +`XTableSparkSync`** — enough to run a relocated, in-cluster sync from one command. The config-only, +driver-side auto-sync (`XTableSyncListener` registered via `spark.sql.queryExecutionListeners`, with +`XTableSparkConfig` and `PlanTargetResolver`) is the headline on-ramp this RFC designs toward, but it +is **deferred to a follow-up PR** — those classes are not in v1. The listener design is retained +below as the target so the entry point and config schema stay aligned. + +### Activation — v1 (CLI entry point) Review Comment: Good call, thank you. Added an "Activation - from application code" section to RFC-3 in 91903423 with Scala/Java and PySpark snippets that build a TableSyncSpec and call XTableSyncService within an existing SparkSession. Folding these into the docs-site quickstart is noted as a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
