vbhanuchander-lang opened a new pull request, #7859:
URL: https://github.com/apache/hop/pull/7859
Fixes #7544
Adds a **Spark SQL** transform to the native Spark pipeline engine (built on
the engine from #7486). Every incoming transform is registered as a temporary
view, so one statement can join, union, filter, window and aggregate them under
a single Catalyst plan — instead of chaining Hop Join/Filter/Calculator
transforms that Catalyst cannot optimize across.
As the issue notes, this is deliberately *not* the classic SQL transforms
(`ExecSql`, `Table Input`, `Dynamic SQL row`): those target RDBMS connections
over JDBC. This compiles SQL against the pipeline's in-flight `Dataset`s, so no
database connection is involved.
## Against the goals in the issue
| # | Goal | Where |
|---|------|-------|
| 1 | Dedicated `SparkSql` transform, native-Spark only | `SparkSqlMeta` —
`supportedEngines = {SparkConst.PLUGIN_ID}` |
| 2 | N enabled inputs as temp views; zero-input SQL allowed |
`SparkSqlHandler.registerInputViews` |
| 3 | Driver-side `${...}` substitution (string concat) |
`SparkSqlHandler.handleTransform` |
| 4 | Correct Hop `IRowMeta`; explicit non-empty field list |
`SparkSqlMeta.getFields` + `projectDeclaredFields` |
| 5 | Metrics/logging consistent with other handlers | `trackMetrics(...,
Role.TRANSFORM)`, `SparkBaseTransformHandler` |
| 6 | hop-run, GUI `local[*]`, `spark-submit` via `MainSpark` | no new
deploy mode — plain native handler |
| 7 | Unit tests + docs | see below |
Non-goals respected: no Structured Streaming SQL, no attempt to make the
JDBC transforms native, no full catalog work.
## Design notes
**View naming.** The default view name is derived from the incoming
transform name — characters outside `[A-Za-z0-9_]` become `_`, a leading digit
is prefixed. `Read orders (raw)` becomes `Read_orders__raw_`, which is correct
but awkward to type, so the dialog has an **Input views** grid to set an
explicit name per input. Two inputs resolving to the same view name is an error
naming both transforms, rather than one silently shadowing the other.
**Why the field list is required.** Hop resolves the row layout at design
time, before any `SparkSession` exists, so the statement's result schema is
genuinely unavailable to it. Declaring the fields keeps design-time metadata
correct and gives downstream generic mapPartitions transforms a layout that
matches the Dataset. The declared list also drives the output — fields are
selected in declared order and cast to the Spark type for the Hop type — so the
two cannot drift apart. Declaring a field the statement doesn't return fails
with the list of columns it did return.
**One shared-code change:**
`HopPipelineMetaToSparkConverter.lookupPreviousDataset()` becomes `public` so
the handler reuses the converter's target-stream rules (Filter/Switch routing)
rather than duplicating them. Happy to revert to a private copy if you'd rather
keep the surface closed.
## Tests
18 new tests, all on a local `SparkSession`, following
`SparkNativeHandlersTest`:
`SparkSqlHandlerTest` (12) — two-input join; union across 4 partitions under
one plan; view-name override; default sanitized naming; zero-input `VALUES`;
variable substitution; declared fields driving output order *and* cast; and the
error paths (missing declared field, empty field list, empty SQL, colliding
view names, invalid SQL reporting the registered views).
`SparkSqlMetaTest` (6) — view-name sanitization incl. leading digits and
null/empty, override lookup, and `getFields` replacing the incoming row.
Full module locally: **186 tests, 0 failures**. `spotless:apply` clean,
`apache-rat:check` reports 0 unapproved.
```
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0 --
SparkSqlHandlerTest
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0 -- SparkSqlMetaTest
[INFO] Tests run: 186, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS
```
## Docs
New `spark-sql.adoc` transform page (description, options, input views,
output fields, variables incl. a warning that substitution is textual,
limitations), registered in `nav.adoc` and `transforms.adoc`, plus a row in the
native-handler table in `getting-started-with-native-spark.adoc`.
## Open questions
- Should a **Get fields** button that runs the statement against a live
session land in v1, or stay out until there's a design for where the session
comes from at design time?
- Field-list ergonomics: happy to add an injection test in the style of
`SparkLakeTableInputMetaInjectionTest` if you want metadata injection coverage
for this transform too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
