This is an automated email from the ASF dual-hosted git repository.
bamaer pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 3306d79a3a Native Spark docs stale "Spark SQL not supported" row in
the limitations table (#9483)
3306d79a3a is described below
commit 3306d79a3ace60954ad4d8a02bd2cfefc4a7c873
Author: Sergio Ramazzina <[email protected]>
AuthorDate: Sat Nov 14 13:51:03 2026 +0100
Native Spark docs stale "Spark SQL not supported" row in the limitations
table (#9483)
---
.../ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc
index b939b16f43..fec32116e9 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/getting-started-with-native-spark.adoc
@@ -613,7 +613,7 @@ Examples: sorted *Group By* (use *Memory Group By*),
Beam-only I/O, barriers (*B
|*Not supported.* There is no Hop transform that maps to Spark’s structured
streaming / Kafka source-sink APIs. For Kafka-oriented Beam pipelines, use the
Beam engines (e.g. Beam Kafka consumer/producer). Hop’s own Kafka transforms
(if used) are not native-Spark Dataset I/O.
|Spark SQL
-|*Not supported* as a first-class surface. You cannot submit arbitrary Spark
SQL / `DataFrame` queries from Hop metadata; the engine only builds a Dataset
graph from the Hop transform graph (plus native handlers for a fixed set of
transforms).
+|*Supported* through the xref:pipeline/transforms/spark-sql.adoc[Spark SQL]
transform: it runs a `spark.sql` statement over the incoming Datasets, each
registered as a temporary view, and requires an explicit output field list.
Arbitrary `DataFrame` API code cannot be submitted from Hop metadata.
|MLlib / Spark ML
|*Not supported.* There are no native handlers or transforms for training,
pipelines, or model inference via Spark MLlib / `spark.ml`. Keep ML workloads
in Spark jobs outside Hop, or call external services from ordinary Hop
transforms where that fits.