Hello,
thank you for the detailed email.
I am more inclined to having it as a plugin.
One question I have though is whether one can enable/disable platforms within
the plugin. For instance, can I declare to use the spatial plugin but without
using the Spark platform. Is that possible?
Without plugins I would just not pass the ".withPlugin(Spark.basicPlugin())" in
my job, and then Wayang would execute the job in the rest of the platforms I
defined. If this is also possible with defining a new plugin then I would vote
that that's the best approach.
What do you think?
Best
--
Zoi
Στις Τετάρτη 14 Ιανουαρίου 2026 στις 02:37:57 μ.μ. CET, ο χρήστης Speer,
Maximilian <[email protected]> έγραψε:
Hello everyone,
as part of our master’s project in cooperation with Zoi Kaoudi, we are
extending Apache Wayang to support spatial data processing.
So far, we implemented:
* SpatialFilterOperator and SpatialJoinOperator
* Support for Java, Spark, and Postgres/JDBC
* A Geometry abstraction to translate between platform-specific geometry
representations (WKT, WBK, geoJson, JTS Geometry)
We are preparing a PR to the main repository and would like guidance on where
these new classes and their dependencies should be placed.
The core issue: dependencies
Our spatial operators require additional dependencies:
* JTS (small, lightweight)
* Apache Sedona (larger; needed for Spark-side spatial processing)
Since many Wayang users likely don’t need spatial functionality, we’re hesitant
to add these dependencies directly into the existing platform modules’ pom.xml
files.
Two structure options we see
Option A: Add spatial operators like regular operators
Place everything alongside existing operators and add the required dependencies
to the respective platform modules (Java/Spark/JDBC). This approach is
implemented in the repo linked below.
Option B: Implement spatial support as a plugin
Keep spatial functionality isolated as a plugin, including its dependencies,
e.g.:
wayang-plugins/
wayang-spatial/
src/main/java/.../spatial/
data/
mapping/
operators/
pom.xml
This approach seems clean, but we are not sure whether adding plugins is a
viable approach. Currently the only existing Plugin appears to be the IEJoin.
What we’d like your input on
1. Where should the spatial dependencies go?
Should we accept adding JTS/Sedona to the platform modules, or keep them
contained via a plugin?
2. Where should shared spatial types live?
Specifically:
* the Geometry class (used across platforms)
* the spatial predicate enum (filter/join predicates), used in API to
specify spatial operator type (eg. intersect, overlaps)
The PR in our own repository (operators added as normal operators) is here:
https://github.com/Spatial-Data-MP/incubator-wayang/pull/1
Please feel free to also comment on our general approach. Thanks a lot for your
feedback.
Best,
Anton, Felix, Max
