Hi,
I am giving a try to sedona SQL spatial functions in Scala in a databricks 7.3
environment (Spark 3.0.1, Scala 2.12) using the following libraries:
org.apache.sedona:sedona-core-3.0_2.12:1.0.1-incubating
org.apache.sedona:sedona-sql-3.0_2.12:1.0.1-incubating
org.apache.sedona:sedona-python-adapter-3.0_2.12:1.0.1-incubating
org.apache.sedona:sedona-viz-3.0_2.12:1.0.1-incubating
After session initialization,
val session = SparkSession.builder().
config("spark.serializer","org.apache.spark.serializer.KryoSerializer").
config("spark.kryo.registrator",
"org.apache.sedona.core.serde.SedonaKryoRegistrator").
master("local[*]").appName("test").getOrCreate()
SedonaSQLRegistrator.registerAll(session)
if I execute something like this in an spark static table, everything runs fine:
spark.sql("""
SELECT
p.*,
ST_GeomFromGeoJSON(p.geometry) as geom
FROM geometries p
WHERE ...)
But if I execute any ST_* function on a spark streaming table I am always
getting the following exception:
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 88.0 failed 4 times, most recent failure: Lost task 0.3 in
stage 88.0 (TID 5914) (10.139.64.9 executor 9):
org.apache.spark.SparkException: org.locationtech.jts.geom.Geometry is not
annotated with SQLUserDefinedType nor registered with UDTRegistration.} at
org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$1(RowEncoder.scala:88)
at scala.Option.getOrElse(Option.scala:189) at
org.apache.spark.sql.catalyst.encoders.RowEncoder$.serializerFor(RowEncoder.scala:87)
at
org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$3(RowEncoder.scala:176)
at
scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
I have very little experience using sedona so I assume I am doing something
wrong.
Any ideas on what clould be happening?
Do I really need to register org.locationtech.jts.geom.Geometry?
Thanks in advance,
Pablo
