Hi Pablo, Are you sure that you executed this line: SedonaSQLRegistrator.registerAll(session)?
Sedona SQL types and other types will be automatically registered via this line. THanks, Jia On Thu, Sep 2, 2021 at 9:35 AM Pablo C <[email protected]> wrote: > Hi, > > I am giving a try to sedona SQL spatial functions in Scala in a databricks > 7.3 environment (Spark 3.0.1, Scala 2.12) using the following libraries: > org.apache.sedona:sedona-core-3.0_2.12:1.0.1-incubating > org.apache.sedona:sedona-sql-3.0_2.12:1.0.1-incubating > org.apache.sedona:sedona-python-adapter-3.0_2.12:1.0.1-incubating > org.apache.sedona:sedona-viz-3.0_2.12:1.0.1-incubating > > After session initialization, > > val session = SparkSession.builder(). > > config("spark.serializer","org.apache.spark.serializer.KryoSerializer"). > config("spark.kryo.registrator", > "org.apache.sedona.core.serde.SedonaKryoRegistrator"). > master("local[*]").appName("test").getOrCreate() > SedonaSQLRegistrator.registerAll(session) > > if I execute something like this in an spark static table, everything runs > fine: > > spark.sql(""" > SELECT > p.*, > ST_GeomFromGeoJSON(p.geometry) as geom > FROM geometries p > WHERE ...) > > But if I execute any ST_* function on a spark streaming table I am always > getting the following exception: > > Caused by: org.apache.spark.SparkException: Job aborted due to stage > failure: Task 0 in stage 88.0 failed 4 times, most recent failure: Lost > task 0.3 in stage 88.0 (TID 5914) (10.139.64.9 executor 9): > org.apache.spark.SparkException: org.locationtech.jts.geom.Geometry is not > annotated with SQLUserDefinedType nor registered with UDTRegistration.} at > org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$1(RowEncoder.scala:88) > at scala.Option.getOrElse(Option.scala:189) at > org.apache.spark.sql.catalyst.encoders.RowEncoder$.serializerFor(RowEncoder.scala:87) > at > org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$3(RowEncoder.scala:176) > at > scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245) > at > scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) > > I have very little experience using sedona so I assume I am doing > something wrong. > Any ideas on what clould be happening? > Do I really need to register org.locationtech.jts.geom.Geometry? > > Thanks in advance, > Pablo >
