Hi Jia, yes, sure. Actually the ST_* functions are working nicely with static tables. The problem arises when I try to use them with streaming data.
Thanks! Pablo On 2021/09/03 04:52:21, Jia Yu <[email protected]> wrote: > Hi Pablo, > > Are you sure that you executed this line: > SedonaSQLRegistrator.registerAll(session)? > > Sedona SQL types and other types will be automatically registered via this > line. > > THanks, > Jia > > > On Thu, Sep 2, 2021 at 9:35 AM Pablo C <[email protected]> > wrote: > > > Hi, > > > > I am giving a try to sedona SQL spatial functions in Scala in a databricks > > 7.3 environment (Spark 3.0.1, Scala 2.12) using the following libraries: > > org.apache.sedona:sedona-core-3.0_2.12:1.0.1-incubating > > org.apache.sedona:sedona-sql-3.0_2.12:1.0.1-incubating > > org.apache.sedona:sedona-python-adapter-3.0_2.12:1.0.1-incubating > > org.apache.sedona:sedona-viz-3.0_2.12:1.0.1-incubating > > > > After session initialization, > > > > val session = SparkSession.builder(). > > > > config("spark.serializer","org.apache.spark.serializer.KryoSerializer"). > > config("spark.kryo.registrator", > > "org.apache.sedona.core.serde.SedonaKryoRegistrator"). > > master("local[*]").appName("test").getOrCreate() > > SedonaSQLRegistrator.registerAll(session) > > > > if I execute something like this in an spark static table, everything runs > > fine: > > > > spark.sql(""" > > SELECT > > p.*, > > ST_GeomFromGeoJSON(p.geometry) as geom > > FROM geometries p > > WHERE ...) > > > > But if I execute any ST_* function on a spark streaming table I am always > > getting the following exception: > > > > Caused by: org.apache.spark.SparkException: Job aborted due to stage > > failure: Task 0 in stage 88.0 failed 4 times, most recent failure: Lost > > task 0.3 in stage 88.0 (TID 5914) (10.139.64.9 executor 9): > > org.apache.spark.SparkException: org.locationtech.jts.geom.Geometry is not > > annotated with SQLUserDefinedType nor registered with UDTRegistration.} at > > org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$1(RowEncoder.scala:88) > > at scala.Option.getOrElse(Option.scala:189) at > > org.apache.spark.sql.catalyst.encoders.RowEncoder$.serializerFor(RowEncoder.scala:87) > > at > > org.apache.spark.sql.catalyst.encoders.RowEncoder$.$anonfun$serializerFor$3(RowEncoder.scala:176) > > at > > scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245) > > at > > scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) > > > > I have very little experience using sedona so I assume I am doing > > something wrong. > > Any ideas on what clould be happening? > > Do I really need to register org.locationtech.jts.geom.Geometry? > > > > Thanks in advance, > > Pablo > > >
