You may need to add one more Geotools dependency
<dependency>
<groupId>org.geotools</groupId>
<artifactId>gt-epsg-extension</artifactId>
<version>24.0</version></dependency>
Note that, GeoTools library is not on Maven Central. You need to add
OSGEO repo to your POM, or sbt or your environment
<repositories>
<repository>
<id>osgeo</id>
<name>OSGeo Release Repository</name>
<url>https://repo.osgeo.org/repository/release/</url>
</repository>
</repositories>
On Thu, Apr 29, 2021 at 11:07 AM Campbell, Scott (IC) <
[email protected]> wrote:
> Hi,
>
> I am reading a .shp file that uses the Lambert Conformal Conic (S2P)
> coordinate system (EPSG:9802) as listed here: https://epsg.io/9802-method
>
> When I try to transform the coordinate system to EPSG:4326 it's throwing
> an except that it doesn't recognise the 9802 code.
>
> I've tried to search for more information about this projection but I
> don't have a lot of expertise in this area. Using other codes will provide
> a result (but with wrong coordinates) so that tells me that I have the
> environment setup properly. I'm wondering if Sedona doesn't support all of
> the EPSG codes?, or perhaps there are other libraries to include that
> contain more?
>
> I'm using Sedona for python with spark 3.0.2 and
> org.datasyslab:geotools-wrapper:geotools-24.0.
>
> Any information would be greatly appreciated.
>
> Incase it helps, here is the call to ST_Transform and the resulting stack
> trace:
>
> shp_rdd = ShapefileReader.readToGeometryRDD(spark.sparkContext,
> shp_location)
> shp_df = Adapter.toDf(shp_rdd, spark)
> shp_df.createOrReplaceTempView("shp_df")
>
> shp_df_wsg84 = spark.sql(
> """
> SELECT ST_Transform(geometry, "epsg:9802", "epsg:4326") as geometry_new
> FROM shp_df
> """
> )
>
> Py4JJavaError: An error occurred while calling o443.showString.
> : org.apache.spark.SparkException: Job aborted due to stage failure: Task
> 0 in stage 138.0 failed 1 times, most recent failure: Lost task 0.0 in
> stage 138.0 (TID 138, host.docker.internal, executor driver):
> org.opengis.referencing.NoSuchAuthorityCodeException: No code "EPSG:9802"
> from authority "European Petroleum Survey Group" found for object of type
> "CoordinateReferenceSystem".
> at
> org.geotools.referencing.factory.AbstractAuthorityFactory.noSuchAuthorityCode(AbstractAuthorityFactory.java:874)
> at
> org.geotools.referencing.factory.epsg.DirectEpsgFactory.createCoordinateReferenceSystem(DirectEpsgFactory.java:2244)
> at
> org.geotools.referencing.factory.BufferedAuthorityFactory.createCoordinateReferenceSystem(BufferedAuthorityFactory.java:731)
> at
> org.geotools.referencing.factory.AuthorityFactoryAdapter.createCoordinateReferenceSystem(AuthorityFactoryAdapter.java:779)
> at
> org.geotools.referencing.factory.FallbackAuthorityFactory.createCoordinateReferenceSystem(FallbackAuthorityFactory.java:624)
> at
> org.geotools.referencing.factory.AuthorityFactoryAdapter.createCoordinateReferenceSystem(AuthorityFactoryAdapter.java:779)
> at
> org.geotools.referencing.factory.ThreadedAuthorityFactory.createCoordinateReferenceSystem(ThreadedAuthorityFactory.java:635)
> at
> org.geotools.referencing.DefaultAuthorityFactory.createCoordinateReferenceSystem(DefaultAuthorityFactory.java:176)
> at org.geotools.referencing.CRS.decode(CRS.java:517)
> at org.geotools.referencing.CRS.decode(CRS.java:433)
> at
> org.apache.spark.sql.sedona_sql.expressions.ST_Transform.eval(Functions.scala:246)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
> Source)
> at
> scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
> at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:346)
> at
> org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
> at
> org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
> at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
> at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> at org.apache.spark.scheduler.Task.run(Task.scala:127)
> at
> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:462)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:465)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
> Driver stacktrace:
> at
> org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2059)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2008)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2007)
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2007)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:973)
> at
> org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:973)
> at scala.Option.foreach(Option.scala:407)
> at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:973)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2239)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2188)
> at
> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2177)
> at
> org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
> at
> org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:775)
> at
> org.apache.spark.SparkContext.runJob(SparkContext.scala:2114)
> at
> org.apache.spark.SparkContext.runJob(SparkContext.scala:2135)
> at
> org.apache.spark.SparkContext.runJob(SparkContext.scala:2154)
> at
> org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:472)
> at
> org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:425)
> at
> org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:47)
> at
> org.apache.spark.sql.Dataset.collectFromPlan(Dataset.scala:3627)
> at
> org.apache.spark.sql.Dataset.$anonfun$head$1(Dataset.scala:2697)
> at
> org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3618)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:100)
> at
> org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:160)
> at
> org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:87)
> at
> org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
> at
> org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
> at
> org.apache.spark.sql.Dataset.withAction(Dataset.scala:3616)
> at org.apache.spark.sql.Dataset.head(Dataset.scala:2697)
> at org.apache.spark.sql.Dataset.take(Dataset.scala:2904)
> at org.apache.spark.sql.Dataset.getRows(Dataset.scala:300)
> at
> org.apache.spark.sql.Dataset.showString(Dataset.scala:337)
> at sun.reflect.GeneratedMethodAccessor111.invoke(Unknown
> Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
> at
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
> at py4j.Gateway.invoke(Gateway.java:282)
> at
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> at py4j.commands.CallCommand.execute(CallCommand.java:79)
> at py4j.GatewayConnection.run(GatewayConnection.java:238)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.opengis.referencing.NoSuchAuthorityCodeException: No code
> "EPSG:9802" from authority "European Petroleum Survey Group" found for
> object of type "CoordinateReferenceSystem".
> at
> org.geotools.referencing.factory.AbstractAuthorityFactory.noSuchAuthorityCode(AbstractAuthorityFactory.java:874)
> at
> org.geotools.referencing.factory.epsg.DirectEpsgFactory.createCoordinateReferenceSystem(DirectEpsgFactory.java:2244)
> at
> org.geotools.referencing.factory.BufferedAuthorityFactory.createCoordinateReferenceSystem(BufferedAuthorityFactory.java:731)
> at
> org.geotools.referencing.factory.AuthorityFactoryAdapter.createCoordinateReferenceSystem(AuthorityFactoryAdapter.java:779)
> at
> org.geotools.referencing.factory.FallbackAuthorityFactory.createCoordinateReferenceSystem(FallbackAuthorityFactory.java:624)
> at
> org.geotools.referencing.factory.AuthorityFactoryAdapter.createCoordinateReferenceSystem(AuthorityFactoryAdapter.java:779)
> at
> org.geotools.referencing.factory.ThreadedAuthorityFactory.createCoordinateReferenceSystem(ThreadedAuthorityFactory.java:635)
> at
> org.geotools.referencing.DefaultAuthorityFactory.createCoordinateReferenceSystem(DefaultAuthorityFactory.java:176)
> at org.geotools.referencing.CRS.decode(CRS.java:517)
> at org.geotools.referencing.CRS.decode(CRS.java:433)
> at
> org.apache.spark.sql.sedona_sql.expressions.ST_Transform.eval(Functions.scala:246)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
> Source)
> at
> scala.collection.Iterator$$anon$10.next(Iterator.scala:459)
> at
> org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:346)
> at
> org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:872)
> at
> org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:872)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
> at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:349)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:313)
> at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> at org.apache.spark.scheduler.Task.run(Task.scala:127)
> at
> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:462)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:465)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> ... 1 more
>
>
> Thanks,
> Scott.
>
> --------------------------------------------------------------------
> Scott Campbell
> Researcher, Communications Research Centre Canada
> Innovation, Science and Economic Development Canada / Government of Canada
> [email protected] / Tel: 613-998-6641 / TTY: 1-866-694-8389
> Chercheur, Centre de recherches sur les communications Canada
> Innovation, Sciences et Développement économique Canada / Gouvernement du
> Canada
> [email protected] / Tél: 613-998-6641 / ATS: 1-866-694-8389
>
>
