Thanks, Gregory. I think this behavior is not expected. We will look into this.
In fact, you should let us know about your situation early on. In fact, you can download the GeoTools jars manually and copy to SPARK_HOME/jars/ folder... You don't have to compile the code. Download links are given in the comments: http://sedona.apache.org/download/GeoSpark-All-Modules-Maven-Central-Coordinates/#geotools-240 We should make our doc more clear. On Thu, Feb 11, 2021 at 12:44 AM Grégory Dugernier <g...@aloalto.com> wrote: > Hi Jia, > > After much sweat and tears, I went the long road and compiled the code > locally. I'm working on Windows so I had to change a few things in the > POM.xml: > > - When trying to compile just the python-adapter lib, Maven didn't > like the dynamic versioning of sedona-core and sedona-sql, so I had to > hardcode the current version. > - For some reason, Maven couldn't find spark-version-converter from > within the python-adapter directory, so I just decided to compile the full > library. It might be possible to just compile the adapter, I just decided > pushing in this direction further seemed like it would take longer. > - When trying to compile the full library, the attach-javadoc goal > just keep erroring-out, even with the latest version of > maven-javadoc-plugin, so I just removed it entirely. > > By the end, I got the jar, uploaded it in Databricks and it works like a > charm so far. > > I did however meet another issue, it seems that when using > *ShapefileReader.readToGeometryRDD(spark.sparkContext, > file_url) *to read multiple Shapefiles files at once, then use the > Adapter, same-named columns aren't combined in the resulting DataFrame (see > example below). It might be normal RDD behavior -I have little experience > using them instead of DataFrames-, and I already found a workaround by > creating multiple dfs and using union(), but I prefer to let you know in > case it isn't the expected behavior. > [image: image.png] > > Regards, > Grégory > > On Thu, 11 Feb 2021 at 07:58, Jia Yu <ji...@apache.org> wrote: > >> Hi Gregory, >> >> Please let us know if you get your issue fixed. I know many of our users >> are also using Databricks cluster. We are also interested in the solution. >> >> Thanks, >> Jia >> >> On Wed, Feb 10, 2021 at 5:17 AM Grégory Dugernier <g...@aloalto.com> wrote: >> >>> Thank you for the quick reply! >>> >>> It seems my particular situation is a bit more complex than that, since >>> I'm running the notebook on a Databricks cluster, and the default spark >>> config doesn't seem to allow for more jar repositories (GeoTools isn't on >>> Maven Central), nor does creating a new SparkSession appears to work. I've >>> tried to download the jars and add them manually to the cluster but it >>> doesn't seem to work either. But at least I know where the issue's at! >>> >>> Thanks again for your help, >>> Regards >>> >>> On Wed, 10 Feb 2021 at 12:22, Jia Yu <ji...@apache.org> wrote: >>> >>>> Hi Gregory, >>>> >>>> Thanks for letting us know. This is not a bug. We cannot include >>>> GeoTools jars due to license issues. But indeed we forgot to update the >>>> docs and jupyter notebook examples. I just updated them. Please read them >>>> here: >>>> >>>> >>>> https://github.com/apache/incubator-sedona/blob/master/python/ApacheSedonaSQL.ipynb >>>> >>>> (Make sure you disable the browser cache or open it in an incognito >>>> window) >>>> http://sedona.apache.org/download/overview/#install-sedona-python >>>> >>>> In short, you need to add the following coordinates in the notebook: >>>> >>>> spark = SparkSession. \ builder. \ appName('appName'). \ config( >>>> "spark.serializer", KryoSerializer.getName). \ config( >>>> "spark.kryo.registrator", SedonaKryoRegistrator.getName). \ config( >>>> "spark.jars.repositories", 'https://repo.osgeo.org/repository/release,' >>>> 'https://download.java.net/maven/2'). \ config('spark.jars.packages', >>>> 'org.apache.sedona:sedona-python-adapter-3.0_2.12:1.0.0-incubating,' >>>> 'org.geotools:gt-main:24.0,' 'org.geotools:gt-referencing:24.0,' >>>> 'org.geotools:gt-epsg-hsql:24.0'). \ getOrCreate() >>>> >>>> On Wed, Feb 10, 2021 at 2:35 AM Grégory Dugernier <g...@aloalto.com> >>>> wrote: >>>> >>>>> Hello, >>>>> >>>>> I've been trying to run Sedona for Python on Databricks for 2 days and >>>>> I >>>>> think I've stumbled upon a bug. >>>>> >>>>> *Configuration*: >>>>> >>>>> - Spark 3.0.1 >>>>> - Scala 2.12 >>>>> - Python 3.7 >>>>> >>>>> *Librairies*: >>>>> >>>>> - apache-sedona (from PyPi) >>>>> - org.apache.sedona:sedona-python-adapter-3.0_2.12:1.0.0-incubating >>>>> (from Maven) >>>>> >>>>> *What I'm trying to do:* >>>>> >>>>> I'm trying to load a series of Shapefiles files into a dataframe for >>>>> geospatial analysis. See code snippet below, based of your example >>>>> notebook >>>>> < >>>>> https://github.com/apache/incubator-sedona/blob/master/python/ApacheSedonaCore.ipynb >>>>> > >>>>> >>>>> >>>>> > from sedona.core.formatMapper.shapefileParser import ShapefileReader >>>>> > from sedona.register import SedonaRegistrator >>>>> > from sedona.utils.adapter import Adapter >>>>> > >>>>> > SedonaRegistrator.registerAll(spark) >>>>> > shape_rdd = ShapefileReader.readToGeometryRDD(spark.sparkContext, >>>>> > file_name) >>>>> > df = Adapter.toDf(shape_rdd, spark) >>>>> > >>>>> >>>>> *Bug*: >>>>> >>>>> The ShapefileReader.readToGeometryRDD() currently throws the following >>>>> error: >>>>> >>>>> > Py4JJavaError: An error occurred while calling >>>>> > >>>>> z:org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader.readToGeometryRDD. >>>>> > : java.lang.NoClassDefFoundError: >>>>> org/opengis/referencing/FactoryException >>>>> > at >>>>> > >>>>> org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader.readToGeometryRDD(ShapefileReader.java:79) >>>>> > at >>>>> > >>>>> org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader.readToGeometryRDD(ShapefileReader.java:66) >>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at >>>>> > >>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>> > at >>>>> > >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> > at java.lang.reflect.Method.invoke(Method.java:498) at >>>>> > py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at >>>>> > py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380) at >>>>> > py4j.Gateway.invoke(Gateway.java:295) at >>>>> > py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) >>>>> at >>>>> > py4j.commands.CallCommand.execute(CallCommand.java:79) at >>>>> > py4j.GatewayConnection.run(GatewayConnection.java:251) at >>>>> > java.lang.Thread.run(Thread.java:748) Caused by: >>>>> > java.lang.ClassNotFoundException: >>>>> org.opengis.referencing.FactoryException >>>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at >>>>> > java.lang.ClassLoader.loadClass(ClassLoader.java:419) at >>>>> > >>>>> com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader.loadClass(ClassLoaders.scala:151) >>>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:352) >>>>> > : java.lang.NoClassDefFoundError: >>>>> org/opengis/referencing/FactoryException >>>>> > at >>>>> > >>>>> org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader.readToGeometryRDD(ShapefileReader.java:79) >>>>> > at >>>>> > >>>>> org.apache.sedona.core.formatMapper.shapefileParser.ShapefileReader.readToGeometryRDD(ShapefileReader.java:66) >>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>> > at >>>>> > >>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>> > at >>>>> > >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> > at java.lang.reflect.Method.invoke(Method.java:498) >>>>> > at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) >>>>> > at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380) >>>>> > at py4j.Gateway.invoke(Gateway.java:295) >>>>> > at >>>>> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) >>>>> > at py4j.commands.CallCommand.execute(CallCommand.java:79) >>>>> > at py4j.GatewayConnection.run(GatewayConnection.java:251) >>>>> > at java.lang.Thread.run(Thread.java:748) >>>>> > Caused by: java.lang.ClassNotFoundException: >>>>> > org.opengis.referencing.FactoryException >>>>> > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) >>>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:419) >>>>> > at >>>>> > >>>>> com.databricks.backend.daemon.driver.ClassLoaders$LibraryClassLoader.loadClass(ClassLoaders.scala:151) >>>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:352) >>>>> >>>>> >>>>> Adding the org.apache.sedona:sedona-core-3.0_2.12:1.0.0-incubating >>>>> library >>>>> from Maven doesn't solve the error. Adding the >>>>> org.datasyslab:geospark:1.3.1 >>>>> library from Maven solves the error, but it creates conflicts with the >>>>> underlying org.locationtech.jts dependencies. This makes me think >>>>> there is >>>>> a missing OpenGIS dependency in the sedona-python-adapter. >>>>> >>>>> Regards, >>>>> G. Dugernier >>>>> >>>>> -- >>>>> >>>>> >>>>> >>>>> Grégory Dugernier >>>>> Software Engineer >>>>> >>>>> g...@aloalto.com <f...@aloalto.com> >>>>> +32 (0)484 11 26 09 >>>>> >>>>> www.aloalto.com >>>>> +32 (0)2 736 10 17 >>>>> >>>>> -- >>>>> >>>>> >>>>> >>>>> >>>>> DISCLAIMER : The content of this e-mail >>>>> message does not constitute a >>>>> commitment of S.A. ALOALTO N.V. or its >>>>> subsidiaries/affiliates. This e-mail >>>>> and any attachments thereto may contain >>>>> information which is confidential >>>>> and/or protected by intellectual property >>>>> rights and are intended for the >>>>> intended recipient only. Any use of the >>>>> information contained herein >>>>> (including, but not limited to, total or partial >>>>> reproduction, >>>>> communication or distribution in any form) by persons other than >>>>> the >>>>> designated recipient(s) is prohibited. If an addressing or transmission >>>>> error has misdirected this e-mail, please notify the author, either by >>>>> telephone or by e-mail and delete the material from any computer. >>>>> >>>>> >>> >>> -- >>> >>> >>> >>> Grégory Dugernier >>> Software Engineer >>> >>> g...@aloalto.com <f...@aloalto.com> >>> +32 (0)484 11 26 09 >>> >>> www.aloalto.com >>> +32 (0)2 736 10 17 >>> >>> DISCLAIMER : The content of this e-mail message does not constitute a >>> commitment of S.A. ALOALTO N.V. or its subsidiaries/affiliates. This e-mail >>> and any attachments thereto may contain information which is confidential >>> and/or protected by intellectual property rights and are intended for the >>> intended recipient only. Any use of the information contained herein >>> (including, but not limited to, total or partial reproduction, >>> communication or distribution in any form) by persons other than the >>> designated recipient(s) is prohibited. If an addressing or transmission >>> error has misdirected this e-mail, please notify the author, either by >>> telephone or by e-mail and delete the material from any computer. >>> >> > > -- > > > > Grégory Dugernier > Software Engineer > > g...@aloalto.com <f...@aloalto.com> > +32 (0)484 11 26 09 > > www.aloalto.com > +32 (0)2 736 10 17 > > DISCLAIMER : The content of this e-mail message does not constitute a > commitment of S.A. ALOALTO N.V. or its subsidiaries/affiliates. This e-mail > and any attachments thereto may contain information which is confidential > and/or protected by intellectual property rights and are intended for the > intended recipient only. Any use of the information contained herein > (including, but not limited to, total or partial reproduction, > communication or distribution in any form) by persons other than the > designated recipient(s) is prohibited. If an addressing or transmission > error has misdirected this e-mail, please notify the author, either by > telephone or by e-mail and delete the material from any computer. >
