Hello,
I'm having some sort of library compatibility issue that I can't track down.
We recently had to create a new Databricks cluster and I re-installed all my
previous libraries, but some python code is failing now. Any suggestions would
be appreciated.
Databricks setup:
10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12) - nothing changed here, same
specs that code was running on successfully on the old cluster
Libraries:
* apache-sedona==1.1.1 (I noticed it was broken with most recent version
and started back tracking versions, but haven't hit one that succeeds yet)
* geopandas
* org.apache.sedona:sedona-python-adapter-3.0_2.12:1.2.0-incubating
* org.datasyslab:geospark-sql_3.0:1.3.2-SNAPSHOT
* https://oss.sonatype.org/content/repositories/snapshots
I have a feeling I'm just on the wrong apache-sedona version, but my old
cluster didn't have the version specified so it's hard to tell which one I need
or if I'm missing something else.
Code:
%python
from sedona.core.SpatialRDD import PointRDD
from sedona.core.SpatialRDD import PolygonRDD
from sedona.utils.adapter import Adapter
from sedona.core.enums import GridType
from sedona.core.enums import IndexType
from sedona.core.spatialOperator import JoinQuery
considerBoundaryIntersection = True
usingIndex = True
buildOnSpatialPartitionedRDD = True
pointQuery = """Select ST_Point(cast(PositionLongitude as decimal(24,20)),
cast(PositionLatitude as decimal(24,20))) as Point
,UniqueId
,EquipmentID
--,MKT_AREA
from containerPings
"""
polygonQuery = """Select GEOM, LocationCode from Geometry"""
spatial_points_df = spark.sql(pointQuery)
pointRDD = Adapter.toSpatialRdd(spatial_points_df,"Point")
[cid:[email protected]]
[cid:[email protected]]
Jerrod Wagnon
Expert Logistics Engineer, E&T
479.820.2129(O)
www.jbhunt.com<https://www.jbhunt.com/?utm_source=Email&utm_medium=Email_Signature&utm_campaign=Email_Signature_Intermodal360&source=Email_Signature_Intermodal>
[cid:[email protected]]<https://www.facebook.com/jbhunttransport/>[Icon
Description automatically
generated]<https://www.linkedin.com/company/jb-hunt-transport-services-inc/>[A
white circle with a black background Description automatically generated with
medium
confidence]<https://twitter.com/jbhunt360?lang=en>[cid:[email protected]]<https://www.instagram.com/jbhunttransport/>
Intermodal | Dedicated | Final Mile | Flatbed | Truckload | Refrigerated | LTL
| Managed Logistics
This email contains confidential material for the sole use of the intended
recipient(s). Any review, use, distribution, or disclosure by others is
strictly prohibited. If you are not the intended recipient (or authorized to
receive for the recipient), please contact the sender by reply email and delete
all copies of this message.
