CommanderWahid commented on issue #2397:
URL: https://github.com/apache/sedona/issues/2397#issuecomment-3419800073
When using the Spark DataFrame API, this coexistence works well because
Sedona and Databricks spatial functions belong to different namespaces.
`
spark.conf.set("spark.databricks.geo.st.enabled", "true")
from pyspark.databricks.sql import functions as dbf
from sedona.spark import SedonaContext
from sedona.spark.sql import st_functions as sdstf
from sedona.spark.sql import st_constructors as sdcnf
from sedona.spark.sql import st_predicates as sdpcf
sedona = SedonaContext.create(spark)
`
However, in Spark SQL, since both libraries define functions with the same
names and Spark SQL does not support namespaces for function resolution, I am
wondering which implementation will Spark use internally?
I’m currently waiting for the LTS release of Databricks Runtime 17 to finish
testing different scenarios.
In the meantime I also discovered that the Carto team provides an Analytics
Toolbox for Databricks where all Sedona spatial functions should be used with a
custom prefix, such as "sedona_":
[Carto Analytics Toolbox for
Databricks](https://docs.carto.com/data-and-analysis/analytics-toolbox-for-databricks/reference)
They are using an **older Sedona version (1.5.1)**.
In case of using Spark SQL, this approach offers much greater flexibility
because most Databricks native ST_ functions are implemented on top of Photon,
which gives them strong performance advantages.
For functions that do not yet exist in Databricks’ spatial SQL or not yet
photonized, we can fall back to Sedona functions (using the "sedona_" prefix).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
