jiayuasu opened a new issue, #3218:
URL: https://github.com/apache/sedona/issues/3218
## Expected behavior
`Moran.getGlobal` / `Moran.get_global` should work when the value column is
an integer type (e.g. counts produced by `COUNT(*)`), or fail with a clear
message asking for a numeric cast.
## Actual behavior
When the value column is a `LONG` (or other integer type), the computation
fails with:
```
pyspark.errors.exceptions.captured.AnalysisException:
[CANNOT_UP_CAST_DATATYPE]
Cannot up cast `sum(z2ss_comp)` from "DECIMAL(38,6)" to "DOUBLE".
```
Root cause: in
`spark/common/src/main/scala/org/apache/sedona/stats/autocorrelation/Moran.scala`,
the intermediate expressions (`z * z AS z2ss_comp`, line 95; summed at line
97) inherit decimal types when the input column is integral, but the results
are consumed through typed `Dataset.as[(Double, Double, Double)]` (lines 79 and
107), which cannot up-cast `DECIMAL(38,6)` to `DOUBLE`.
Casting the value column to `DOUBLE` before calling the function works, e.g.
`CAST(COUNT(*) AS DOUBLE)`, which confirms the diagnosis.
Suggested fix: defensively cast the value column to `DoubleType` at the
start of `getGlobal` (or document that the value column must be a
floating-point type). Counts are the most natural input for spatial
autocorrelation on aggregated data, so the integer case is common.
## Steps to reproduce the problem
```python
from sedona.spark import SedonaContext
from sedona.spark.stats.weighting import add_binary_distance_band_column
from sedona.spark.stats.autocorrelation.moran import Moran
sedona =
SedonaContext.create(SedonaContext.builder().master("local[*]").getOrCreate())
# any point dataset; aggregate to cells with an *integer* count column
pois = sedona.read.format("shapefile").load("docs/usecases/data")
pois.createOrReplaceTempView("pois")
cells = sedona.sql("""
SELECT ST_GeoHash(geometry, 5) AS id,
COUNT(*) AS poi_count, -- LONG: triggers the
failure
ST_Centroid(ST_Envelope_Aggr(geometry)) AS geometry
FROM pois GROUP BY ST_GeoHash(geometry, 5)
""")
weighted = add_binary_distance_band_column(
cells, 0.08, include_self=False, saved_attributes=["id", "poi_count"]
)
Moran.get_global(weighted, value_column="poi_count", id_column="id")
# -> AnalysisException: CANNOT_UP_CAST_DATATYPE ... DECIMAL(38,6) to DOUBLE
# Workaround: CAST(COUNT(*) AS DOUBLE) in the aggregation makes it succeed
# (verified: I = 0.2519, z = 12.69 on the bundled dataset above).
```
## Settings
Sedona version = master (1.9.1-SNAPSHOT); Moran introduced in 1.8.0
Apache Spark version = 3.4.4
Apache Flink version = N/A
API type = Python (the Scala path has the same typed casts)
Scala version = 2.12
JRE version = 11
Python version = 3.12
Environment = Standalone (local[*])
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
