jiayuasu opened a new issue, #3213:
URL: https://github.com/apache/sedona/issues/3213
## Describe the bug
A raster returned from a Python UDF cannot carry a NODATA value that differs
from its input, and Python and the JVM report different NODATA for the same
band.
`GridSampleDimensionSerializer.write()` writes a nodata double "for
interoperability with Python RasterType", but `read()` discards it and rebuilds
the sample dimension from the Kryo `Category[]` blob instead:
```java
double offset = input.readDouble();
double scale = input.readDouble();
input.readDouble(); // noDataValue is included in categories, so we just
skip it
input.readInt();
Category[] categories = kryo.readObject(input, Category[].class);
return new GridSampleDimension(description, categories, offset, scale);
```
`InDbSedonaRaster.with_bands()` replays those blobs from the source raster,
so the value Python declares in `bands_meta` never reaches the JVM. Two
consequences:
**1. Inherited NODATA silently corrupts derived rasters.** Narrowing keeps
`_category_blobs[:n_bands]`, so a mask carved out of a scene inherits band 1's
NODATA. Given a 4-band raster with NODATA `0` on band 1 and a UDF thresholding
it to a 0/1 mask, every `0` in the mask is treated as NODATA:
```python
@udf(returnType=RasterType())
def mask(raster):
return raster.with_bands((raster.as_numpy()[0] < 6).astype(np.float64))
# on a 12-pixel raster:
# RS_Count(out, 1, true) -> 6 (zeros skipped as NODATA)
# RS_Count(out, 1, false) -> 12
# RS_BandNoDataValue(out, 1) -> 0.0
```
There is no way to correct this from inside the UDF; `RS_SetBandNoDataValue`
afterwards is the only workaround.
**2. Python and the JVM disagree.** Widening replicates the last source
band's category blob while `bands_meta` records `NaN`. Widening a 4-band raster
whose band 4 has NODATA `-1` to 8 bands:
```
python raster.bands_meta -> [nan, nan, nan, -1.0, nan, nan, nan, nan]
sql RS_BandNoDataValue(out, 5..8) -> -1.0
```
## Expected behavior
The nodata double on the wire should be authoritative, and `with_bands()`
should accept an explicit NODATA — parity with `RS_MapAlgebra`'s `noDataValue`
argument.
## Proposed fix
Have `read()` reconcile the two via the existing
`RasterUtils.createSampleDimensionWithNoDataValue()`, which early-returns when
the values already agree. The JVM writer emits
`RasterUtils.getNoDataValue(sampleDimension)` — derived from the categories —
so JVM-to-JVM round trips reconcile to a no-op and are unaffected. This mirrors
the `reconcileColorModel()` that #2956 added for the colour model, for the same
class of Python-mutated-raster mismatch.
Then add `with_bands(new_data, nodata=...)` on the Python side, and make
widened bands report the value they will actually receive rather than `NaN`.
## Sedona version
master (`1.9.1-SNAPSHOT`). Introduced with the Python raster serializer in
#2956, so not present in any release yet.
## Environment
Reproduced with Spark 3.4 / Scala 2.12, `-Dgeotools`, pyspark 3.4.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
