#30489: Django RasterField deserialization bug with pixeltype flags
-------------------------------+------------------------------------
Reporter: Ivor Bosloper | Owner: nobody
Type: Bug | Status: new
Component: GIS | Version: 2.2
Severity: Normal | Resolution:
Keywords: RasterField | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------+------------------------------------
Description changed by Ivor Bosloper:
Old description:
> After inserting some raster data with raster2pgsql into a Django model
> table with a RasterField column, I get a `list index out of range` when
> querying the table with a Django Queryset.
>
> {{{
> ...
> File "django/contrib/gis/db/models/fields.py" in from_db_value
> 360. return connection.ops.parse_raster(value)
> File "django/contrib/gis/db/backends/postgis/operations.py" in
> parse_raster
> 369. return from_pgraster(value)
> File "django/contrib/gis/db/backends/postgis/pgraster.py" in
> from_pgraster
> 57. pixeltype = POSTGIS_TO_GDAL[pixeltype]
> }}}
>
> It turns out the `pixeltype` value used is 39 while the `POSTGIS_TO_GDAL`
> list is only 16 elements long. The database field contains valid data but
> can not be deserialized with Django.
>
> **Steps for reproduction:**
>
> {{{
> # Django model
> class RasterModel(models.Model):
> rast = models.RasterField(srid=4326)
>
> # raw sql, single pixel raster with nodata bit set
> insert into app_rastermodel values(1, REPLACE('01 0000 0100
> 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> 0000000000000000 0000000000000000 E6100000 0100 0100 6 2 03 03', ' ',
> '')::raster);
>
> # query generating Exception
> RasterModel.objects.get(pk=1)
> }}}
>
> **Analysis**: if we look at the
> [https://trac.osgeo.org/postgis/wiki/WKTRaster/RFC/RFC1_V0SerialFormat
> Raster specification], the pixeltype is a byte of which the 4 highest
> bits are flags and the lowest 4 bits are the real pixeltype, but the
> Django deserialization code only considers one bit-flag:
>
> {{{
> # django/contrib/gis/db/backends/postgis/pgraster.py
> def from_pgraster(data):
> ...
> # Subtract nodata byte from band nodata value if it exists
> has_nodata = pixeltype >= 64
> if has_nodata:
> pixeltype -= 64
> ...
> }}}
>
> The erroneous pixeltype 39 in my example actually had the
> `BANDTYPE_FLAG_ISNODATA` (32) bit set which indicates all rastervalues
> are nodata.
>
> I have created (my first django) patch and hope somebody can assist me in
> getting it correct and merged.
New description:
After inserting some raster data with raster2pgsql into a Django model
table with a RasterField column, I get a `list index out of range` when
querying the table with a Django Queryset.
{{{
...
File "django/contrib/gis/db/models/fields.py" in from_db_value
360. return connection.ops.parse_raster(value)
File "django/contrib/gis/db/backends/postgis/operations.py" in
parse_raster
369. return from_pgraster(value)
File "django/contrib/gis/db/backends/postgis/pgraster.py" in from_pgraster
57. pixeltype = POSTGIS_TO_GDAL[pixeltype]
}}}
It turns out the `pixeltype` value used is 39 while the `POSTGIS_TO_GDAL`
list is only 16 elements long. The database field contains valid data but
can not be deserialized with Django.
**Steps for reproduction:**
{{{
# Django model
class RasterModel(models.Model):
rast = models.RasterField(srid=4326)
# raw sql, single pixel raster with nodata bit set
insert into app_rastermodel values(1, REPLACE('01 0000 0100
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000 E6100000 0100 0100 6 2 03 03', ' ',
'')::raster);
# query generating Exception
RasterModel.objects.get(pk=1)
}}}
**Analysis**: if we look at the
[https://trac.osgeo.org/postgis/wiki/WKTRaster/RFC/RFC1_V0SerialFormat
Raster specification], the pixeltype is a byte of which the 4 highest bits
are flags and the lowest 4 bits are the real pixeltype. Quoting the
specification:
{{{
Pixel type and storage flag
---------------------------
Pixel type specifies type of pixel values in a band.
Storage flag specifies whether the band data is stored
as part of the datum or is to be found on the server's
filesytem.
There are currently 11 supported pixel value types, so 4
bits are enough to account for all. We'll reserve
the upper 4 bits for generic flags and define upmost as
storage flag:
#define BANDTYPE_FLAGS_MASK 0xF0
#define BANDTYPE_PIXTYPE_MASK 0x0F
#define BANDTYPE_FLAG_OFFDB (1<<7)
#define BANDTYPE_FLAG_HASNODATA (1<<6)
#define BANDTYPE_FLAG_ISNODATA (1<<5)
#define BANDTYPE_FLAG_RESERVED3 (1<<4)
}}}
However, Django deserialization code only considers a single flag
(`BANDTYPE_FLAG_HASNODATA`, bit 6, value 64):
{{{
# django/contrib/gis/db/backends/postgis/pgraster.py
def from_pgraster(data):
...
# Subtract nodata byte from band nodata value if it exists
has_nodata = pixeltype >= 64
if has_nodata:
pixeltype -= 64
...
}}}
The erroneous pixeltype 39 in my example actually had the
`BANDTYPE_FLAG_ISNODATA` (bit 5, value 32) bit set which indicates all
rastervalues are nodata.
I have created (my first django) patch and hope somebody can assist me in
getting it correct and merged.
--
--
Ticket URL: <https://code.djangoproject.com/ticket/30489#comment:4>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-updates/070.edd2663f05627a657c4a032ce723a04c%40djangoproject.com.
For more options, visit https://groups.google.com/d/optout.
