#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc
speed
--------------------------+---------------------------
Reporter: sprice | Owner: grass-dev@…
Type: enhancement | Status: new
Priority: normal | Milestone: 7.1.0
Component: Raster | Version: svn-trunk
Resolution: | Keywords: ZLIB LZ4 ZSTD
CPU: OSX/Intel | Platform: MacOSX
--------------------------+---------------------------
Comment (by wenzeslaus):
In the diff I see the following pattern (this if from
[source:grass/trunk/lib/raster/get_row.c#L84 lib/raster/get_row.c],
`read_data_fp_compressed()` function):
{{{
#!diff
- if ((size_t) G_zlib_read(fcb->data_fd, readamount, data_buf, bufsize)
!= bufsize)
+ cmp = G_alloca(readamount);
+
+ if (read(fcb->data_fd, cmp, readamount) != readamount) {
+ G_freea(cmp);
G_fatal_error(_("Error reading raster data for row %d of <%s>"),
row, fcb->name);
}
+ if (cmp[0] == '0') // flate.c::G_ZLIB_COMPRESSED_NO
+ // Uncompressed
+ memcpy(data_buf, cmp+1, bufsize);
+ else
+ if (cmp[0] == 3 || cmp[0] == 4)
+// LZ4_decompress_safe((char *)(cmp+1), (char *)data_buf,
+// readamount-1, bufsize);
+ LZ4_decompress_fast((char *)(cmp+1), (char *)data_buf, bufsize);
+ else if (cmp[0] == 5)
+ ZSTD_decompress(data_buf, bufsize, cmp+1, readamount-1);
+ else
+ G_zlib_expand(cmp+1, readamount-1, data_buf, bufsize);
+
+ G_freea(cmp);
+}
}}}
In words, `G_zlib_read()` function is replaced by a block of code. Similar
applies to `G_zlib_write()` function and to certain extent to
`G_zlib_expand()` and `zlib_compress()` as well.
My question is if it wouldn't be more advantageous to create some wrapper
which would take the all necessary inputs including compression type and
do the necessary switches and format specific things.
I suppose this would make easier to add the same compression options also
to 3D raster library where `G_zlib_read()` and `G_zlib_write()` functions
are already used ([source:grass/trunk/lib/raster3d/fpcompress.c#L718
lib/raster3d/fpcompress.c]) or perhaps even to some other code such as
[source:grass/trunk/lib/segment lib/segment] where no compression is used.
--
Ticket URL: <https://trac.osgeo.org/grass/ticket/2750#comment:13>
GRASS GIS <https://grass.osgeo.org>
_______________________________________________
grass-dev mailing list
[email protected]
http://lists.osgeo.org/mailman/listinfo/grass-dev
