Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+-
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  closed
  Priority:  normal   |  Milestone:  7.6.0
 Component:  Raster   |Version:  svn-releasebranch74
Resolution:  fixed|   Keywords:  ZLIB LZ4 ZSTD
   CPU:  All  |   Platform:  All
--+-
Changes (by neteler):

 * status:  new => closed
 * resolution:   => fixed


-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+-
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.6.0
 Component:  Raster   |Version:  svn-releasebranch74
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  All  |   Platform:  All
--+-
Changes (by neteler):

 * platform:  MacOSX => All
 * cpu:  OSX/Intel => All
 * milestone:  7.4.2 => 7.6.0


Comment:

 All including ZSTD compression is available and tested in the upcoming
 7.6.0 (see relbranch76).

 Thanks for much for all the contributions here! Closing.

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+-
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.4.1
 Component:  Raster   |Version:  svn-releasebranch74
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+-

Comment (by neteler):

 Concerning LZ4 compression: Shall this be backported?

  * r71999 libgis: update lz4 to 1.8.0

-- 
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+-
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.4.1
 Component:  Raster   |Version:  svn-releasebranch74
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+-
Changes (by neteler):

 * version:  svn-trunk => svn-releasebranch74


Comment:

 Compress NULL file by default since r71674. On 12 Nov 2017: Creation of
 the GRASS GIS 7.4 release branch (r71701), hence it is default also in
 7.4.0.

 Concerning ZSTD compression: backport candidate for 7.4.1.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.4.1
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by martinl):

 What is status of this issue?

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.4.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by Nikos Alexandris):

 Suggested tests implemented at
 https://github.com/NikosAlexandris/test_r_compress. Expectedly, tests will
 work with either `GRASS_COMPRESS_NULLS=0` or `GRASS_COMPRESS_NULLS=1`.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.2.1
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:61 neteler]:
 > Replying to [comment:60 mmetz]:
 > > Notes on NULL file compression could be added to the raster intro.
 >
 > Done in r70669 and r70670.
 >
 > Proposal: I would suggest to make NULL file compression the default in
 trunk.

 Suggested tests:

 * have a raster map with NULL cells without NULL compression

  export GRAS_COMPRESS_NULLS=1

  test the result of r.compress

  test the result of r.null -z

 * have a raster map with NULL cells with NULL compression

  unset GRAS_COMPRESS_NULLS

  test the result of r.compress

  test the result of r.null -z

 * have two raster maps, one with compressed NULL cells, one with
 uncompressed NULL cells

  export GRAS_COMPRESS_NULLS=1

  test the result of mapa + mapb

  unset GRAS_COMPRESS_NULLS

  test the result of mapa + mapb

 * add your tests here

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.2.1
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by lucadelu):

 > Replying to [comment:60 mmetz]:
 >
 > Done in r70669 and r70670.
 >
 > Proposal: I would suggest to make NULL file compression the default in
 trunk.

 +1

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.2.1
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 Replying to [comment:60 mmetz]:
 > Notes on NULL file compression could be added to the raster intro.

 Done in r70669 and r70670.

 Proposal: I would suggest to make NULL file compression the default in
 trunk.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.2.1
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:59 neteler]:
 > Replying to [comment:58 mmetz]:
 > > Maybe some documentation that BZIP2 is not always providing the
 highest compression ratio, it needs to be evaluated for each dataset
 separately.
 >
 > Probably the notes on BZIP2 are sufficient here?
 >
 > https://grass.osgeo.org/grass72/manuals/r.compress.html#compression-
 algorithm-details

 and here

 https://grass.osgeo.org/grass72/manuals/rasterintro.html#raster-
 compression

 That should be sufficient.

 Notes on NULL file compression could be added to the raster intro.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.2.1
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 Replying to [comment:58 mmetz]:
 > Maybe some documentation that BZIP2 is not always providing the highest
 compression ratio, it needs to be evaluated for each dataset separately.

 Probably the notes on BZIP2 are sufficient here?

 https://grass.osgeo.org/grass72/manuals/r.compress.html#compression-
 algorithm-details

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.2.1
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:57 wenzeslaus]:
 > This seems like a great success!
 >
 > Seth, do you want to add another method?
 >
 Other possible methods are LZ4H and lzma from xz, but IMHO the currently
 implemented methods are sufficient.

 > MarkusM, is there something else remaining?

 I don't think so.
 >
 > MarkusN, anything you want to change or document?

 Maybe some documentation that BZIP2 is not always providing the highest
 compression ratio, it needs to be evaluated for each dataset separately.
 >
 > If everybody is OK, we can close this.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.2.1
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 This seems like a great success!

 Seth, do you want to add another method?

 MarkusM, is there something else remaining?

 MarkusN, anything you want to change or document?

 If everybody is OK, we can close this.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.2.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 r.compress manual updated in r69403, r69404.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.2.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 For the record - concerning NULL file compression:

 At time `cell_misc/null` is uncompressed by default.

 Using the environment variable `GRASS_COMPRESS_NULLS=1` NULL compression
 is activated for the session for newly created raster maps; and

 {{{
 export GRASS_COMPRESS_NULLS=1
 r.null -z raster_map
 }}}

 generates a new compressed file `cell_misc/nullcmpr` and removes the old
 uncompressed `cell_misc/null` file for existing maps.

 Note: At least GRASS GIS 7.2 is needed to read a raster map with
 compressed null file.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.2.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 Given that these improvements will be released as 7.2.0, the only missing
 issue seems to be the enabling of NULL file compression by default.

 Right?

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mlennert):

 Replying to [comment:50 mmetz]:
 > Replying to [comment:49 neteler]:
 > > Replying to [comment:48 mmetz]:
 > > > In your opinion:
 > > > * Adding new compression methods in G71 for cell/fcell files that
 are not readable by G70 is ok.
 > >
 > > No. If the format is broken, we need to call it G8 and provide
 converters.
 >
 > If we call it G8, converters are not needed because trunk as it is now
 can read G7 and G6 rasters.
 >
 > Assuming trunk should be released as G7.1:
 >
 > For full compatibility of relbr70 with trunk, not only read support for
 compressed null files would need to be backported, but also and more
 importantly read support for compressed cell/fcell files. These are new
 features at library level involving quite a few changes.

 -1 to backporting. Only bug fixes should go into relbr70.

 >
 > The current default settings in trunk are compatible with relbr70, and I
 do not see a reason to change the respective default settings in trunk.

 +1, as long as the default settings lead to compatible maps, then I do not
 think any backporting is needed.

 >What exactly is the reason why null file compression should be enabled by
 default in >trunk?

 Again +1. I don't understand all the effects and issues involved, but null
 file compression seems to be something that is mostly needed for specific
 big data situations. People dealing with such situations can alter
 specific settings.

 >
 > I will not backport the changes I did to trunk with regard to cell/cell
 file and null file compression because these are new features which should
 IMHO not go into an existing release branch.

 +1

 > Someone else would need to do the backporting.

 Please don't.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:49 neteler]:
 > Replying to [comment:48 mmetz]:
 > > In your opinion:
 > > * Adding new compression methods in G71 for cell/fcell files that are
 not readable by G70 is ok.
 >
 > No. If the format is broken, we need to call it G8 and provide
 converters.

 If we call it G8, converters are not needed because trunk as it is now can
 read G7 and G6 rasters.

 Assuming trunk should be released as G7.1:

 For full compatibility of relbr70 with trunk, not only read support for
 compressed null files would need to be backported, but also and more
 importantly read support for compressed cell/fcell files. These are new
 features at library level involving quite a few changes.

 The current default settings in trunk are compatible with relbr70, and I
 do not see a reason to change the respective default settings in trunk.
 What exactly is the reason why null file compression should be enabled by
 default in trunk?

 I will not backport the changes I did to trunk with regard to cell/cell
 file and null file compression because these are new features which should
 IMHO not go into an existing release branch. Someone else would need to do
 the backporting.

 IMHO, the new compression methods for cell/fcell files are more important
 than null file compression because the potential gains in disk space and
 processing speed are much higher. Recompressing a large cell file from
 ZLIB to BZIP2 can gain disk space several times larger than the
 corresponding uncompressed(!) null file. In relation, a compressed null
 file would only be a small added bonus but not a substantial gain in free
 disk space. On the other hand, compressing cell/fcell files with LZ4 can
 substantially speed up processing time.

 Short answer: release trunk now as G8, no need for converters because
 backward compatibility exists.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 Replying to [comment:48 mmetz]:
 > In your opinion:
 > * Adding new compression methods in G71 for cell/fcell files that are
 not readable by G70 is ok.

 No. If the format is broken, we need to call it G8 and provide converters.

 > * Compression of null files in G71 not readable by G70 is not ok.

 Yes (I find double negation sentences confusing, though).

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:47 neteler]:
 > Replying to [comment:46 mmetz]:
 > > Why have this also in relbr70? Backporting new features from trunk to
 a release branch is a bad idea and against GRASS policy,
 >
 > I wrote only that there must be one method to *read* compressed NULL
 files in G70. Not much of a big backport if I recall a personal discussion
 we had on this topic last year. That's all I meant above.

 Even only read support in G70 will require quite some changes to the
 rasterlib.
 >
 > Or we need to release trunk as G8 in order to be allowed to break raster
 readability compatibility completely.

 In your opinion:
 * Adding new compression methods in G71 for cell/fcell files that are not
 readable by G70 is ok.
 * Compression of null files in G71 not readable by G70 is not ok.

 That does not make sense to me. Please explain.

 BTW, the new compression methods for cell/fcell files available in trunk
 have a much larger impact on performance and disk space requirements than
 null file compression.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 Replying to [comment:46 mmetz]:
 > Why have this also in relbr70? Backporting new features from trunk to a
 release branch is a bad idea and against GRASS policy,

 I wrote only that there must be one method to *read* compressed NULL files
 in G70. Not much of a big backport if I recall a personal discussion we
 had on this topic last year. That's all I meant above.

 Or we need to release trunk as G8 in order to be allowed to break raster
 readability compatibility completely.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:45 neteler]:
 > Thanks. It would be good to now also pave the way to compress the NULL
 files by default (chose default method to be able to have read-support in
 relbr70).

 Why have this also in relbr70? Backporting new features from trunk to a
 release branch is a bad idea and against GRASS policy, I thought. You
 could instead release 7.1 if you want these features in a release branch.

 ZLIB is not really the best method for null bits compression. With large
 chunks of NULL values and/or large chunks of non-NULL values, the new RLE
 method seems to be as fast as no null file compression and provides the
 best compression. For special cases, LZ4 should compress better than RLE
 at the same speed.

 Some tests with a CELL map (MODIS LST Europe) with 18711 rows and 22195
 columns using r.compress (processing time and size of the null2 file):

 {{{
 no null compression
 14.8 sec, 51.9 MB
 RLE
 14.8 sec, 1.2 MB
 ZLIB
 16.8 sec, 1.4 MB
 LZ4
 14.8 sec, 1.4 MB
 BZIP2
 18.1 sec, 1.7 MB
 }}}

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 Thanks. It would be good to now also pave the way to compress the NULL
 files by default (chose default method to be able to have read-support in
 relbr70). See also wish above and here:

 
https://grasswiki.osgeo.org/wiki/GRASS_Community_Sprint_Paris_2016#Raster_compression

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:43 mmetz]:
 > TODO: documentation + r.compress

 I have updated r.compress to 1) compress a raster map again even if it is
 already compressed, 2) print the method used to compress a given raster.

 GRASS documentation in raster/rasterintro.html and
 raster/r.compress/r.compress.html now explains the currently available
 raster compression methods.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:41 neteler]:
 >
 > Overall, ZLIB at Level 1 performs well followed by LZ4 (BZIP2 compresses
 best but was very slow; ZLIB level 6's small size gain over ZLIB level 1
 is "paid" by performance).

 Now we have compression options for different use cases. For large raster
 maps (many columns, e.g. + 20,000), BZIP2 should provide the best
 compression and should require the least disk space. If disk space
 consumption and I/O speed do not matter, LZ4 could be used. The best
 compression method depends on the size of the raster to be compressed, the
 I/O reading and writing speed, and the available storage capacity. ZLIB
 level 1 provides a pretty good compromise. Optimizing raster map
 compression is now possible but requires testing with the current raster
 map(s) and the available storage infrastructure (I/O speed + capacity).

 TODO: documentation + r.compress

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by martinl):

 Replying to [comment:33 neteler]:
 > --> not yet reflected in the patch (so, the meaning of the var
 `GRASS_COMPRESS_NULLS` should be inverted or, better, implemented as
 `GRASS_NOCOMPRESS_NULLS=1`. I have terabytes of uncompressed NULL files at
 time and would like to get rid of wasting too much disk space.

 Or just perform NULL compression if environmental variable
 `GRASS_COMPRESS_NULLS` is not defined or defined and set to `1`, otherwise
 do not perform compression.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 Some MODIS LST benchmark, using a European LST coverage from
 http://gis.cri.fmach.it/eurolst/

 '''File size tests''': trunk, r67403 (build_date=2015-12-28)

 {{{
 # dataset
 r.univar -g lst_2002_196_average
 n=188341794
 null_cells=226948851
 cells=415290645
 min=-481
 max=4570
 ...


 
  RLE
 export GRASS_COMPRESSOR=RLE
 export GRASS_COMPRESS_NULLS=1

 ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
 -rw-r--r-- 1 neteler gis   10 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis 51923025 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/null
 -rw-r--r-- 1 neteler gis   44 Sep 22 15:15
 modis_lst_daily/cell_misc/lst_2002_196_average/timestamp
 -rw-r--r-- 1 neteler gis   10 Dec 28 15:51
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis  1423931 Dec 28 15:51
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/null2

 ls -la modis_lst_daily*/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 257730745 Jun  9  2014
 modis_lst_daily/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 529008029 Dec 28 15:51
 modis_lst_daily_compr/cell/lst_2002_196_average


 
  BZIP Level 1
 export GRASS_COMPRESSOR=ZLIB
 export GRASS_ZLIB_LEVEL=1
 export GRASS_COMPRESS_NULLS=1

 ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
 -rw-r--r-- 1 neteler gis   10 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis 51923025 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/null
 -rw-r--r-- 1 neteler gis   44 Sep 22 15:15
 modis_lst_daily/cell_misc/lst_2002_196_average/timestamp
 -rw-r--r-- 1 neteler gis   10 Dec 28 16:49
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis  1579674 Dec 28 16:49
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/null2

 ls -la modis_lst_daily*/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 257730745 Jun  9  2014
 modis_lst_daily/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 263256860 Dec 28 16:49
 modis_lst_daily_compr/cell/lst_2002_196_average


 
  BZIP Level 6 (default)
 export GRASS_COMPRESSOR=ZLIB
 export GRASS_ZLIB_LEVEL=6
 export GRASS_COMPRESS_NULLS=1

 ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
 -rw-r--r-- 1 neteler gis   10 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis 51923025 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/null
 -rw-r--r-- 1 neteler gis   44 Sep 22 15:15
 modis_lst_daily/cell_misc/lst_2002_196_average/timestamp
 -rw-r--r-- 1 neteler gis   10 Dec 28 17:54
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis  1358116 Dec 28 17:54
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/null2

 ls -la modis_lst_daily*/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 257730745 Jun  9  2014
 modis_lst_daily/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 257730745 Dec 28 17:54
 modis_lst_daily_compr/cell/lst_2002_196_average


 
  LZ4
 export GRASS_COMPRESSOR=LZ4
 export GRASS_COMPRESS_NULLS=1

 ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
 -rw-r--r-- 1 neteler gis   10 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis 51923025 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/null
 -rw-r--r-- 1 neteler gis   44 Sep 22 15:15
 modis_lst_daily/cell_misc/lst_2002_196_average/timestamp
 -rw-r--r-- 1 neteler gis   10 Dec 28 17:52
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis  1573603 Dec 28 17:52
 modis_lst_daily_compr/cell_misc/lst_2002_196_average/null2

 ls -la modis_lst_daily*/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 257730745 Jun  9  2014
 modis_lst_daily/cell/lst_2002_196_average
 -rw-r--r-- 1 neteler gis 362932376 Dec 28 17:52
 modis_lst_daily_compr/cell/lst_2002_196_average

 
  BZIP2
 export GRASS_COMPRESSOR=BZIP2
 export GRASS_COMPRESS_NULLS=1

 ls -la modis_lst_daily*/cell_misc/lst_2002_196_average/*
 -rw-r--r-- 1 neteler gis   10 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/range
 -rw-r--r-- 1 neteler gis 51923025 Jun  9  2014
 modis_lst_daily/cell_misc/lst_2002_196_average/null
 -rw-r--r-- 1 neteler gis   44 Sep 22 15:15
 

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 '''To summarize the current state'''

 We currently have the following compression methods in trunk (according to
 lib/gis/compress.c; set by `export GRASS_COMPRESSOR=XXX`):

  * `NONE` (uncompressed)
  * `RLE`  (generic Run-Length Encoding of single bytes)
  * `ZLIB` (DEFLATE, good speed and compression)
* with zlib compression levels (`export GRASS_ZLIB_LEVEL=X`): -1..9 (-1
 is default which is level 6)
* Notes from #comment:23: ZLIB level = 0 tells ZLIB to copy the data as
 is from source to destination. With CELL maps, the rasterlib will then
 still trim high zero bytes with trim_bytes() which can already reduce the
 data size considerably, but ZLIB will not compress the data.
  * `LZ4`  (fastest, low compression)
  * `BZIP2` (slowest, high compression)

 NULL file compression: At time it must be explicitly turned on (IMHO it
 should become the default) with
  * `export GRASS_COMPRESS_NULLS=1`

 Backward NULL file compression compatibility could be implemented in
 relbranch70 only by ZLIB's DEFLATE (see #comment:36). Hence ZLIB may
 qualify for the default NULL compression algorithm.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[Paulo van Breugel]

On 28-12-15 17:44, GRASS GIS wrote:

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
   Reporter:  sprice   |  Owner:  grass-dev@…
   Type:  enhancement  | Status:  new
   Priority:  normal   |  Milestone:  7.1.0
  Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
CPU:  OSX/Intel|   Platform:  MacOSX
--+---


I was assuming this is for all platforms, but see above MacOSX being 
ticked as platform?


Comment (by neteler):

  '''To summarize the current state'''

  We currently have the following compression methods in trunk (according to
  lib/gis/compress.c; set by `export GRASS_COMPRESSOR=XXX`):

   * `NONE` (uncompressed)
   * `RLE`  (generic Run-Length Encoding of single bytes)
   * `ZLIB` (DEFLATE, good speed and compression)
 * with zlib compression levels (`export GRASS_ZLIB_LEVEL=X`): -1..9 (-1
  is default which is level 6)
 * Notes from #comment:23: ZLIB level = 0 tells ZLIB to copy the data as
  is from source to destination. With CELL maps, the rasterlib will then
  still trim high zero bytes with trim_bytes() which can already reduce the
  data size considerably, but ZLIB will not compress the data.
   * `LZ4`  (fastest, low compression)
   * `BZIP2` (slowest, high compression)

  NULL file compression: At time it must be explicitly turned on (IMHO it
  should become the default) with
   * `export GRASS_COMPRESS_NULLS=1`
If I want to test this, is there an overview page how to use this? E.g., 
do I need to run the above every time I start GRASS? And will it then 
compress every NULL file when opening, or is there a way to batch 
compress all NULL files in a mapset for example. And if compressed, can 
I only open the raster layers when compression is (explicitly) enabled?




  Backward NULL file compression compatibility could be implemented in
  relbranch70 only by ZLIB's DEFLATE (see #comment:36). Hence ZLIB may
  qualify for the default NULL compression algorithm.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev


___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 (requires:  r65272, r65273, r65322, r65323, r65489, r65490, r65491, r65775
 from #2349)

 For the record, all changesets related to this ticket which I could
 identify (for a future backport):

 r67212, r67213, r67214, r67215, r67216, r67217, r67218, r67254, r67295,
 r67296

 Note: In a local backport, I found a merge issue between r67214 and r67215
 which I could not resolve. Any changeset missing from my list?

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:35 neteler]:
 > Sorry for the delay.
 >
 > Replying to [comment:34 sprice]:
 > > For "backward compatibility" do you mean files written by grass71 can
 be read by grass70?
 >
 > Yes, exactly.
 >
 > > Or would you like a similar patch to also work with the grass70
 branch?
 >
 > Perhaps in future, once we know that the grass71 implementation runs
 well on all OS.

 I have added different compression methods (LZ4, BZIP2) in r67212-16.
 Please test.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 Replying to [comment:36 mmetz]:
 > For relbr70, null files could be (un-)compressed with ZLIB's DEFLATE,
 other methods would not be supported.

 OK, also fine as long as one compliant solution exists.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:33 neteler]:
 > Replying to [comment:32 sprice]:
 > > Hey all, just checking in. What is remaining to get this pushed to the
 GRASS v7.1 code?
 >
 >
 > For me two things are yet open.
 >
 > First:
 >
 > Replying to [comment:30 neteler]:
 > > "User" comment: I would suggest to always activate the NULL file
 compression by default, anything else would be confusing.
 >
 > --> not yet reflected in the patch (so, the meaning of the var
 `GRASS_COMPRESS_NULLS` should be inverted or, better, implemented as
 `GRASS_NOCOMPRESS_NULLS=1`. I have terabytes of uncompressed NULL files at
 time and would like to get rid of wasting too much disk space.

 That concerns #2349.
 >
 > Second:
 >
 > We need a patch for GRASS GIS 7.0 for backward compatibility, at least
 for the NULL file compression. Which leads to the suggestion that either
 RLE or BZIP should be used here since these two compressors are already
 present in releasebranch70.

 No, neither the generic RLE algorithm nor BZIP support is available in
 relbr70. Note that CELL RLE compression is rasterlib-internal and tailored
 for integer numbers with variable byte size. Therefore it does not make
 sense to use the rasterlib-internal RLE method to compress null files
 where the informative unit is one bit, not 1-4 bytes. For relbr70, null
 files could be (un-)compressed with ZLIB's DEFLATE, other methods would
 not be supported.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 Sorryfor the delay.

 Replying to [comment:34 sprice]:
 > For "backward compatibility" do you mean files written by grass71 can be
 read by grass70?

 Yes, exactly.

 > Or would you like a similar patch to also work with the grass70 branch?

 Perhaps in future, once we know that the grass71 implementation runs well
 on all OS.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by sprice):

 For "backward compatibility" do you mean files written by grass71 can be
 read by grass70? Or would you like a similar patch to also work with the
 grass70 branch?

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 Replying to [comment:32 sprice]:
 > Hey all, just checking in. What is remaining to get this pushed to the
 GRASS v7.1 code?


 For me two things are yet open.

 First:

 Replying to [comment:30 neteler]:
 > "User" comment: I would suggest to always activate the NULL file
 compression by default, anything else would be confusing.

 --> not yet reflected in the patch (so, the meaning of the var
 `GRASS_COMPRESS_NULLS` should be inverted or, better, implemented as
 `GRASS_NOCOMPRESS_NULLS=1`. I have terabytes of uncompressed NULL files at
 time and would like to get rid of wasting too much disk space.

 Second:

 We need a patch for GRASS GIS 7.0 for backward compatibility, at least for
 the NULL file compression. Which leads to the suggestion that either RLE
 or BZIP should be used here since these two compressors are already
 present in releasebranch70.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by sprice):

 Hey all, just checking in. What is remaining to get this pushed to the
 GRASS v7.1 code?

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 "User" comment: I would suggest to always activate the NULL file
 compression by default, anything else would be confusing.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 Replying to [comment:26 mmetz]:
 > Data for the null2 file are now compressed with the same method like
 actual raster values. Because NULL data usually occur in long sequences of
 identical values, the new generic RLE algorithm I wrote for
 lib/gis/compress.c seems to perform best (highest compression, very fast).
 However, for consistency it would be better to use the same compressor for
 both the null bitsream and the raster data.

 For advanced users it seems that different compression for FCELL+DCELL and
 for NULL+CELL would make sense. If I have a MODIS LST for an island, then
 I want to use RLE for NULL files but BZIP2 for DCELL.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by glynn):

 Replying to [comment:23 mmetz]:

 > As before, the rasterlib makes only partial use of the generic
 compression methods: no compression and RLE is handled by the rasterlib
 internally, and RLE is not supported for fp maps. Creating uncompressed
 raster maps has been and should be only possible with
 `Rast_open_new_uncompressed()`.

 There's no fundamental reason why RLE can't be supported for FP maps; it's
 just seldom useful. RLE is most useful for category data where you have
 contiguous areas with the same value, and such maps tend to be integer
 maps (e.g. you can't create a reclass of a FP map).

 No compression used to be a special case because it means that maps don't
 have to be written sequentially: 6.x has G_put_map_row_random() which
 allows data to be written to an arbitrary row rather than to the "next"
 row. 7.x doesn't have an equivalent because nothing actually used it.

 In theory, even compressed maps can be written non-sequentially: just
 append the data to the end of the file then update the row pointer for the
 row. However, overwriting a row will result in the old data taking up
 space in the file.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 Thanks for the patch for BZIP2, I've tried to do it before but I did
 something wrong. Now it works and the tests are running:

 {{{
 GRASS_COMPRESSOR=BZIP2 ./bin.x86_64-unknown-linux-gnu/grass71
 /grassdata/nc_basic_spm_grass7/user1 --exec python -m grass.gunittest.main
 --location nc_basic_spm_grass7 --location-type nc
 }}}

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by mmetz):

 * Attachment "compressors.patch" added.

 minimal modification to add more compressors

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by mmetz):

 * Attachment "giscompress.tar.gz" added.

 new files for lib/gis to support different compressors

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 Replying to [comment:26 mmetz]:
 > The patches compressors.patch and compressors.patch are also updated to
 properly support NULL compression. Data for the null2 file are now
 compressed with the same method like actual raster values.

 Tested with `GRASS_COMPRESSOR=LZ4`, `GRASS_COMPRESSOR=BZIP2` using the
 [comment:27 command above] but as I mentioned here or in #2349, the test
 do not cover NULL files well.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:24 wenzeslaus]:
 > I was not able to make BZIP2 work for me. I would say I miss something
 like addition to `GISDEPS` but I was not able to fix it (getting undefined
 reference to some BZ function).

 You need to add $(BZLIB) to GISDEPS in include/Make/Grass.make. I will
 update bzipsupport.patch accordingly.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by mmetz):

 * Attachment "bzipsupport.patch" added.

 bzip2 support patch

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:20 wenzeslaus]:
 > Replying to [comment:19 mmetz]:
 > > I have implemented something like this recently and added support for
 LZ4 (and BZIP2) compression to my local copy of GRASS trunk.
 > >
 > > I am attaching a patch for trunk r66775 and an archive with new files
 to go to lib/gis
 >
 > The design in the patch looks really good. I did tests and benchmark but
 it was not as successful as I hoped for.
 >
 > The benchmark was the same as [comment:10 before] but modified for this
 patch. It is more for testing than benchmark anyway. It was on 30,000,000
 cells but perhaps the previous one was on more and it is not completely
 precise overall due to some other computations running at the same time
 (although the result is from 10 runs aggregated by ''perf'').
 >
 > || type || write || read ||
 > || NONE || 2.58 || 0.72 ||
 > || ZLIB || 1.52 || 0.93 ||
 > || LZ4 || 1.56 || 0.85 ||
 >

 Some more explanation about the proposed new mechanism:

 The proposed `G_compress()` interface provides a generic mechanism to data
 compression in libgis, not restricted to raster data but generic. Built-in
 compression methods would be no compression, RLE, ZLIB, ZL4. BZIP2
 compression would be available if GRASS is configured --with-bzlib. Other
 compression methods could be added by cloning lib/gis/flate.c and adding
 new `G_*_compress()` and `G_*_expand()` functions to
 lib/gis/compress.[h|c]. The raster lib does not need to be modified any
 more.

 As before, the rasterlib makes only partial use of the generic compression
 methods: no compression and RLE is handled by the rasterlib internally,
 and RLE is not supported for fp maps. Creating uncompressed raster maps
 has been and should be only possible with `Rast_open_new_uncompressed()`.

 That means that the behaviour of the rasterlib with using
 GRASS_COMPRESSOR=NONE needs to be defined: really use no compression or
 use default compression instead? Using GRASS_COMPRESSOR=RLE affects only
 new CELL maps. For fp maps, compression_type = 1 (RLE) is as before
 interpreted as ZLIB compression.

 If you want to know the original amount of data passed to any compressor,
 you need to use

 {{{
 GRASS_COMPRESSOR=ZLIB
 GRASS_ZLIB_LEVEL=0
 }}}

 ZLIB level = 0 tells ZLIB to copy the data as is from source to
 destination. With CELL maps, the rasterlib will then still trim high zero
 bytes with trim_bytes() which can already reduce the data size
 considerably, but ZLIB will not compress the data.

 I modified gislib_compressor_benchmark.sh to use
 {{{
 GRASS_COMPRESSOR=ZLIB
 GRASS_ZLIB_LEVEL=0
 }}}
 for no compression and discarded RLE because it is inefficient for CELL
 maps and not supported for fp maps. I tested also ZLIB levels 1 (fastest)
 and 6 (ZLIB default).

 I used the nc_basic_spm_grass7 location and set the region with
 {{{
 g.region -p rast=elevation res=2.5
 }}}
 resulting in 32,000,000 cells

 The test raster was generated with
 {{{
 r.mapcalc expression="test_rast_z_base = rand(double(-200.), 900)"
 seed=100
 }}}
 random numbers very difficult to compress.

 The write and read columns in the tables below have seconds as unit.

 || compressor || size MB || size % || write || read ||
 || NONE || 259.2 || 100 || 5.2 || 1.4 ||
 || ZLIB 1 || 247.9 || 95.6 || 14.3 || 2.5 ||
 || ZLIB 6 || 246.9 || 95.3 || 16.3 || 2.4 ||
 || LZ4 || 259.2 || 100 || 4.5 || 1.1 ||
 || BZIP2 || 249.4 || 96.2 || 63.4 || 19.9 ||
 LZ4 is the fastest method, no method is really the best because these
 random numbers could not be compressed to less than 95% of the original
 size.

 The next test raster was generated with
 {{{
 r.mapcalc expression="test_rast2_z_base = elevation"
 }}}
 which had 4x4 blocks of identical raster values, should be easy to
 compress

 || compressor || size MB || size % || write || read ||
 || NONE || 259.2 || 100 || 4.2 || 1.1 ||
 || ZLIB 1 || 41.8 || 16.1 || 4.7 || 1.6 ||
 || ZLIB 6 || 32.1 || 12.4 || 11.1 || 1.4 ||
 || LZ4 || 71.5 || 27.6 || 2.2 || 0.9 ||
 || BZIP2 || 49.8 || 19.2 || 28.0 || 7.1 ||

 LZ4 was again the fastest, and the best was ZLIB level 6. Here, the
 performance of BZIP2 was not convincing: by far the slowest and not as
 good as ZLIB.

 Then I tested with MODIS land surface temperature for Europe, a bit more
 than 400,000,000 cells:

 LST as CELL
 || compressor || size MB || size % || write || read ||
 || NONE || 829 || 100 || 28.5 || 14.8 ||
 || ZLIB 1 || 269 || 32.4 || 30.5 || 14.9 ||
 || ZLIB 6 || 261 || 31.5 || 36.7 

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:21 mmetz]:
 > Replying to [comment:20 wenzeslaus]:
 > > Replying to [comment:19 mmetz]:
 > > > I have implemented something like this recently and added support
 for LZ4 (and BZIP2) compression to my local copy of GRASS trunk.
 > > >
 > > > I am attaching a patch for trunk r66775 and an archive with new
 files to go to lib/gis
 > >

 > >
 > > I guess I'm missing something in configure because with BZIP2 it says
 "ERROR: GRASS needs to be compiled with BZIP2 for BZIP2 compression".
 >
 > You will need to hack in BZIP2 support, or make distclean, apply the
 patch "bzipsupport.patch", configure --with-bzlib, make

 ..and use the updated files in "giscompress.tar.gz"

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by mmetz):

 * Attachment "giscompress.tar.gz" added.

 new files for lib/gis to support different compressors

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:20 wenzeslaus]:
 > Replying to [comment:19 mmetz]:
 > > I have implemented something like this recently and added support for
 LZ4 (and BZIP2) compression to my local copy of GRASS trunk.
 > >
 > > I am attaching a patch for trunk r66775 and an archive with new files
 to go to lib/gis
 >
 > The design in the patch looks really good. I did tests and benchmark but
 it was not as successful as I hoped for.
 >
 > The benchmark was the same as [comment:10 before] but modified for this
 patch. It is more for testing than benchmark anyway. It was on 30,000,000
 cells

 I tested on 552,000,000 cells

 but perhaps the previous one was on more and it is not completely precise
 overall due to some other computations running at the same time (although
 the result is from 10 runs aggregated by ''perf'').
 >
 > || type || write || read ||
 > || NONE || 2.58 || 0.72 ||
 > || ZLIB || 1.52 || 0.93 ||
 > || LZ4 || 1.56 || 0.85 ||
 >
 > RLE and BZIP2 are missing because I got and error when writing using
 r.mapcalc with RLE:
 >
 {{{
 > *** Error in `r.mapcalc': free(): invalid next size (normal):
 0x00edacd0 ***
 > r.mapcalc: Aborted
 }}}

 Fixed in the updated patch "compressors.patch"
 >
 > The the commands are:
 >
 > {{{
 > export GRASS_COMPRESSOR=RLE
 > r.mapcalc expression="test_rast_z_base = rand(double(-200.), 900)"
 seed=100
 > r.mapcalc expression=test_rast_rle=double(test_rast_z_base)
 > }}}

 RLE compression is not and was never supported for fp maps. Also, if you
 use r.mapcalc's rand() function, you test the rand() function and not the
 compressor because all compressors are much faster than the rand()
 function. (BTW, there are nice fast random number generators in gsl).

 >
 > I guess I'm missing something in configure because with BZIP2 it says
 "ERROR: GRASS needs to be compiled with BZIP2 for BZIP2 compression".

 You will need to hack in BZIP2 support, or make distclean, apply the patch
 "bzipsupport.patch", configure --with-bzlib, make
 >
 > The report from `r.compress`:
 I have not updated `r.compress`. For `r.compress`, and for meaningful
 warning/error messages, something like `char *G_compressor_name(int
 compressor)` that gives the compressor name could be useful.
 >
 > When running the tests with
 >
 > {{{
 > grass71 /grassdata/ncarolina_smp_base/practice1 --exec python -m
 grass.gunittest.main--location ncarolina_smp_base --location-type nc
 > }}}
 >
 > I get
 >
 {{{
 > ERROR: Error reading raster data for row 0 of 
 }}}
 >
 > from many raster-related tests. Sometimes it is a different row number.
 The environmental variable `GRASS_COMPRESSOR` was not set. I didn't get if
 you already meant it to be backwards compatible with existing maps.

 Fixed in the new patch "compressors.patch". I meant it to be backwards
 compatible, but that one slipped through: old fp map compression is
 sometimes 1, sometimes 2, but both 1 and 2 mean ZLIB.

 Thanks for testing!

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by mmetz):

 * Attachment "compressors.patch" added.

 minimal modification to add more compressors

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by mmetz):

 * Attachment "bzipsupport.patch" added.

 bzip2 support patch

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 I was not able to make BZIP2 work for me. I would say I miss something
 like addition to `GISDEPS` but I was not able to fix it (getting undefined
 reference to some BZ function).

 I was able to run the tests successfully with LZ4 (on modified r66770):

 {{{
 GRASS_COMPRESSOR=LZ4 \
 ./bin.../grass71  /grassdata/nc_basic/practice1 --exec \
 python -m grass.gunittest.main --location nc_basic --location-type nc
 }}}

 The benchmarks look really good. I was getting quite good results before
 even on the random data, perhaps pure luck. But I wrongly recorded the
 region, it seems, so even with the seed I cannot reproduce; not important
 anyway.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by mmetz):

 * Attachment "compressors.patch" added.

 minimal modification to add more compressors

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by mmetz):

 Replying to [comment:14 glynn]:
 > Replying to [comment:13 wenzeslaus]:
 >
 > > My question is if it wouldn't be more advantageous to create some
 wrapper which would take the all necessary inputs including compression
 type and do the necessary switches and format specific things.
 >
 > Agreed.
 >
 > In practical terms, there are only two distinct cases: uncompressed
 (where the size of the data read or written matches the size of the data
 stored in the file) and compressed (where the sizes differ). Everything
 else is just options.

 Without knowing about this ticket, I have implemented something like this
 recently and added support for LZ4 (and BZIP2) compression to my local
 copy of GRASS trunk. The motivation was to have a compressor that is
 substantially faster than ZLIB but still better than RLE, and another
 compressor that is substantially better (higher compression) than ZLIB but
 not exceedingly slow. XZ with lzma2 is 1) too slow, 2) uses too much
 memory, 3) does not compress binary raster data better than BZIP2.

 In particular, I have added support for new compressors to gislib, not
 rasterlib. As before gislib does the actual compression, not rasterlib. My
 gislib now also handles LZ4 and BZIP2 compression. The actual change to
 rasterlib is to replace `G_zlib_compress()` with `G_compress(..., int
 compressor)` and `G_zlib_expand()` with `G_expand(..., int compressor)`.
 `G_zlib_write()` and `G_zlib_read()` are now `G_write_compressed(..., int
 compressor)` and `G_read_compressed(..., int compressor)`. Here, "..."
 means same arguments as before. The new argument "compressor" is actually
 "cellhd.compressed" with the same meaning as before. The internal function
 `zlib_compress` is no longer needed.

 As before, the compressor type is encoded in cellhd.compressed with
 previously 0: no compression, 1: RLE, 2: ZLIB, now also 3: LZ4, 4: BZIP2.

 r.univar results for CELL, FCELL, and DCELL maps are identical,
 independent of the compressor. The new gislib interface to compress data
 is generic and it is easy to add any other compressor, e.g. LZ4HC or ZSTD.

 Generally, any new compression method should go into gislib and not into
 rasterlib, just like ZLIB compression has been done by gislib. This keeps
 changes to the rasterlib to a minimum and makes debugging easier.

 For fast storage devices with plenty of space, LZ4 is by far the fastest,
 at the same time providing some reasonable compression where possible.

 For slow storage devices, e.g. accessed over network, BZIP2 compression is
 the fastest (yes, faster than LZ4) because the amount of data is the least
 (50% - 70% of ZLIB). That reduces network traffic and saves disk space.
 For my work, it would be a big advantage to use LZ4 for actual processing
 on fast local disks and BZIP2 for storing the final results on sometimes
 very slow network attached storage.

 The compressor type for new raster maps could be selected with one other
 environment variable GRASS_COMPRESSOR, e.g. GRASS_COMPRESSOR=LZ4

 I am not sure about the pro's and con's for using compressors other than
 ZLIB. ZLIB is a good compromise of speed and compression. Adding other
 compressors to G7.1 means that raster data compressed with a new method
 can not be opened by G7.0 or earlier. New compression types should, if
 added to G7.1, be clearly marked as "use it only if you really know what
 you are doing". I would profit from the choice of other compressors, but
 on a standard laptop/desktop system the current G7 default of ZLIB is
 probably the best alround solution.

 I am attaching a patch for trunk r66775 and an archive with new files to
 go to lib/gis

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by mmetz):

 * Attachment "compressors.patch" added.

 minimal modification to add more compressors

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 Replying to [comment:19 mmetz]:
 > I have implemented something like this recently and added support for
 LZ4 (and BZIP2) compression to my local copy of GRASS trunk.
 >
 > I am attaching a patch for trunk r66775 and an archive with new files to
 go to lib/gis

 The design in the patch looks really good. I did tests and benchmark but
 it was not as successful as I hoped for.

 The benchmark was the same as [comment:10 before] but modified for this
 patch. It is more for testing than benchmark anyway. It was on 30,000,000
 cells but perhaps the previous one was on more and it is not completely
 precise overall due to some other computations running at the same time
 (although the result is from 10 runs aggregated by ''perf'').

 || type || write || read ||
 || NONE || 2.58 || 0.72 ||
 || ZLIB || 1.52 || 0.93 ||
 || LZ4 || 1.56 || 0.85 ||

 RLE and BZIP2 are missing because I got and error when writing using
 r.mapcalc with RLE:

 {{{
 *** Error in `r.mapcalc': free(): invalid next size (normal):
 0x00edacd0 ***
 r.mapcalc: Aborted
 }}}

 The the commands are:

 {{{
 export GRASS_COMPRESSOR=RLE
 r.mapcalc expression="test_rast_z_base = rand(double(-200.), 900)"
 seed=100
 r.mapcalc expression=test_rast_rle=double(test_rast_z_base)
 }}}

 I guess I'm missing something in configure because with BZIP2 it says
 "ERROR: GRASS needs to be compiled with BZIP2 for BZIP2 compression".

 The report from `r.compress`:

 {{{
  is compressed (level 2: DEFLATE). Data type: 
  is uncompressed (level 0: NONE). Data type: 
  is compressed (level 2: DEFLATE). Data type: 
  is compressed (level 3: DEFLATE). Data type: 
 }}}

 When running the tests with

 {{{
 grass71 /grassdata/ncarolina_smp_base/practice1 --exec python -m
 grass.gunittest.main--location ncarolina_smp_base --location-type nc
 }}}

 I get

 {{{
 ERROR: Error reading raster data for row 0 of 
 }}}

 from many raster-related tests. Sometimes it is a different row number.
 The environmental variable `GRASS_COMPRESSOR` was not set. I didn't get if
 you already meant it to be backwards compatible with existing maps.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by wenzeslaus):

 * Attachment "gislib_compressor_benchmark.sh" added.

 Benchmark DCELL rasters with GRASS_COMPRESSOR

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by mmetz):

 * Attachment "giscompress.tar.gz" added.

 new files for lib/gis to support different compressors

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by sprice):

 This makes sense, but requires work that neither of us has time for. My
 main concern really is to get this merged & working before additional
 changes such as those discussed in #2762 & #2349 make merging difficult.
 What do you think about merging these changes now, and then upgrading
 other I/O functions as you have time?

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 Replying to [comment:17 sprice]:
 > My main concern really is to get this merged & working before additional
 changes such as those discussed in #2762 & #2349 make merging difficult.

 That's right, but I would like to see some more testing before merging it.
 Any chance somebody can do some? There are roughly three ways how to do
 that after applying the patch:

  * Set the variables (see above) and run the automatic tests on NC SPM and
 report the results back.
  * Run the benchmark Bash script attached to this ticket and compare with
 results already reported.
  * Set the variables and go through some your workflow and check the
 results.

 It seems to be that #2349 has all issues solved and committed, but lacks
 feedback/testing. Conflicts with #2762 would be probably solvable, but
 yes, it would be better to avoid them.

 > What do you think about merging these changes now, and then upgrading
 other I/O functions as you have time?

 This can work. The refactoring I'm suggesting shouldn't be difficult at
 the end, although we should take some time to do it right.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by sprice):

 I considered writing a more general function for compression but I held
 off for two reasons:

 1) There seems to be a lot of very specific code depending on exactly what
 you're doing. For example, for read_data_fp_compressed() there was
 functionality hidden in G_zlib_read to save a header for each compressed
 row. Or the use of G_ZLIB_COMPRESSED_NO ('0') as a flag for FP
 compression, vs various integers defining CELL compression. To keep
 backwards compatibility, I delt with the different formats (CELL vs NULL
 vs FCELL/DCELL) on a case-by-case basis.

 2) I'm not familiar with the best practices for the code in GRASS GIS. I'd
 encourage someone else to create new files (or libraries) as required. I
 can help as required, but I'm concerned that as each edge case is
 considered (as mentioned above) there won't be all that much common code
 between the various compression uses.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 Replying to [comment:15 sprice]:
 > I considered writing a more general function for compression but I held
 off for two reasons:
 >
 > 1) There seems to be a lot of very specific code depending on exactly
 what you're doing.

 My point is to put these specific cases into one function. So for example,
 whatever in the comment:13, should be in one function. What I suggest
 should be simple to moving of the existing code unless I miss some
 important part.

 > For example, for read_data_fp_compressed() there was functionality
 hidden in G_zlib_read to save a header for each compressed row.

 Here I perhaps hit my limited understanding of the code in question. What
 is doing this job now? Or it is not needed?

 > Or the use of G_ZLIB_COMPRESSED_NO ('0') as a flag for FP compression,
 vs various integers defining CELL compression.

 This is the kind of things which I would like to have hidden in one place.
 (In this way, we can keep the strange things at one place and e.g. easily
 replace, literals by defines (which we should do anyway, BTW).

 > To keep backwards compatibility, I delt with the different formats (CELL
 vs NULL vs FCELL/DCELL) on a case-by-case basis.

 You mean e.g. `read_data_compressed()` versus `read_data_fp_compressed()`?
 I had no time to actually try to identify all the differences, but the
 idea would be that these functions should deal with the differences among
 CELL, NULL and FCELL/DCELL while the proposed "`G_compresslib_write()`"
 function would deal with the differences between compression formats.

 > I'm concerned that as each edge case is considered (as mentioned above)
 there won't be all that much common code between the various compression
 uses.

 Another way to look at it is to think about, what do I have to replicate
 in 3D raster library where only `G_zlib_*` is used now. The things I need
 to replicate to enable other compressions should be in the
 "`G_compresslib_*()`" functions.

 I won't be able to give it attention it deserves in the following weeks,
 so please don't expect any actual code from me now. But feel free to
 oppose me or try it yourself in the mean time.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by glynn):

 Replying to [comment:13 wenzeslaus]:

 > My question is if it wouldn't be more advantageous to create some
 wrapper which would take the all necessary inputs including compression
 type and do the necessary switches and format specific things.

 Agreed.

 In practical terms, there are only two distinct cases: uncompressed (where
 the size of the data read or written matches the size of the data stored
 in the file) and compressed (where the sizes differ). Everything else is
 just options.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 In the diff I see the following pattern (this if from
 [source:grass/trunk/lib/raster/get_row.c#L84 lib/raster/get_row.c],
 `read_data_fp_compressed()` function):

 {{{
 #!diff
 -if ((size_t) G_zlib_read(fcb->data_fd, readamount, data_buf, bufsize)
 != bufsize)
 +cmp = G_alloca(readamount);
 +
 +if (read(fcb->data_fd, cmp, readamount) != readamount) {
 +G_freea(cmp);
 G_fatal_error(_("Error reading raster data for row %d of <%s>"),
   row, fcb->name);
  }

 +if (cmp[0] == '0') // flate.c::G_ZLIB_COMPRESSED_NO
 +// Uncompressed
 +memcpy(data_buf, cmp+1, bufsize);
 +else
 +if (cmp[0] == 3 || cmp[0] == 4)
 +//  LZ4_decompress_safe((char *)(cmp+1), (char *)data_buf,
 +//  readamount-1, bufsize);
 +LZ4_decompress_fast((char *)(cmp+1), (char *)data_buf, bufsize);
 +else if (cmp[0] == 5)
 +ZSTD_decompress(data_buf, bufsize, cmp+1, readamount-1);
 +else
 +G_zlib_expand(cmp+1, readamount-1, data_buf, bufsize);
 +
 +G_freea(cmp);
 +}
 }}}

 In words, `G_zlib_read()` function is replaced by a block of code. Similar
 applies to `G_zlib_write()` function and to certain extent to
 `G_zlib_expand()` and `zlib_compress()` as well.

 My question is if it wouldn't be more advantageous to create some wrapper
 which would take the all necessary inputs including compression type and
 do the necessary switches and format specific things.

 I suppose this would make easier to add the same compression options also
 to 3D raster library where `G_zlib_read()` and `G_zlib_write()` functions
 are already used ([source:grass/trunk/lib/raster3d/fpcompress.c#L718
 lib/raster3d/fpcompress.c]) or perhaps even to some other code such as
 [source:grass/trunk/lib/segment lib/segment] where no compression is used.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 Replying to [comment:11 sprice]:
 > Completely random data is not a good test because (in theory) it won't
 compress. However, it looks like it demonstrated the expected results.

 Yes, probably G7:r.surf.fractal or G7:r.random.surface are better, but
 G7:r.mapcalc with `rand()` is kind of worst-case scenario and the
 compression still worked well!

 > If you do a diff with n_les_assemble.c...

 I've created a separate ticket #2754 for this. Let's discuss it there.

 I've executed the tests with `GRASS_COMPRESS_NULLS=1` in combination with
 default compression and `GRASS_INT_LZ4=1` and they seems to be OK
 (although there is some issue with ''r.slope.aspect'' test but that's
 unrelated). But still, the test coverage is limited, user testing is
 needed.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 Replying to [comment:8 sprice]:
 > I think I've fixed all issues.

 It works for me as well. With `GRASS_INT_LZ4=1` I get

 {{{
 Executed 129 test files in 0:25:10.484136.
 From them 120 files (93%) were successful and 9 files (7%) failed.
 }}}

 without `GRASS_INT_LZ4` it is the same including the time (I don't expect
 that the improvement should be visible on speed of the tests, at least not
 much). (7% is little bit more than
 
[http://fatra.cnr.ncsu.edu/grassgistests/reports_for_date-2015-10-09-07-00/report_for_nc_basic_spm_grass7_nc/testfiles.html
 on the server] but the one test which is different is unrelated to this.)
 Now running tests also with the other compressions.

 > And a bad memory read that was in there before that valgrind discovered.
 And a segfault in n_les_assemble.c (via r.gwflow) that someone should
 check out.

 I don't understand you completely. `r.gwflow` doesn't fail with the
 current trunk and it doesn't fail even with your changes now for me. Does
 it fail for you?

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by sprice):

 * Attachment "lz4_zstd4.tgz" added.


--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by sprice):

 I think I've fixed all issues. And a bad memory read that was in there
 before that valgrind discovered. And a segfault in n_les_assemble.c (via
 r.gwflow) that someone should check out. I've uploaded a new set of files.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by wenzeslaus):

 * Attachment "raster_compress_benchmark.sh" added.

 Benchmark DCELL

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 The tests are running well for me also with `GRASS_INT_LZ4HC=1`,
 `GRASS_INT_ZSTD=1` and `GRASS_INT_ZLIB=0` (RLE). I haven't tried
 `GRASS_COMPRESS_NULLS=1`.

 Here is a report (polished) created by the attached benchmark script based
 on what you posted which uses completely random data. I have used
 `GRASS_COMPRESS_NULLS=1` and region with 30,000,000 cells. The disk was
 SSD, OS was Linux.

 {{{
 #!rst
 ZLIB compression writing
 

 Performance counter stats for 'r.mapcalc
 expression=test_rast_orig=double(test_rast_z_base)' (10 runs):

 ::


   10415.903368 task-clock (msec) #0.993 CPUs utilized
 ( +-  0.36% )
   2798 context-switches  #0.269 K/sec
 ( +-  2.11% )
 23 cpu-migrations#0.002 K/sec
 ( +-  4.20% )
 325702 page-faults   #0.031 M/sec
 ( +-  0.00% )
30804357778 cycles#2.957 GHz
 ( +-  0.42% )
 9055572140 stalled-cycles-frontend   #   29.40% frontend cycles
 idle ( +-  1.81% )
47982328290 instructions  #1.56  insns per cycle
  #0.19  stalled cycles per
 insn  ( +-  0.02% )
 7087070642 branches  #  680.409 M/sec
 ( +-  0.02% )
  341325584 branch-misses #4.82% of all branches
 ( +-  0.05% )

   10.489354952 seconds time elapsed
 ( +-  0.41% )

 RLE compression writing
 ===

 Performance counter stats for 'r.mapcalc
 expression=test_rast_rle=double(test_rast_z_base)' (10 runs):

 ::


   10367.674362 task-clock (msec) #0.999 CPUs utilized
 ( +-  0.53% )
   1642 context-switches  #0.158 K/sec
 ( +- 18.72% )
 22 cpu-migrations#0.002 K/sec
 ( +-  5.32% )
 325702 page-faults   #0.031 M/sec
 ( +-  0.00% )
3090391 cycles#2.958 GHz
 ( +-  0.38% )
 8921313281 stalled-cycles-frontend   #   29.09% frontend cycles
 idle ( +-  1.80% )
47975696799 instructions  #1.56  insns per cycle
  #0.19  stalled cycles per
 insn  ( +-  0.02% )
 7085878436 branches  #  683.459 M/sec
 ( +-  0.02% )
  340649966 branch-misses #4.81% of all branches
 ( +-  0.04% )

   10.382500561 seconds time elapsed
 ( +-  0.53% )

 LZ4 compression writing
 ===

 Performance counter stats for 'r.mapcalc
 expression=test_rast_lz4=double(test_rast_z_base)' (10 runs):

 ::


2490.815692 task-clock (msec) #0.999 CPUs utilized
 ( +-  0.23% )
321 context-switches  #0.129 K/sec
 ( +- 13.63% )
 20 cpu-migrations#0.008 K/sec
 ( +-  5.02% )
684 page-faults   #0.274 K/sec
 ( +-  0.12% )
 7259170408 cycles#2.914 GHz
 ( +-  0.12% )
 2305705372 stalled-cycles-frontend   #   31.76% frontend cycles
 idle ( +-  0.20% )
13796117271 instructions  #1.90  insns per cycle
  #0.17  stalled cycles per
 insn  ( +-  0.06% )
 2790495244 branches  # 1120.314 M/sec
 ( +-  0.05% )
   33371582 branch-misses #1.20% of all branches
 ( +-  0.41% )

2.492994675 seconds time elapsed
 ( +-  0.23% )

 LZ4HC compression writing
 =

 Performance counter stats for 'r.mapcalc
 expression=test_rast_lz4hc=double(test_rast_z_base)' (10 runs):

 ::


6867.635439 task-clock (msec) #0.999 CPUs utilized
 ( +-  0.25% )
648 context-switches  #0.094 K/sec
 ( +-  0.29% )
 21 cpu-migrations#0.003 K/sec
 ( +-  5.28% )
745 page-faults   #0.108 K/sec
 ( +-  0.18% )
20199681252 cycles#2.941 GHz
 ( +-  0.28% )
 6449729534 stalled-cycles-frontend   #   31.93% frontend cycles
 idle ( +-  0.62% )
31860120047 instructions  #1.58  insns per cycle
  #0.20  stalled cycles per
 insn  ( +-  0.03% )
 5196919230 branches  #  756.726 M/sec
 ( +-  0.03% )

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by sprice):

 Completely random data is not a good test because (in theory) it won't
 compress. However, it looks like it demonstrated the expected results.

 If you do a diff with n_les_assemble.c you'll see that I added a few if
 statements to ensure that the indexing stays within row/col bounds. I was
 getting segfaults with corrupt data before I fixed the other bugs. I
 figure it should be able to handle any data without segfaulting, even if
 corrupt.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 Replying to [comment:6 wenzeslaus]:
 > Replying to [comment:5 sprice]:
 > > I just uploaded a new version of the code that shouldn't fail any
 additional unit tests, and I've added unit tests to r.compress that test
 reading/writing the different compression schemes.
 >
 > Nice. Runs for me as well.

 Bad news. Only the r.compress tests runs well for me. I messed up the
 environmental variables.

 Now (with `export GRASS_INT_LZ4=1`) I'm getting fails from various tests.

 Wrong values in the result (for example):

 {{{
 ./raster/r.series.interp/testsuite/interp_test.py
 ==
 FAIL: test_infile (__main__.InterpolationTest)
 --
 AssertionError: The actual maximum (269) is greater than the reference
 one (200) for raster map prec_2 (with minimum 200)
 }}}

 Possible segmentation faults:

 {{{
 ./raster/r.gwflow/testsuite/validation_7x7_grid.py
 =
 FAIL: test_transient (__main__.Validation7x7Grid)
 -
 AssertionError: Running  module ended with non-zero return code
 (-11)
 }}}

 {{{
 ./temporal/t.rast.univar/testsuite/test.t.rast.univar.sh
 ...
 grass.exceptions.CalledModuleError: Module run ['r.univar', '-ge',
 u'map=prec_1@__temporal_t_rast_univar_test.t. rast.univar'] ended with
 error
 Process ended with non-zero return code -11.
 }}}

 With the changes applied but with default settings (`unset
 GRASS_INT_LZ4`), I'm getting the same results as at the
 
[http://fatra.cnr.ncsu.edu/grassgistests/reports_for_date-2015-10-08-07-00/report_for_nc_basic_spm_grass7_nc/testfiles.html
 test server]. r.compress test runs well in both cases.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 Replying to [comment:5 sprice]:
 > I just uploaded a new version of the code that shouldn't fail any
 additional unit tests, and I've added unit tests to r.compress that test
 reading/writing the different compression schemes.

 Nice. Runs for me as well. Test is well written, but to be sure, please
 improve setting of the environmental variables. When you do
 `os.environ[env_var] = '1'`, `env_var` stays in the environment, so next
 time it might be picked up (to be honest, right now I don't understand why
 it is not picked up). You can run a module in an isolated environment by
 something like this:

 {{{
 env = os.environ.copy()
 env[env_var] = '1'
 ...Module(..., env_=env)
 }}}

 I haven't tried that but in theory it should work with all `*Module`
 functions as well as classes (all is based on
 
[https://grass.osgeo.org/grass70/manuals/libpython/pygrass.modules.interface.html
 #module-pygrass.modules.interface.module PyGRASS Module]).

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by sprice):

 * Attachment "lz4_zstd3.tgz" added.


--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by sprice):

 I just uploaded a new version of the code that shouldn't fail any
 additional unit tests, and I've added unit tests to r.compress that test
 reading/writing the different compression schemes.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by neteler):

 Replying to [comment:1 sprice]:
 > I've gone ahead and added code to also compress FCELL/DCELL.

 Great!

 A question: is the NULL file also compressed (see the related ticket
 #2349) with the new compression algorithm? This would be important - I
 could not yet check your patches myself.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by wenzeslaus):

 I have
 [https://grass.osgeo.org/grass71/manuals/libpython/gunittest_running_tests.html
 #running-tests-report executed all tests] with `lz4_zstd2.tgz​` and I got
 22 (17%) more failing tests, unfortunately. One example is `r.viewshed`
 test where the results are just completely wrong:

 {{{
 ...
 mismatch values (key, reference, actual):
 [('max', 43.15356, -3.4405722046),
  ('min', -24.98421, -3.4405722046),
  ('null_cells', 1963758, 0)]
 ...
 }}}

 I haven't investigated further. Can anybody confirm or disproof?

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by sprice):

 The null file is compressed when GRASS_COMPRESS_NULLS is set.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by sprice):

 * Attachment "lz4_zstd2.tgz" added.


--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---

Comment (by sprice):

 I've gone ahead and added code to also compress FCELL/DCELL. Same tests at
 before. Late 2013 Mac Pro, v10.10.5, 64 GB RAM, 3.5 GHz 6-Core. The
 `r.mapcalc` commands are intended to test real-world write performance.
 The `ls -l` command shows file sizes. `r.univar` is intended to show read
 performance.

 tl;dr: Write speed with LZ4 & ZSTD continues to be double the current
 method. Read speed also significantly faster. LZ4 shows the best
 performance if you have the disk space. ZSTD is very good all around, and
 doesn't fill the hard drive as fast.

 {{{
 > time r.mapcalc expression="out_fp_orig=float(out_test_lz4hc)"
 --overwrite
  100%

 real4m57.518s
 user4m48.823s
 sys 0m7.118s
 > time r.mapcalc expression="out_fp_orig=float(out_test_lz4hc)"
 --overwrite
  100%

 real4m57.555s
 user4m49.946s
 sys 0m6.382s
 > export GRASS_INT_LZ4=1
 > time r.mapcalc expression="out_fp_lz4=float(out_test_lz4hc)" --overwrite
  100%

 real1m51.136s
 user1m44.766s
 sys 0m5.879s
 > time r.mapcalc expression="out_fp_lz4=float(out_test_lz4hc)" --overwrite
  100%

 real1m51.898s
 user1m44.991s
 sys 0m5.928s
 > unset GRASS_INT_LZ4
 > export GRASS_INT_LZ4HC=1
 > time r.mapcalc expression="out_fp_lz4hc=float(out_test_lz4hc)"
 --overwrite
  100%

 real4m22.887s
 user4m15.556s
 sys 0m6.206s
 > time r.mapcalc expression="out_fp_lz4hc=float(out_test_lz4hc)"
 --overwrite
  100%

 real4m22.143s
 user4m15.164s
 sys 0m6.075s
 > unset GRASS_INT_LZ4HC
 > export GRASS_INT_ZSTD=1
 > time r.mapcalc expression="out_fp_zstd=float(out_test_lz4hc)"
 --overwrite
  100%

 real1m49.065s
 user1m43.401s
 sys 0m5.183s
 > time r.mapcalc expression="out_fp_zstd=float(out_test_lz4hc)"
 --overwrite
  100%

 real1m48.771s
 user1m43.238s
 sys 0m5.093s
 > ls -l vrt_test/PERMANENT/fcell/
 total 46206984
 -rw-r--r--  1 sprice  staff  7268345798 Sep 27 19:57 out_fp_lz4
 -rw-r--r--  1 sprice  staff  5979129773 Sep 27 20:06 out_fp_lz4hc
 -rw-r--r--  1 sprice  staff  4969771486 Sep 27 19:53 out_fp_orig
 -rw-r--r--  1 sprice  staff  5440720320 Sep 27 20:10 out_fp_zstd
 > unset GRASS_INT_ZSTD
 > r.compress -p out_test_zlib
  is compressed (level 2: DEFLATE). Data type: 
 > r.compress -p out_test_lz4
  is compressed (level 3: LZ4). Data type: 
 > r.compress -p out_test_lz4hc
  is compressed (level 4: LZ4HC). Data type: 
 > r.compress -p out_test_zstd
  is compressed (level 5: ZSTD). Data type: 
 > r.compress -p out_fp_zstd
  is compressed (level 5: ZSTD). Data type: 
 > r.compress -p out_fp_orig
  is compressed (level 2: DEFLATE). Data type: 
 > r.compress -p out_fp_lz4
  is compressed (level 3: LZ4). Data type: 
 > r.compress -p out_fp_lz4hc
  is compressed (level 4: LZ4HC). Data type: 
 > time r.univar out_fp_orig
  100%
 total null and non-null cells: 3526771952
 total null cells: 1502448926

 Of the non-null cells:
 --
 n: 2024323026
 minimum: 807
 maximum: 32767
 range: 31960
 mean: 9385.79
 mean of absolute values: 9385.79
 standard deviation: 6620.52
 variance: 4.38312e+07
 variation coefficient: 70.5377 %
 sum: 18999862195879

 real1m49.227s
 user1m46.152s
 sys 0m2.843s
 > time r.univar out_fp_lz4
  100%
 total null and non-null cells: 3526771952
 total null cells: 1502448926

 Of the non-null cells:
 --
 n: 2024323026
 minimum: 807
 maximum: 32767
 range: 31960
 mean: 9385.79
 mean of absolute values: 9385.79
 standard deviation: 6620.52
 variance: 4.38312e+07
 variation coefficient: 70.5377 %
 sum: 18999862195879

 real1m8.749s
 user1m4.596s
 sys 0m3.564s
 > time r.univar out_fp_lz4hc
  100%
 total null and non-null cells: 3526771952
 total null cells: 1502448926

 Of the non-null cells:
 --
 n: 2024323026
 minimum: 807
 maximum: 32767
 range: 31960
 mean: 9385.79
 mean of absolute values: 9385.79
 standard deviation: 6620.52
 variance: 4.38312e+07
 variation coefficient: 70.5377 %
 sum: 18999862195879

 real1m10.307s
 user1m6.546s
 sys 0m3.352s
 > time r.univar out_fp_zstd
  100%
 total null and non-null cells: 3526771952
 total null cells: 1502448926

 Of the non-null cells:
 --
 n: 2024323026
 minimum: 807
 maximum: 32767
 range: 31960
 mean: 9385.79
 mean of absolute values: 9385.79
 standard deviation: 6620.52
 variance: 4.38312e+07
 variation coefficient: 70.5377 %
 sum: 18999862195879

 real1m31.310s
 user1m27.916s
 sys 

[GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
---+-
 Reporter:  sprice |  Owner:  grass-dev@…
 Type:  enhancement| Status:  new
 Priority:  normal |  Milestone:  7.1.0
Component:  Raster |Version:  svn-trunk
 Keywords:  ZLIB LZ4 ZSTD  |CPU:  OSX/Intel
 Platform:  MacOSX |
---+-
 I've added the ability for reading/writing raster rows in compression
 formats LZ4, LZ4HC, & ZSTD (in addition to the existing RLE & ZLIB.) These
 new algorithms are extremely fast (an order of magnitude faster than ZLIB)
 and are a better fit for modern, fast, hard drives & SSDs.

 I've attached a .tgz file with the necessary added & changed files. The
 new algorithms can be enabled/disabled with environment vars the same way
 as ZLIB, as described in the r.compress documentation:
 https://grass.osgeo.org/grass70/manuals/r.compress.html

 Algorithm summary:

 LZ4 produces a slightly worse compression ratio than ZLIB, but it
 compresses about an order of magnitude faster than ZLIB. It decompresses
 even faster.

 LZ4-HC is supposed to produce a compression ratio similar to ZLIB, and at
 about the same speed as ZLIB. It decompresses as fast as the regular LZ4
 (>2 GB/s). Unfortunately, the improved compression ratio doesn't show in
 my tests, probably due to the fact we're compressing each row
 individually. This may change against floating point data if someone wants
 to test it.

 ZSTD is a new algorithm by the author of LZ4 that is intended to replace
 ZLIB. It compresses and decompresses extremely quickly, while maintaining
 a similar compression ratio as ZLIB. Unfortunately, it's still in beta.

 They are all under the BSD license. Links below show more info &
 performance numbers.

 https://github.com/Cyan4973/zstd

 https://github.com/Cyan4973/lz4

 http://www.lz4.org


 I recommend incorporating the attached changes into GRASS, and leaving it
 as optional for users. At some point in the future, after testing, GRASS
 should move to using ZSTD as default. Power users who want the best
 performance '''now''' (and have the disk space) can use LZ4 immediately.

 It would be trivial to further alter get_row.c & put_row.c to use LZ4 for
 floating point compression. (And I would recommend someone do that.)

 Note: I've decided to use LZ4_decompress_fast() instead of
 LZ4_decompress_safe(). In my test, it was noticeably faster. According to
 the documentation, it leaves LZ4 open to a malicious attack. If this is a
 serious concern in the GRASS GIS internal data structures, change the
 commenting in get_row.c to use the safer code.

 On my computer, I've better than halved (!) the runtime of a r.mapcalc
 identity operation when using LZ4. Below are my tests while working with a
 RapidEye scene.

 {{{
 > time r.mapcalc expression="out_test_zlib=out_test_lz4hc" --overwrite
  100%

 real3m33.503s
 user3m25.451s
 sys 0m6.750s
 > time r.mapcalc expression="out_test_zlib=out_test_lz4hc" --overwrite
  100%

 real3m34.398s
 user3m26.684s
 sys 0m6.138s
 > export GRASS_INT_LZ4=1
 > time r.mapcalc expression="out_test_lz4=out_test_lz4hc" --overwrite
  100%

 real1m31.222s
 user1m25.379s
 sys 0m5.035s
 > time r.mapcalc expression="out_test_lz4=out_test_lz4hc" --overwrite
  100%

 real1m29.792s
 user1m24.029s
 sys 0m4.858s
 > unset GRASS_INT_LZ4
 > export GRASS_INT_LZ4HC=1
 > time r.mapcalc expression="out_test_lz4hc2=out_test_lz4hc" --overwrite
  100%

 real3m5.332s
 user2m58.610s
 sys 0m5.603s
 > time r.mapcalc expression="out_test_lz4hc2=out_test_lz4hc" --overwrite
  100%

 real3m3.710s
 user2m56.606s
 sys 0m5.858s
 > unset GRASS_INT_LZ4HC
 > export GRASS_INT_ZSTD=1
 > time r.mapcalc expression="out_test_zstd=out_test_lz4hc" --overwrite
  100%

 real1m38.322s
 user1m32.654s
 sys 0m4.897s
 > time r.mapcalc expression="out_test_zstd=out_test_lz4hc" --overwrite
  100%

 real1m42.370s
 user1m35.487s
 sys 0m5.282s
 > unset GRASS_INT_ZSTD
 > ls -l vrt_test/PERMANENT/cell/out_test_*
 -rw-r--r--  1 sprice  staff  4080217012 Sep 26 14:01
 vrt_test/PERMANENT/cell/out_test_lz4
 -rw-r--r--  1 sprice  staff  4069728048 Sep 26 13:34
 vrt_test/PERMANENT/cell/out_test_lz4hc
 -rw-r--r--  1 sprice  staff  4069728048 Sep 26 14:08
 vrt_test/PERMANENT/cell/out_test_lz4hc2
 -rw-r--r--  1 sprice  staff  3737100577 Sep 26 13:57
 vrt_test/PERMANENT/cell/out_test_zlib
 -rw-r--r--  1 sprice  staff  3811356101 Sep 26 14:12
 vrt_test/PERMANENT/cell/out_test_zstd
 > time r.univar out_test_zlib
  100%
 total null and non-null cells: 3526771952
 total null cells: 1502448926

 Of the non-null cells:
 --
 n: 2024323026
 minimum: 807
 maximum: 32767
 range: 31960
 mean: 9385.79
 mean of absolute values: 9385.79
 standard deviation: 6620.52
 variance: 4.38312e+07
 variation 

Re: [GRASS-dev] [GRASS GIS] #2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc speed

[GRASS GIS]

#2750: LZ4 when writing raster rows; better than double I/O bound r.mapcalc 
speed
--+---
  Reporter:  sprice   |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  normal   |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  ZLIB LZ4 ZSTD
   CPU:  OSX/Intel|   Platform:  MacOSX
--+---
Changes (by sprice):

 * Attachment "lz4_zstd.tgz" added.


--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev