Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  closed
  Priority:  critical |  Milestone:  7.2.0
 Component:  Raster   |Version:  svn-trunk
Resolution:  fixed|   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by neteler):

 r.compress manual updated in r69403, r69404.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  closed
  Priority:  critical |  Milestone:  7.2.0
 Component:  Raster   |Version:  svn-trunk
Resolution:  fixed|   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by neteler):

 For the record - concerning NULL file compression:

 At time `cell_misc/null` is uncompressed by default.

 Using the environment variable `GRASS_COMPRESS_NULLS=1` NULL compression
 is activated for the session for newly created raster maps; and

 {{{
 export GRASS_COMPRESS_NULLS=1
 r.null -z raster_map
 }}}

 generates a new compressed file `cell_misc/nullcmpr` and removes the old
 uncompressed `cell_misc/null` file for existing maps.

 Note: At least GRASS GIS 7.2 is needed to read a raster map with
 compressed null file.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  closed
  Priority:  critical |  Milestone:  7.2.0
 Component:  Raster   |Version:  svn-trunk
Resolution:  fixed|   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by neteler):

 As per

  * https://lists.osgeo.org/pipermail/grass-dev/2016-September/082092.html
  * https://lists.osgeo.org/pipermail/grass-dev/2016-September/082101.html

 ... the default ZLIB compression level was invalid, causing ZLIB to not
 compress at all. Fixed in r69387 + r69388

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  closed
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:  fixed|   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---
Changes (by neteler):

 * milestone:   => 7.1.0


--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  closed
  Priority:  critical |  Milestone:  7.2.0
 Component:  Raster   |Version:  svn-trunk
Resolution:  fixed|   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---
Changes (by neteler):

 * status:  new => closed
 * resolution:   => fixed
 * milestone:  7.0.4 => 7.2.0


Comment:

 Replying to [comment:46 martinl]:
 > ASAIU the backport is not planned, right? Can we close this ticket?

 Closing. Thanks for the great contributions and to all testers.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.0.4
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by martinl):

 ASAIU the backport is not planned, right? Can we close this ticket?

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.0.4
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---
Changes (by martinl):

 * milestone:  7.0.3 => 7.0.4


--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.0.4
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by martinl):

 Should be backported at the beginning of release cycle.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.0.3
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by neteler):

 Replying to [comment:38 glynn]:
 > Replying to [comment:34 glynn]:
 > > I added some support for the improved r.null/r.compress functionality
 to lib/raster
 >
 > With r65489, r65490, and r65491, r.null and r.support should work with
 compressed nulls. r.null now has a -z flag to re-create the null file
 using the current compression setting (i.e. compress/decompress).

 For the record, here all changesets related to this ticket which I could
 identify (for a future backport, needed also for a #2750 backport):

 r65272, r65273, r65322, r65323, r65489, r65490, r65491, r65775

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.0.3
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---
Changes (by neteler):

 * milestone:  7.1.0 => 7.0.3


Comment:

 See #2750 for a new compression scheme. Ideally this ticket handled
 together with #2750.

 The NULL compression should definitely go into relbranch70.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by wenzeslaus):

 Replying to [comment:38 glynn]:
 > Replying to [comment:34 glynn]:
 > > I added some support for the improved r.null/r.compress functionality
 to lib/raster
 >
 > With r65489, r65490, and r65491, r.null and r.support should work with
 compressed nulls. r.null now has a -z flag to re-create the null file
 using the current compression setting (i.e. compress/decompress).

 This ticket seems like completed to me regarding the code. Is that right?
 Were there some other tests besides comment:40?

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by wenzeslaus):

 I've executed tests with `GRASS_COMPRESS_NULLS=1` and they worked well.
 Here is what I was running:

 {{{
 export GRASS_COMPRESS_NULLS=1
 grass71 /grassdata/nc_basic/user1/ --exec python -m grass.gunittest.main
 --location nc_basic --location-type nc
 export GRASS_INT_ZLIB=0
 grass71 /grassdata/nc_basic/user1/ --exec python -m grass.gunittest.main
 ...
 }}}

 However, I think that our test coverage is quite low for this especially
 for the usage of NULLs.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by sprice):

 Probably related to #2750, where I've implemented LZ4, LZ4HC, and ZSTD
 support in GRASS for up to 3x speedup in I/O bound modules.

 GRASS_COMPRESS_NULLS is slightly slower than not when using the original
 ZLIB, but faster when using LZ4. LZ4 does a better job compressing null
 files than ZLIB, in addition to the speedup.

--
Ticket URL: 
GRASS GIS 

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by glynn):

 Replying to [comment:34 glynn]:
  I added some support for the improved r.null/r.compress functionality to
 lib/raster

 With r65489, r65490, and r65491, r.null and r.support should work with
 compressed nulls. r.null now has a -z flag to re-create the null file
 using the current compression setting (i.e. compress/decompress).

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:38
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by annakrat):

 Please see probably related #2689.

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:37
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by glynn):

 Replying to [comment:33 neteler]:
  I am just not sure how to test it (I would need the improved
 r.null/r.compress functionality to not ruin all metadata). Or do you
 prefer to have tests with newly generated data first? I can do that as
 well.

 I added some support for the improved r.null/r.compress functionality to
 lib/raster, but it was amongst the things broken by r65348 (the tempfile
 changes), so this needs to wait until that gets fixed or reverted.

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:34
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by martinl):

 Replying to [comment:34 glynn]:
  Replying to [comment:33 neteler]:

  I added some support for the improved r.null/r.compress functionality to
 lib/raster, but it was amongst the things broken by r65348 (the tempfile
 changes), so this needs to wait until that gets fixed or reverted.

 by fix you mean to ignore GRASS_TMPDIR_MAPSET when working with raster
 data?

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:35
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by glynn):

 Replying to [comment:35 martinl]:

  by fix you mean to ignore GRASS_TMPDIR_MAPSET when working with raster
 data?

 Yes. I don't particularly want to do anything with lib/raster while that's
 outstanding (the tempfile changes broke Rast!__close_null(); while a
 workaround would be trivial, that's symptomatic of the sort of thing that
 happens when two people are working on the same code concurrently).

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:36
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by neteler):

 Replying to [comment:32 glynn]:
  For now, it should exist only in trunk.

 Yes, this is the case, no related backports done.

 I am just not sure how to test it (I would need the improved
 r.null/r.compress functionality to not ruin all metadata). Or do you
 prefer to have tests with newly generated data first? I can do that as
 well.

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:33
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by glynn):

 Replying to [comment:31 neteler]:

  Is there any snipped which should be backported?

 I'd suggest not including this feature in any release version until it has
 received more thorough testing. If it has already been backported in some
 form, that should be reverted. For now, it should exist only in trunk.

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:32
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by neteler):

 Replying to [comment:30 glynn]:
  Replying to [comment:29 glynn]:
 
   In the meantime, don't use compressed nulls, because they aren't
 actually working yet. At the point that the row pointers are written out,
 the null file has already been closed.
 
  This should be fixed in r65323, along with the issue that a compressed
 null file was being ignored (resulting in the zero-is-null compatibility
 behaviour).

 Is there any snipped which should be backported?

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:31
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by glynn):

 Replying to [comment:28 neteler]:

  I hope you are willing to implement this, looking forward to it. Thanks.

 I'm working on it. In the meantime, don't use compressed nulls, because
 they aren't actually working yet. At the point that the row pointers are
 written out, the null file has already been closed.

--
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:29
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by glynn):

 Replying to [comment:29 glynn]:

  In the meantime, don't use compressed nulls, because they aren't
 actually working yet. At the point that the row pointers are written out,
 the null file has already been closed.

 This should be fixed in r65323, along with the issue that a compressed
 null file was being ignored (resulting in the zero-is-null compatibility
 behaviour).

--
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:30
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by neteler):

 Replying to [comment:27 glynn]:
  Ultimately we need something which will just update the null file,
 probably an enhancement to r.null.

 Yes - this approach would save quite some CPU time.

  Note that, at present, r.null -c or r.support -n will generate a
 broken map if GRASS_COMPRESS_NULLS is set, as they will write compressed
 data to the null2 file but won't write the row pointers.
 
  Actually, those modules are already on shaky ground because they're
 passing an invalid map index as the fd parameter to
 Rast!__write_null_bits().

 I hope you are willing to implement this, looking forward to it. Thanks.

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:28
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by glynn):

 Replying to [comment:26 neteler]:

  Would r.compress -u followed by r.compress do the job? From a user's
 point this might be the best solution.

 It would probably work. A better (but still simple) option would be to add
 a flag to r.compress so that it will process an already-compressed map
 (rather than assuming it doesn't need to do anything).

 Ultimately we need something which will just update the null file,
 probably an enhancement to r.null.

 Note that, at present, r.null -c or r.support -n will generate a
 broken map if GRASS_COMPRESS_NULLS is set, as they will write compressed
 data to the null2 file but won't write the row pointers.

 Actually, those modules are already on shaky ground because they're
 passing an invalid map index as the fd parameter to
 Rast!__write_null_bits().

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:27
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by neteler):

 Thanks, I can do more testing now.

 Which is the best way to achieve null2 files after setting the new env
 var?
 Using r.mapcalc I would ruin all metadata of my raster maps.

 Would r.compress -u followed by r.compress do the job? From a user's point
 this might be the best solution.

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:26
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by glynn):

 Replying to [comment:23 neteler]:
  After cleaning up well on that machine, it is now finally working (sorry
 for the delay).

 Committed in r65272, with one minor change: before writing out the null
 file, both cell_misc/null and cell_misc/null2 are deleted, not just the
 file which is being replaced.

 This avoids the situation where overwriting a map with null compression
 disabled would leave any compressed null file in place, resulting in the
 map having both a new uncompressed null file and an old compressed null
 file, with the new file being ignored (a compressed null file takes
 precedence).

 Actually, the precedence should probably be changed. If someone uses an
 older version of GRASS to overwrite a map which already has a compressed
 null file, newer versions of GRASS will use the (stale) compressed null
 file. It would be more robust to use the absence of an uncompressed null
 file to dictate the use of the compressed file, rather than just the
 presence of a compressed file.

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:24
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by glynn):

 Replying to [comment:24 glynn]:

  Actually, the precedence should probably be changed. If someone uses an
 older version of GRASS to overwrite a map which already has a compressed
 null file, newer versions of GRASS will use the (stale) compressed null
 file. It would be more robust to use the absence of an uncompressed null
 file to dictate the use of the compressed file, rather than just the
 presence of a compressed file.

 Done in r65273.

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:25
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by neteler):

 After cleaning up well on that machine, it is now finally working (sorry
 for the delay).

 '''Test 1: mostly no-data cells'''

 {{{
 r.univar eu_dem25_rand100k
 total null and non-null cells: 480
 total null cells: 470
 ...

 r.mapcalc new_int = round(eu_dem25_rand100k)

 ls -la grassdata/eu_laea/PERMANENT/cell_misc/new_int:
 total 5.7G
 -rw-r--r-- 1 root root 5.6G May 14 19:03 null
 -rw-r--r-- 1 root root  32M May 16 14:42 null2
 -rw-r--r-- 1 root root9 May 16 14:42 range
 }}}

 ... that is less than 1% of the original null file size!


 '''Test 2: 54% no-data cells'''

 {{{
 r.univar MOD11A1.A2000101.LST_Day_1km.reconstruct
 total null and non-null cells: 415290645
 total null cells: 226949285

 r.info -g MOD11A1.A2000101.LST_Day_1km.reconstruct | grep datatype
 datatype=CELL

 r.mapcalc modis_new_int = MOD11A1.A2000101.LST_Day_1km.reconstruct


 # original:
 ls -la
 grassdata/eu_laea/PERMANENT/cell_misc/MOD11A1.A2000101.LST_Day_1km.reconstruct
 total 50M
 drwxr-xr-x 2 root root 4.0K Apr  8  2014 .
 drwxr-xr-x 6 root root 4.0K May 17 14:14 ..
 -rw-r--r-- 1 root root  50M Apr  8  2014 null
 -rw-r--r-- 1 root root   12 Apr  8  2014 range

 # new compressed null2:
 ls -la grassdata/eu_laea/PERMANENT/cell_misc/modis_new_int
 total 1.6M
 drwxr-xr-x 2 root root 4.0K May 17 14:14 .
 drwxr-xr-x 6 root root 4.0K May 17 14:14 ..
 -rw-r--r-- 1 root root 1.6M May 17 14:14 null2
 -rw-r--r-- 1 root root   12 May 17 14:14 range
 }}}

 ... that is 3% of the original null file size.

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:23
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by neteler):

 I tried again but don't see a difference in the NULL file size:

 {{{
 export GRASS_COMPRESS_NULLS=1
 g.region raster=eu_dem25_rand100k -p
 r.mapcalc new_int = round(eu_dem25_rand100k)
 echo $GRASS_COMPRESS_NULLS
 1
 }}}

 Verification:

 {{{
 root@grass ~ # l grassdata/eu_laea/PERMANENT/cell_misc/*
 grassdata/eu_laea/PERMANENT/cell_misc/eu_dem25_rand100k:
 total 5.6G
 -rw-r--r-- 1 root root   53 Aug 15  2014 f_format
 -rw-r--r-- 1 root root5 Aug 15  2014 f_quant
 -rw-r--r-- 1 root root   16 Aug 15  2014 f_range
 -rw-r--r-- 1 root root 5.6G Aug 15  2014 null

 grassdata/eu_laea/PERMANENT/cell_misc/new_int:
 total 5.6G
 -rw-r--r-- 1 root root 5.6G May 14 19:03 null
 -rw-r--r-- 1 root root9 May 14 19:03 range
 }}}

--
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:21
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by glynn):

 Replying to [comment:21 neteler]:
  I tried again but don't see a difference in the NULL file size:

 Check that the patch applied successfully, that compilation was
 successful, and that r.mapcalc is using the new libgrass_raster.so, not
 e.g. an installed version.

 The first two can be confirmed by running strings on the library and
 confirming that GRASS_COMPRESS_NULLS appears in the output. The last can
 be confirmed with e.g.
 {{{
 ldd $(which r.mapcalc)
 }}}

--
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:22
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by glynn):

 Replying to [comment:19 neteler]:

  Are my tests wrong or am I missing anything else?

 First, it shouldn't be necessary to recompile anything except lib/raster
 (but you should probably run make clean in that directory first, as the
 patch changes the R!__ and fileinfo structures which are used throughout
 the library).

 Second, you need to use
 {{{
 export GRASS_COMPRESS_NULLS=1
 }}}
 in order for compressed null files to be generated
 (cell_misc/name/null2).

 Reading should automatically use the compressed null file if it exists or
 the raw null file if it doesn't.

 If you don't set GRASS_COMPRESS_NULLS, everything should continue to work
 as before (including r.null and r.support). If it doesn't, that needs to
 be fixed now (i.e. before the changes committed; issues which only arise
 when setting GRASS_COMPRESS_NULLS can be dealt with later).

--
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:20
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by neteler):

 Thanks for the patch. I applied it to my local copy of relbr70. Testing
 with this map

 {{{
 GRASS 7.0.1svn (eu_laea):~/releasebranch_7_0  r.info -g eu_dem25_rand100k
 north=600
 south=100
 east=800
 west=200
 nsres=25
 ewres=25
 rows=20
 cols=24
 cells=480
 datatype=FCELL
 ncats=0
 }}}

 I created a new one using the 100k random points FCELL map sampled earlier
 from the EU 25m DEM:

 {{{
  r.mapcalc new = eu_dem25_rand100k
 }}}

 Using the patch, I see that a NULL file is no longer generated.
 Expectedly, since the other modules are not updated yet, I cannot read
 that map:

 {{{
 GRASS 7.0.1svn (eu_laea):~/releasebranch_7_0  r.univar new
 ERROR: Raster map new@PERMANENT: format field in header file invalid
 }}}

 Next test with INT conversion:


 {{{
 GRASS 7.0.1svn (eu_laea):~/releasebranch_7_0  r.mapcalc new_int =
 round(eu_dem25_rand100k)

 GRASS 7.0.1svn (eu_laea):~/releasebranch_7_0  r.univar new_int
 0%
 }}}

 ... the map is readable and comes with the NULL file but that's as before
 6GB big (so no compression occurred).

 Are my tests wrong or am I missing anything else?

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:19
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---
Changes (by glynn):

 * Attachment compressed_nulls.diff added.

 implement compressed nulls

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by neteler):

 Replying to [comment:16 glynn]:
  How large are the null files compared to the cell/fcell files?

  * With MODIS LST data, the null files are between 1.7 and 7.6 times
 larger than the cell files (we store the LST maps in deg C * 100 as
 integer to save disk space).
  * With 100k random points, the null file is 7.1 times larger than the
 fcell map
  * With the EU 25m DEM, the null file is way smaller that the derived
 aspect map (17% of the DEM fcell file)

   What about having a null2 file which is compressed and with index.
 If present, fine, otherwise use the uncompressed well known null file
 format?
 
  That's probably not a great deal of work, but as with any such change,
 we need to consider the migration strategy. If we just start creating
 compressed null files, mapsets will cease to be usable with older
 versions.

 Right but this could be covered with an addon/new script in G6 and earlier
 (as v.build does for vector data).

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:17
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
--+---
  Reporter:  neteler  |  Owner:  grass-dev@…
  Type:  enhancement  | Status:  new
  Priority:  critical |  Milestone:  7.1.0
 Component:  Raster   |Version:  svn-trunk
Resolution:   |   Keywords:  compression, r.compress, null
   CPU:  Unspecified  |   Platform:  All
--+---

Comment (by glynn):

 Replying to [comment:15 neteler]:
  back to this topic: Here on our system I found  1.7TB of NULL files in
 a single location, all
  uncompressed.

 How large are the null files compared to the cell/fcell files?

  What about having a null2 file which is compressed and with index. If
 present, fine, otherwise use the uncompressed well known null file format?

 That's probably not a great deal of work, but as with any such change, we
 need to consider the migration strategy. If we just start creating
 compressed null files, mapsets will cease to be usable with older
 versions.

--
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:16
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
---+
 Reporter:  neteler|   Owner:  grass-dev@…  

 Type:  enhancement|  Status:  new  

 Priority:  critical   |   Milestone:  7.1.0

Component:  Raster | Version:  svn-trunk

 Keywords:  compression, r.compress, null  |Platform:  All  

  Cpu:  Unspecified|  
---+
Changes (by neteler):

  * keywords:  compression, r.compress = compression, r.compress, null
  * milestone:  8.0.0 = 7.1.0


Comment:

 back to this topic: Here on our system I found  1.7TB of NULL files in a
 single location, all
 uncompressed.

 What about having a null2 file which is compressed and with index. If
 present, fine, otherwise use the uncompressed well known null file format?

 For backward compatibility, r.null could extended to convert from
 compressed null2 to
 uncompressed null (similar to v.build for the new spatial index in G7).

-- 
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:15
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
-+--
 Reporter:  neteler  |   Owner:  grass-dev@…  
 Type:  enhancement  |  Status:  new  
 Priority:  critical |   Milestone:  8.0.0
Component:  Raster   | Version:  svn-trunk
 Keywords:  compression, r.compress  |Platform:  All  
  Cpu:  Unspecified  |  
-+--
Changes (by neteler):

  * priority:  blocker = critical
  * version:  svn-releasebranch70 = svn-trunk
  * milestone:  7.0.0 = 8.0.0


Comment:

 Replying to [comment:13 glynn]:
 ...
  The main issue is likely to be the need to support both formats. We need
 to

 ... at this point raster format changes may go into GRASS GIS 8.

-- 
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:14
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
-+--
 Reporter:  neteler  |   Owner:  grass-dev@…  
 Type:  enhancement  |  Status:  new  
 Priority:  blocker  |   Milestone:  7.0.0
Component:  Raster   | Version:  svn-releasebranch70  
 Keywords:  compression, r.compress  |Platform:  All  
  Cpu:  Unspecified  |  
-+--

Comment(by glynn):

 Replying to [comment:12 neteler]:

  It seems that the NULL file is not compressed (file cell_misc/null).
 According to
 
  http://lists.osgeo.org/pipermail/grass-user/2010-January/054216.html
 
  it is one bit (null/non-null) for each cell.

  Question: Would it be possible to compress also the null files, even
 with just a
  weak compression?

 Being uncompressed, the null files don't contain an index. The offset to
 the beginning of a given row is obtained by multiplying the row number by
 the number of bytes per row (which is just the number of columns divided
 by 8, rounded upwards).

 The main issue is likely to be the need to support both formats. We need
 to

  * Be able to read and write the uncompressed format, for compatibility
 with existing versions of GRASS.
  * Be able to distinguish between compressed and uncompressed formats on
 read.
  * Provide some mechanism (i.e. yet another environment variable) to
 indicate which format to use on write.

-- 
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:13
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
-+--
 Reporter:  neteler  |   Owner:  grass-dev@…  
 Type:  enhancement  |  Status:  new  
 Priority:  blocker  |   Milestone:  7.0.0
Component:  Raster   | Version:  svn-releasebranch70  
 Keywords:  compression, r.compress  |Platform:  All  
  Cpu:  Unspecified  |  
-+--

Comment(by neteler):

 It seems that the NULL file is not compressed (file cell_misc/null).
 According to

 http://lists.osgeo.org/pipermail/grass-user/2010-January/054216.html

 it is one bit (null/non-null) for each cell. It looks like this:

 {{{
 [neteler@giscluster modis_lst_reconstructed_europe_daily]$ hexdump
 cell_misc/lst_2002_196_average/null | head
 000        
 *
 ad0    ffe0    
 ae0        
 *
 00015a0       e0ff 
 00015b0        
 *
 0002080   ffe0     
 0002090        
 ...
 }}}

 For our 12829 daily min/average/max MODIS LST maps covering Europe, the
 null files
 consume a lot of space:

 {{{
 [neteler@giscluster cell_misc]$ du -hs .
 621G.
 }}}

 Question: Would it be possible to compress also the null files, even with
 just a
 weak compression?

-- 
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:12
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
-+--
 Reporter:  neteler  |   Owner:  grass-dev@…  
 Type:  enhancement  |  Status:  new  
 Priority:  blocker  |   Milestone:  7.0.0
Component:  Raster   | Version:  svn-releasebranch70  
 Keywords:  compression, r.compress  |Platform:  All  
  Cpu:  Unspecified  |  
-+--
Changes (by neteler):

  * keywords:  compression = compression, r.compress


Comment:

 Replying to [comment:8 neteler]:
  In case of a backport to relbr7, this should be the needed changes:
 
  r61380 + r61420 + r61422 + r61500

 Backported to relbr7 in r61797.

 Remains open which default ZLIB level should be used, see comment:10

-- 
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:11
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
-+--
 Reporter:  neteler  |   Owner:  grass-dev@…  
 Type:  enhancement  |  Status:  new  
 Priority:  blocker  |   Milestone:  7.0.0
Component:  Raster   | Version:  svn-releasebranch70  
 Keywords:  compression  |Platform:  All  
  Cpu:  Unspecified  |  
-+--

Comment(by neteler):

 Replying to [comment:5 glynn]:
   * The compression level can be set via GRASS_ZLIB_LEVEL, whose value
 should be an integer between 0 and 9. If not set (or if the value cannot
 be parsed as an integer), zlib's default compression level will be used
 (lib/gis/flate.c:333, if a different default is preferred).

 In relbranch7 there is currently:

 Z_DEFAULT_COMPRESSION = 1 - gives the best compromise between speed and
 compression as per r61424.

 Perhaps zlib compression 1 should be adopted for trunk?

-- 
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:10
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
-+--
 Reporter:  neteler  |   Owner:  grass-dev@…  
 Type:  enhancement  |  Status:  new  
 Priority:  blocker  |   Milestone:  7.0.0
Component:  Raster   | Version:  svn-releasebranch70  
 Keywords:  compression  |Platform:  All  
  Cpu:  Unspecified  |  
-+--
Changes (by neteler):

  * priority:  critical = blocker


Comment:

 Any objections to backport?

-- 
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:9
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
-+--
 Reporter:  neteler  |   Owner:  grass-dev@…  
 Type:  enhancement  |  Status:  new  
 Priority:  critical |   Milestone:  7.0.0
Component:  Raster   | Version:  svn-releasebranch70  
 Keywords:  compression  |Platform:  All  
  Cpu:  Unspecified  |  
-+--
Changes (by neteler):

  * keywords:  = compression


Comment:

 In case of a backport to relbr7, this should be the needed changes:

 r61380 + r61420 + r61422 + r61500

-- 
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:8
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
-+--
 Reporter:  neteler  |   Owner:  grass-dev@…  
 Type:  enhancement  |  Status:  new  
 Priority:  critical |   Milestone:  7.0.0
Component:  Raster   | Version:  svn-releasebranch70  
 Keywords:   |Platform:  All  
  Cpu:  Unspecified  |  
-+--

Comment(by neteler):

 Putting here to not forget about: raster/r.compress/r.compress.html needs
 to be updated

-- 
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:6
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
-+--
 Reporter:  neteler  |   Owner:  grass-dev@…  
 Type:  enhancement  |  Status:  new  
 Priority:  critical |   Milestone:  7.0.0
Component:  Raster   | Version:  svn-releasebranch70  
 Keywords:   |Platform:  All  
  Cpu:  Unspecified  |  
-+--

Comment(by glynn):

 Replying to [comment:6 neteler]:
  raster/r.compress/r.compress.html needs to be updated

 Done in r61500.

-- 
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:7
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
-+--
 Reporter:  neteler  |   Owner:  grass-dev@…  
 Type:  enhancement  |  Status:  new  
 Priority:  critical |   Milestone:  7.0.0
Component:  Raster   | Version:  svn-releasebranch70  
 Keywords:   |Platform:  All  
  Cpu:  Unspecified  |  
-+--

Comment(by glynn):

 Replying to [ticket:2349 neteler]:

  Proposal: make ZLIB, level 3 the standard compression.

 r61380 implements the following behaviour:

  * zlib compression is the default. Set GRASS_INT_ZLIB=0 to use RLE
 compression.

  * The compression level can be set via GRASS_ZLIB_LEVEL, whose value
 should be an integer between 0 and 9. If not set (or if the value cannot
 be parsed as an integer), zlib's default compression level will be used
 (lib/gis/flate.c:333, if a different default is preferred).

-- 
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:5
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
-+--
 Reporter:  neteler  |   Owner:  grass-dev@…  
 Type:  enhancement  |  Status:  new  
 Priority:  critical |   Milestone:  7.0.0
Component:  Raster   | Version:  svn-releasebranch70  
 Keywords:   |Platform:  All  
  Cpu:  Unspecified  |  
-+--

Comment(by glynn):

 Replying to [comment:3 neteler]:

   Aside: if there are still systems out there using the historical limit
 of 4096 bytes of memory for the combination of environment variables and
 arguments (argv), we might want to think about making GRASS less greedy
 with respect to environment variables.
 
  You mean the number and/or the length?

 Mainly the number.

-- 
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:4
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
-+--
 Reporter:  neteler  |   Owner:  grass-dev@…  
 Type:  enhancement  |  Status:  new  
 Priority:  critical |   Milestone:  7.0.0
Component:  Raster   | Version:  svn-releasebranch70  
 Keywords:   |Platform:  All  
  Cpu:  Unspecified  |  
-+--

Comment(by neteler):

 Replying to [comment:2 glynn]:
  Replying to [ticket:2349 neteler]:
   At time, integer maps (CELL) are still compressed with RLE
   This leads to a huge waste of disk space when it comes to large
   data.
  
   Proposal: make ZLIB, level 3 the standard compression.
 
  Is GRASS_INT_ZLIB support now old enough that it can be taken for
 granted?

 I hope yes. I am not aware of negative reports.

   At time we can enable the environment variable GRASS_INT_ZLIB
   but it will use the default ZLIB level 6 compression which
   is too CPU intensive. So a  (user) control over this is important.
 
  The current behaviour is that setting GRASS_INT_ZLIB to anything (even
 an empty string) will enable zlib compression at the hard-coded level.

 Exactly.

  One option is to parse the value as an integer and use the result as the
 compression level. However, it's possible that people are currently using
 e.g. GRASS_INT_ZLIB=1 to enable it with the existing default level.
 
  Another option is to add another environment variable for the level.

 Yes, a new GRASS_ZLIBLEVEL may be less invasive.

  Aside: if there are still systems out there using the historical limit
 of 4096 bytes of memory for the combination of environment variables and
 arguments (argv), we might want to think about making GRASS less greedy
 with respect to environment variables.

 You mean the number and/or the length?

-- 
Ticket URL: https://trac.osgeo.org/grass/ticket/2349#comment:3
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
-+--
 Reporter:  neteler  |   Owner:  grass-dev@…  
 Type:  enhancement  |  Status:  new  
 Priority:  critical |   Milestone:  7.0.0
Component:  Raster   | Version:  svn-releasebranch70  
 Keywords:   |Platform:  All  
  Cpu:  Unspecified  |  
-+--

Comment(by glynn):

 Replying to [ticket:2349 neteler]:
  At time, integer maps (CELL) are still compressed with RLE
  This leads to a huge waste of disk space when it comes to large
  data.
 
  Proposal: make ZLIB, level 3 the standard compression.

 Is GRASS_INT_ZLIB support now old enough that it can be taken for granted?

  At time we can enable the environment variable GRASS_INT_ZLIB
  but it will use the default ZLIB level 6 compression which
  is too CPU intensive. So a  (user) control over this is important.

 The current behaviour is that setting GRASS_INT_ZLIB to anything (even an
 empty string) will enable zlib compression at the hard-coded level. One
 option is to parse the value as an integer and use the result as the
 compression level. However, it's possible that people are currently using
 e.g. GRASS_INT_ZLIB=1 to enable it with the existing default level.

 Another option is to add another environment variable for the level.

 Aside: if there are still systems out there using the historical limit of
 4096 bytes of memory for the combination of environment variables and
 arguments (argv), we might want to think about making GRASS less greedy
 with respect to environment variables.

-- 
Ticket URL: http://trac.osgeo.org/grass/ticket/2349#comment:2
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev

[GRASS-dev] [GRASS GIS] #2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE

[GRASS GIS]

#2349: CELL raster format: make ZLIB level 3 standard compression instead of RLE
-+--
 Reporter:  neteler  |   Owner:  grass-dev@…  
 Type:  enhancement  |  Status:  new  
 Priority:  critical |   Milestone:  7.0.0
Component:  Raster   | Version:  svn-releasebranch70  
 Keywords:   |Platform:  All  
  Cpu:  Unspecified  |  
-+--
 At time, integer maps (CELL) are still compressed with RLE
 This leads to a huge waste of disk space when it comes to large
 data.

 Proposal: make ZLIB, level 3 the standard compression.

 At time we can enable the environment variable GRASS_INT_ZLIB
 but it will use the default ZLIB level 6 compression which
 is too CPU intensive. So a  (user) control over this is important.

 BTW: Manual of r.compress updated in r60814, needs to be backported.

-- 
Ticket URL: http://trac.osgeo.org/grass/ticket/2349
GRASS GIS http://grass.osgeo.org

___
grass-dev mailing list
grass-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/grass-dev