Re: [gdal-dev] Zarr BLOCKSIZE in gdalmdimtranslate with variables of varying dimension

2025-03-15 Thread Even Rouault via gdal-dev

Simon,

see the ARRAY:IF(DIM=2):BLOCKSIZE=256,256 example of 
GDALGroup::CopyFrom() mentioned at 
https://gdal.org/en/stable/doxygen/classGDALGroup.html#a28324579902842d7dbd1f93e9698b1ec


Even

Le 12/03/2025 à 09:23, Simon Lyngby Kokkendorff via gdal-dev a écrit :

Hi list,

   gdalmdimtranslate is a very handy tool e.g. to translate netCDF
files to Zarr as it can handle huge files that xarray just chokes on.
I have a question regarding handling output blocksizes for the Zarr
format for files with variables of varying dimensions though. For
example I need to translate a netCDF file with time, z, lat, lon
dimensions and variables which are either 4D (time, z, lat, lon), 3D
(time, lat, lon) and 2D (lat, lon). I would like to remap chunksizes /
blocksizes to better suit my use case. However, it seems that I can
only specify blocksizes as e.g. -co ARRAY:BLOCKSIZE 2, 2, 128, 128 -
which will throw an error and skip the variables that don't  have 4
dimensions. Also, the output doesn't contain the dimension variables
in this case. Likewise, for 4D variables if I only specify 3
blocksizes.
   Have tried this with gdal versions 3.8.4 and 3.4.1 with similar results.
So the question is if there is any way to specify blocksizes e.g.
by dimension name? Or if the approach should rather be to just create
3 output Zarr folders - one for each variable dimensionality (2D, 3D
and 4D)? I can of course manually "merge" these folders afterwards.

Best regards,
Simon
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


Re: [gdal-dev] Zarr BLOCKSIZE in gdalmdimtranslate with variables of varying dimension

2025-03-12 Thread Simon Lyngby Kokkendorff via gdal-dev
Thanks Even,

  That does the trick :)

Cheers,
Simon

On Wed, Mar 12, 2025 at 9:54 AM Even Rouault  wrote:
>
> Simon,
>
> see the ARRAY:IF(DIM=2):BLOCKSIZE=256,256 example of
> GDALGroup::CopyFrom() mentioned at
> https://gdal.org/en/stable/doxygen/classGDALGroup.html#a28324579902842d7dbd1f93e9698b1ec
>
> Even
>
> Le 12/03/2025 à 09:23, Simon Lyngby Kokkendorff via gdal-dev a écrit :
> > Hi list,
> >
> >gdalmdimtranslate is a very handy tool e.g. to translate netCDF
> > files to Zarr as it can handle huge files that xarray just chokes on.
> > I have a question regarding handling output blocksizes for the Zarr
> > format for files with variables of varying dimensions though. For
> > example I need to translate a netCDF file with time, z, lat, lon
> > dimensions and variables which are either 4D (time, z, lat, lon), 3D
> > (time, lat, lon) and 2D (lat, lon). I would like to remap chunksizes /
> > blocksizes to better suit my use case. However, it seems that I can
> > only specify blocksizes as e.g. -co ARRAY:BLOCKSIZE 2, 2, 128, 128 -
> > which will throw an error and skip the variables that don't  have 4
> > dimensions. Also, the output doesn't contain the dimension variables
> > in this case. Likewise, for 4D variables if I only specify 3
> > blocksizes.
> >Have tried this with gdal versions 3.8.4 and 3.4.1 with similar results.
> > So the question is if there is any way to specify blocksizes e.g.
> > by dimension name? Or if the approach should rather be to just create
> > 3 output Zarr folders - one for each variable dimensionality (2D, 3D
> > and 4D)? I can of course manually "merge" these folders afterwards.
> >
> > Best regards,
> > Simon
> > ___
> > gdal-dev mailing list
> > gdal-dev@lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev