Hello community,
here is the log from the commit of package python-h5netcdf for openSUSE:Factory
checked in at 2020-02-18 10:42:42
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Comparing /work/SRC/openSUSE:Factory/python-h5netcdf (Old)
and /work/SRC/openSUSE:Factory/.python-h5netcdf.new.26092 (New)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "python-h5netcdf"
Tue Feb 18 10:42:42 2020 rev:2 rq:774995 version:0.8.0
Changes:
--------
--- /work/SRC/openSUSE:Factory/python-h5netcdf/python-h5netcdf.changes
2019-12-04 14:20:34.978448398 +0100
+++
/work/SRC/openSUSE:Factory/.python-h5netcdf.new.26092/python-h5netcdf.changes
2020-02-18 10:43:17.529297901 +0100
@@ -1,0 +2,8 @@
+Mon Feb 10 15:21:03 UTC 2020 - Todd R <[email protected]>
+
+- Update to 0.8.0
+ * Support for reading Datasets with missing dimension scales.
+ * Fixed a bug where Datatype objects were treated as Datasets.
+ * Fixed several issues with upstream deprecations.
+
+-------------------------------------------------------------------
Old:
----
h5netcdf-0.7.4.tar.gz
New:
----
h5netcdf-0.8.0.tar.gz
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Other differences:
------------------
++++++ python-h5netcdf.spec ++++++
--- /var/tmp/diff_new_pack.vEfs2K/_old 2020-02-18 10:43:19.585302096 +0100
+++ /var/tmp/diff_new_pack.vEfs2K/_new 2020-02-18 10:43:19.589302104 +0100
@@ -1,7 +1,7 @@
#
# spec file for package python-h5netcdf
#
-# Copyright (c) 2019 SUSE LINUX GmbH, Nuernberg, Germany.
+# Copyright (c) 2020 SUSE LLC.
#
# All modifications and additions to the file contributed by third parties
# remain the property of their copyright owners, unless otherwise agreed
@@ -17,8 +17,9 @@
%{?!python_module:%define python_module() python-%{**} python3-%{**}}
+%define skip_python2 1
Name: python-h5netcdf
-Version: 0.7.4
+Version: 0.8.0
Release: 0
Summary: A Python library to use netCDF4 files via h5py
License: BSD-3-Clause
++++++ h5netcdf-0.7.4.tar.gz -> h5netcdf-0.8.0.tar.gz ++++++
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/h5netcdf-0.7.4/PKG-INFO new/h5netcdf-0.8.0/PKG-INFO
--- old/h5netcdf-0.7.4/PKG-INFO 2019-06-01 23:42:49.000000000 +0200
+++ new/h5netcdf-0.8.0/PKG-INFO 2020-02-04 18:29:33.000000000 +0100
@@ -1,6 +1,6 @@
-Metadata-Version: 1.1
+Metadata-Version: 1.2
Name: h5netcdf
-Version: 0.7.4
+Version: 0.8.0
Summary: netCDF4 via h5py
Home-page: https://github.com/shoyer/h5netcdf
Author: Stephan Hoyer
@@ -176,8 +176,42 @@
we will raise ``h5netcdf.CompatibilityError``. Use
``invalid_netcdf=False`` to switch to the new behavior now.
+ Datasets with missing dimension scales
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ By default [*]_ h5netcdf raises a ``ValueError`` if variables with no
dimension
+ scale associated with one of their axes are accessed.
+ You can set ``phony_dims='sort'`` when opening a file to let h5netcdf
invent
+ phony dimensions according to `netCDF`_ behaviour.
+
+ .. code-block:: python
+
+ # mimic netCDF-behaviour for non-netcdf files
+ f = h5netcdf.File('mydata.h5', mode='r', phony_dims='sort')
+ ...
+
+ Note, that this iterates once over the whole group-hierarchy. This has
affects
+ on performance in case you rely on lazyness of group access.
+ You can set ``phony_dims='access'`` instead to defer phony dimension
creation
+ to group access time. The created phony dimension naming will differ
from
+ `netCDF`_ behaviour.
+
+ .. code-block:: python
+
+ f = h5netcdf.File('mydata.h5', mode='r', phony_dims='access')
+ ...
+
+ .. _netCDF:
https://www.unidata.ucar.edu/software/netcdf/docs/interoperability_hdf5.html
+ .. [*] Keyword default setting ``phony_dims=None`` for backwards
compatibility.
+
Change Log
----------
+ Version 0.8.0 (February 4, 2020):
+
+ - Support for reading Datasets with missing dimension scales.
+ By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_.
+ - Fixed a bug where ``Datatype`` objects were treated as ``Datasets``.
+ - Fixed several issues with upstream deprecations.
Version 0.7.4 (June 1, 2019):
@@ -279,11 +313,9 @@
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 2
-Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.4
-Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.6
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/h5netcdf-0.7.4/README.rst
new/h5netcdf-0.8.0/README.rst
--- old/h5netcdf-0.7.4/README.rst 2019-06-01 23:38:44.000000000 +0200
+++ new/h5netcdf-0.8.0/README.rst 2020-02-04 18:28:53.000000000 +0100
@@ -168,8 +168,42 @@
we will raise ``h5netcdf.CompatibilityError``. Use
``invalid_netcdf=False`` to switch to the new behavior now.
+Datasets with missing dimension scales
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+By default [*]_ h5netcdf raises a ``ValueError`` if variables with no dimension
+scale associated with one of their axes are accessed.
+You can set ``phony_dims='sort'`` when opening a file to let h5netcdf invent
+phony dimensions according to `netCDF`_ behaviour.
+
+.. code-block:: python
+
+ # mimic netCDF-behaviour for non-netcdf files
+ f = h5netcdf.File('mydata.h5', mode='r', phony_dims='sort')
+ ...
+
+Note, that this iterates once over the whole group-hierarchy. This has affects
+on performance in case you rely on lazyness of group access.
+You can set ``phony_dims='access'`` instead to defer phony dimension creation
+to group access time. The created phony dimension naming will differ from
+`netCDF`_ behaviour.
+
+.. code-block:: python
+
+ f = h5netcdf.File('mydata.h5', mode='r', phony_dims='access')
+ ...
+
+.. _netCDF:
https://www.unidata.ucar.edu/software/netcdf/docs/interoperability_hdf5.html
+.. [*] Keyword default setting ``phony_dims=None`` for backwards compatibility.
+
Change Log
----------
+Version 0.8.0 (February 4, 2020):
+
+- Support for reading Datasets with missing dimension scales.
+ By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_.
+- Fixed a bug where ``Datatype`` objects were treated as ``Datasets``.
+- Fixed several issues with upstream deprecations.
Version 0.7.4 (June 1, 2019):
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/h5netcdf-0.7.4/h5netcdf/attrs.py
new/h5netcdf-0.8.0/h5netcdf/attrs.py
--- old/h5netcdf-0.7.4/h5netcdf/attrs.py 2018-04-11 17:28:39.000000000
+0200
+++ new/h5netcdf-0.8.0/h5netcdf/attrs.py 2020-02-04 18:28:06.000000000
+0100
@@ -1,4 +1,4 @@
-from collections import MutableMapping
+from collections.abc import MutableMapping
import numpy as np
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/h5netcdf-0.7.4/h5netcdf/core.py
new/h5netcdf-0.8.0/h5netcdf/core.py
--- old/h5netcdf-0.7.4/h5netcdf/core.py 2019-06-01 23:37:29.000000000 +0200
+++ new/h5netcdf-0.8.0/h5netcdf/core.py 2020-02-04 18:28:47.000000000 +0100
@@ -1,6 +1,7 @@
# For details on how netCDF4 builds on HDF5:
#
http://www.unidata.ucar.edu/software/netcdf/docs/file_format_specifications.html#netcdf_4_spec
-from collections import Mapping
+from collections import defaultdict
+from collections.abc import Mapping
import os.path
import warnings
@@ -23,7 +24,7 @@
no_h5pyd = False
h5_group_types = (h5py.Group, h5pyd.Group)
-__version__ = '0.7.4'
+__version__ = '0.8.0'
_NC_PROPERTIES = (u'version=2,h5netcdf=%s,hdf5=%s,h5py=%s'
@@ -95,16 +96,25 @@
return (child_name,)
dims = []
+ phony_dims = defaultdict(int)
for axis, dim in enumerate(self._h5ds.dims):
- # TODO: read dimension labels even if there is no associated
- # scale? it's not netCDF4 spec, but it is unambiguous...
- # Also: the netCDF lib can read HDF5 datasets with unlabeled
- # dimensions.
- if len(dim) == 0:
- raise ValueError('variable %r has no dimension scale '
- 'associated with axis %s'
- % (self.name, axis))
- name = _name_from_dimension(dim)
+ # get current dimension
+ dimsize = self.shape[axis]
+ phony_dims[dimsize] += 1
+ if len(dim):
+ name = _name_from_dimension(dim)
+ else:
+ # if unlabeled dimensions are found
+ if self._root._phony_dims_mode is None:
+ raise ValueError('variable %r has no dimension scale '
+ 'associated with axis %s. \n'
+ 'Use phony_dims=%r for sorted naming or '
+ 'phony_dims=%r for per access naming.'
+ % (self.name, axis, 'sort', 'access'))
+ else:
+ # get dimension name
+ name = self._parent._phony_dims[(dimsize,
+ phony_dims[dimsize] - 1)]
dims.append(name)
return tuple(dims)
@@ -211,6 +221,16 @@
return NOT_A_VARIABLE in h5py_dataset.attrs.get('NAME', b'')
+def _unlabeled_dimension_mix(h5py_dataset):
+ dims = sum([len(j) for j in h5py_dataset.dims])
+ if dims:
+ if dims != h5py_dataset.ndim:
+ name = h5py_dataset.name.split('/')[-1]
+ raise ValueError('malformed variable %r has mixing of labeled and '
+ 'unlabeled dimensions.'.format(name))
+ return dims
+
+
class Group(Mapping):
_variable_cls = Variable
@@ -233,6 +253,12 @@
self._variables = _LazyObjectLookup(self, self._variable_cls)
self._groups = _LazyObjectLookup(self, self._group_cls)
+ # # initialize phony dimension counter
+ if self._root._phony_dims_mode is not None:
+ self._phony_dims = {}
+ phony_dims = defaultdict(int)
+ labeled_dims = defaultdict(int)
+
for k, v in self._h5group.items():
if isinstance(v, h5_group_types):
# add to the groups collection if this is a h5py(d) Group
@@ -253,6 +279,12 @@
current_size = v.size
self._dim_sizes[k] = size
+
+ # keep track of found labeled dimensions
+ if self._root._phony_dims_mode is not None:
+ labeled_dims[size] += 1
+ self._phony_dims[(size, labeled_dims[size] - 1)] = k
+
# Figure out the current size of a dimension, which for
# unlimited dimensions requires looking at the actual
# variables.
@@ -260,14 +292,52 @@
self._determine_current_dimension_size(k, current_size)
self._dim_order[k] = dim_id
+ else:
+ if self._root._phony_dims_mode is not None:
+ # check if malformed variable
+ if not _unlabeled_dimension_mix(v):
+ # if unscaled variable, get phony dimensions
+ vdims = defaultdict(int)
+ for i in v.shape:
+ vdims[i] += 1
+ for dimsize, cnt in vdims.items():
+ phony_dims[dimsize] = max(phony_dims[dimsize],
+ cnt)
+
if not _netcdf_dimension_but_not_variable(v):
- var_name = k
- if k.startswith('_nc4_non_coord_'):
- var_name = k[len('_nc4_non_coord_'):]
- self._variables.add(var_name)
+ if isinstance(v, h5py.Dataset):
+ var_name = k
+ if k.startswith('_nc4_non_coord_'):
+ var_name = k[len('_nc4_non_coord_'):]
+ self._variables.add(var_name)
+
+ # iterate over found phony dimensions and create them
+ if self._root._phony_dims_mode is not None:
+ grp_phony_count = 0
+ for size, cnt in phony_dims.items():
+ # only create missing dimensions
+ for pcnt in range(labeled_dims[size], cnt):
+ name = (grp_phony_count +
+ self._root._phony_dim_count)
+ grp_phony_count += 1
+ if self._root._phony_dims_mode == "access":
+ name = "phony_dim_{}".format(name)
+ self._create_dimension(name, size)
+ self._phony_dims[(size, pcnt)] = name
+ # finally increase phony dim count at file level
+ self._root._phony_dim_count += grp_phony_count
self._initialized = True
+ def _create_phony_dimensions(self):
+ # this is for 'sort' naming
+ for key, value in self._phony_dims.items():
+ if isinstance(value, int):
+ value += self._root._labeled_dim_count
+ name = "phony_dim_{}".format(value)
+ self._create_dimension(name, key[0])
+ self._phony_dims[key] = name
+
def _determine_current_dimension_size(self, dim_name, max_size):
"""
Helper method to determine the current size of a dimension.
@@ -478,7 +548,10 @@
# current version of h5py, this would require using the low-level
# h5py.h5ds.is_scale interface to detect pre-existing scales.
scale_name = dim if dim in self.variables else NOT_A_VARIABLE
- h5ds.dims.create_scale(h5ds, scale_name)
+ if h5py.__version__ < LooseVersion('2.10.0'):
+ h5ds.dims.create_scale(h5ds, scale_name)
+ else:
+ h5ds.make_scale(scale_name)
for subgroup in self.groups.values():
subgroup._create_dim_scales()
@@ -578,7 +651,8 @@
class File(Group):
- def __init__(self, path, mode='a', invalid_netcdf=None, **kwargs):
+ def __init__(self, path, mode='a', invalid_netcdf=None, phony_dims=None,
+ **kwargs):
try:
if isinstance(path, str):
if path.startswith(('http://', 'https://', 'hdf5://')):
@@ -618,6 +692,16 @@
# if we actually use invalid NetCDF features.
self._write_ncproperties = invalid_netcdf is not True
+ # phony dimension handling
+ self._phony_dims_mode = phony_dims
+ if phony_dims is not None:
+ self._phony_dim_count = 0
+ if phony_dims not in ['sort', 'access']:
+ raise ValueError('unknown value %r for phony_dims\n'
+ 'Use phony_dims=%r for sorted naming, '
+ 'phony_dims=%r for per access naming.'
+ % (phony_dims, 'sort', 'access'))
+
# These maps keep track of dimensions in terms of size (might be
# unlimited), current size (identical to size for limited dimensions),
# their position, and look-up for HDF5 datasets corresponding to a
@@ -626,8 +710,26 @@
self._current_dim_sizes = ChainMap()
self._dim_order = ChainMap()
self._all_h5groups = ChainMap(self._h5group)
-
super(File, self).__init__(self, self._h5path)
+ # initialize all groups to detect/create phony dimensions
+ # mimics netcdf-c style naming
+ if phony_dims == 'sort':
+ self._determine_phony_dimensions()
+
+ def _determine_phony_dimensions(self):
+ def get_labeled_dimension_count(grp):
+ count = len(grp._dim_sizes.maps[0])
+ for name in grp.groups:
+ count += get_labeled_dimension_count(grp[name])
+ return count
+
+ def create_phony_dimensions(grp):
+ grp._create_phony_dimensions()
+ for name in grp.groups:
+ create_phony_dimensions(grp[name])
+
+ self._labeled_dim_count = get_labeled_dimension_count(self)
+ create_phony_dimensions(self)
def _check_valid_netcdf_dtype(self, dtype, stacklevel=3):
dtype = np.dtype(dtype)
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/h5netcdf-0.7.4/h5netcdf/dimensions.py
new/h5netcdf-0.8.0/h5netcdf/dimensions.py
--- old/h5netcdf-0.7.4/h5netcdf/dimensions.py 2018-04-11 17:28:39.000000000
+0200
+++ new/h5netcdf-0.8.0/h5netcdf/dimensions.py 2020-02-04 18:28:06.000000000
+0100
@@ -1,4 +1,4 @@
-from collections import MutableMapping
+from collections.abc import MutableMapping
class Dimensions(MutableMapping):
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/h5netcdf-0.7.4/h5netcdf/tests/test_h5netcdf.py
new/h5netcdf-0.8.0/h5netcdf/tests/test_h5netcdf.py
--- old/h5netcdf-0.7.4/h5netcdf/tests/test_h5netcdf.py 2019-06-01
23:37:16.000000000 +0200
+++ new/h5netcdf-0.8.0/h5netcdf/tests/test_h5netcdf.py 2020-02-04
18:28:06.000000000 +0100
@@ -36,7 +36,7 @@
@pytest.fixture(params=['testfile.nc', 'hdf5://testfile'])
def tmp_local_or_remote_netcdf(request, tmpdir):
if request.param.startswith(remote_h5):
- if not pytest.config.option.restapi:
+ if not request.config.option.restapi:
pytest.skip('Do not test with HDF5 REST API')
elif without_h5pyd:
pytest.skip('h5pyd package not available')
@@ -482,13 +482,17 @@
def test_optional_netcdf4_attrs(tmp_local_or_remote_netcdf):
h5 = get_hdf5_module(tmp_local_or_remote_netcdf)
- with h5.File(tmp_local_or_remote_netcdf) as f:
+ with h5.File(tmp_local_or_remote_netcdf, 'w') as f:
foo_data = np.arange(50).reshape(5, 10)
f.create_dataset('foo', data=foo_data)
f.create_dataset('x', data=np.arange(5))
f.create_dataset('y', data=np.arange(10))
- f['foo'].dims.create_scale(f['x'])
- f['foo'].dims.create_scale(f['y'])
+ if h5py.__version__ < LooseVersion('2.10.0'):
+ f['foo'].dims.create_scale(f['x'])
+ f['foo'].dims.create_scale(f['y'])
+ else:
+ f['x'].make_scale()
+ f['y'].make_scale()
f['foo'].dims[0].attach_scale(f['x'])
f['foo'].dims[1].attach_scale(f['y'])
with h5netcdf.File(tmp_local_or_remote_netcdf, 'r') as ds:
@@ -514,15 +518,165 @@
ds.create_group('subgroup')
+def create_invalid_netcdf_data():
+ foo_data = np.arange(125).reshape(5, 5, 5)
+ bar_data = np.arange(625).reshape(25, 5, 5)
+ var = {'foo1': foo_data, 'foo2': bar_data,
+ 'foo3': foo_data, 'foo4': bar_data}
+ var2 = {'x': 5, 'y': 5, 'z': 5, 'x1': 25, 'y1': 5, 'z1': 5}
+ return var, var2
+
+
+def check_invalid_netcdf4(var, i):
+ pdim = 'phony_dim_{}'
+ assert var['foo1'].dimensions[0] == pdim.format(i * 4)
+ assert var['foo1'].dimensions[1] == pdim.format(1 + i * 4)
+ assert var['foo1'].dimensions[2] == pdim.format(2 + i * 4)
+ assert var['foo2'].dimensions[0] == pdim.format(3 + i * 4)
+ assert var['foo2'].dimensions[1] == pdim.format(0 + i * 4)
+ assert var['foo2'].dimensions[2] == pdim.format(1 + i * 4)
+ assert var['foo3'].dimensions[0] == pdim.format(i * 4)
+ assert var['foo3'].dimensions[1] == pdim.format(1 + i * 4)
+ assert var['foo3'].dimensions[2] == pdim.format(2 + i * 4)
+ assert var['foo4'].dimensions[0] == pdim.format(3 + i * 4)
+ assert var['foo4'].dimensions[1] == pdim.format(i * 4)
+ assert var['foo4'].dimensions[2] == pdim.format(1 + i * 4)
+ assert var['x'].dimensions[0] == pdim.format(i * 4)
+ assert var['y'].dimensions[0] == pdim.format(i * 4)
+ assert var['z'].dimensions[0] == pdim.format(i * 4)
+ assert var['x1'].dimensions[0] == pdim.format(3 + i * 4)
+ assert var['y1'].dimensions[0] == pdim.format(i * 4)
+ assert var['z1'].dimensions[0] == pdim.format(i * 4)
+
+
def test_invalid_netcdf4(tmp_local_or_remote_netcdf):
h5 = get_hdf5_module(tmp_local_or_remote_netcdf)
+ with h5.File(tmp_local_or_remote_netcdf, 'w') as f:
+ var, var2 = create_invalid_netcdf_data()
+ grps = ['bar', 'baz']
+ for grp in grps:
+ fx = f.create_group(grp)
+ for k, v in var.items():
+ fx.create_dataset(k, data=v)
+ for k, v in var2.items():
+ fx.create_dataset(k, data=np.arange(v))
+
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'r',
+ phony_dims='sort') as dsr:
+ i = len(grps) - 1
+ for grp in grps[::-1]:
+ var = dsr[grp].variables
+ check_invalid_netcdf4(var, i)
+ i -= 1
+
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'r',
+ phony_dims='access') as dsr:
+ for i, grp in enumerate(grps[::-1]):
+ print(dsr[grp])
+ var = dsr[grp].variables
+ check_invalid_netcdf4(var, i)
+
+ with netCDF4.Dataset(tmp_local_or_remote_netcdf, mode='r') as dsr:
+ for i, grp in enumerate(grps):
+ print(dsr[grp])
+ var = dsr[grp].variables
+ check_invalid_netcdf4(var, i)
+
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'r') as ds:
+ with raises(ValueError):
+ ds['bar'].variables['foo1'].dimensions
+
+ with raises(ValueError):
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'r',
+ phony_dims='srt') as ds:
+ pass
+
+
+def check_invalid_netcdf4_mixed(var, i):
+ pdim = 'phony_dim_{}'.format(i)
+ assert var['foo1'].dimensions[0] == 'y1'
+ assert var['foo1'].dimensions[1] == 'z1'
+ assert var['foo1'].dimensions[2] == pdim
+ assert var['foo2'].dimensions[0] == 'x1'
+ assert var['foo2'].dimensions[1] == 'y1'
+ assert var['foo2'].dimensions[2] == 'z1'
+ assert var['foo3'].dimensions[0] == 'y1'
+ assert var['foo3'].dimensions[1] == 'z1'
+ assert var['foo3'].dimensions[2] == pdim
+ assert var['foo4'].dimensions[0] == 'x1'
+ assert var['foo4'].dimensions[1] == 'y1'
+ assert var['foo4'].dimensions[2] == 'z1'
+ assert var['x'].dimensions[0] == 'y1'
+ assert var['y'].dimensions[0] == 'y1'
+ assert var['z'].dimensions[0] == 'y1'
+ assert var['x1'].dimensions[0] == 'x1'
+ assert var['y1'].dimensions[0] == 'y1'
+ assert var['z1'].dimensions[0] == 'z1'
+
+
+def test_invalid_netcdf4_mixed(tmp_local_or_remote_netcdf):
+ h5 = get_hdf5_module(tmp_local_or_remote_netcdf)
with h5.File(tmp_local_or_remote_netcdf) as f:
- f.create_dataset('foo', data=np.arange(5))
- # labeled dimensions but no dimension scales
- f['foo'].dims[0].label = 'x'
+ var, var2 = create_invalid_netcdf_data()
+ for k, v in var.items():
+ f.create_dataset(k, data=v)
+ for k, v in var2.items():
+ f.create_dataset(k, data=np.arange(v))
+
+ if h5py.__version__ < LooseVersion('2.10.0'):
+ f['foo2'].dims.create_scale(f['x1'])
+ f['foo2'].dims.create_scale(f['y1'])
+ f['foo2'].dims.create_scale(f['z1'])
+ else:
+ f['x1'].make_scale()
+ f['y1'].make_scale()
+ f['z1'].make_scale()
+ f['foo2'].dims[0].attach_scale(f['x1'])
+ f['foo2'].dims[1].attach_scale(f['y1'])
+ f['foo2'].dims[2].attach_scale(f['z1'])
+
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'r',
+ phony_dims='sort') as ds:
+ var = ds.variables
+ check_invalid_netcdf4_mixed(var, 3)
+
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'r',
+ phony_dims='access') as ds:
+ var = ds.variables
+ check_invalid_netcdf4_mixed(var, 0)
+
+ with netCDF4.Dataset(tmp_local_or_remote_netcdf, 'r') as ds:
+ var = ds.variables
+ check_invalid_netcdf4_mixed(var, 3)
+
with h5netcdf.File(tmp_local_or_remote_netcdf, 'r') as ds:
with raises(ValueError):
- ds.variables['foo'].dimensions
+ ds.variables['foo1'].dimensions
+
+
+def test_invalid_netcdf_malformed_dimension_scales(tmp_local_or_remote_netcdf):
+ h5 = get_hdf5_module(tmp_local_or_remote_netcdf)
+ with h5.File(tmp_local_or_remote_netcdf) as f:
+ foo_data = np.arange(125).reshape(5, 5, 5)
+ f.create_dataset('foo1', data=foo_data)
+ f.create_dataset('x', data=np.arange(5))
+ f.create_dataset('y', data=np.arange(5))
+ f.create_dataset('z', data=np.arange(5))
+
+ if h5py.__version__ < LooseVersion('2.10.0'):
+ f['foo1'].dims.create_scale(f['x'])
+ f['foo1'].dims.create_scale(f['y'])
+ f['foo1'].dims.create_scale(f['z'])
+ else:
+ f['x'].make_scale()
+ f['y'].make_scale()
+ f['z'].make_scale()
+ f['foo1'].dims[0].attach_scale(f['x'])
+
+ with raises(ValueError):
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'r',
+ phony_dims='sort') as ds:
+ pass
def test_hierarchical_access_auto_create(tmp_local_or_remote_netcdf):
@@ -548,7 +702,7 @@
g.dimensions['x'] = 2
g.dimensions['y'] = 3
- with h5py.File(tmp_local_netcdf) as f:
+ with h5py.File(tmp_local_netcdf, 'r') as f:
# all dimension IDs should be present exactly once
dim_ids = {
f[name].attrs['_Netcdf4Dimid'] for name in ['x', 'foo/x', 'foo/y']
@@ -610,24 +764,24 @@
if h5 is not h5py:
pytest.xfail('https://github.com/shoyer/h5netcdf/issues/48')
- with h5netcdf.File(tmp_local_or_remote_netcdf) as f:
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'w') as f:
f.dimensions['x'] = 2
f.create_variable('y', data=[1, 2], dimensions=('x',))
- with h5.File(tmp_local_or_remote_netcdf) as f:
+ with h5.File(tmp_local_or_remote_netcdf, 'r') as f:
assert f['y'].dims[0].keys() == [NOT_A_VARIABLE.decode('ascii')]
- with h5netcdf.File(tmp_local_or_remote_netcdf) as f:
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'a') as f:
f.create_variable('x', data=[0, 1], dimensions=('x',))
- with h5.File(tmp_local_or_remote_netcdf) as f:
+ with h5.File(tmp_local_or_remote_netcdf, 'r') as f:
assert f['y'].dims[0].keys() == ['x']
def test_invalid_netcdf_warns(tmp_local_or_remote_netcdf):
if tmp_local_or_remote_netcdf.startswith(remote_h5):
pytest.skip('h5pyd does not support NumPy complex dtype yet')
- with h5netcdf.File(tmp_local_or_remote_netcdf) as f:
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'w') as f:
# valid
with pytest.warns(None) as record:
f.create_variable('lzf_compressed', data=[1], dimensions=('x'),
@@ -642,7 +796,7 @@
f.create_variable('scaleoffset', data=[1], dimensions=('x',),
scaleoffset=0)
h5 = get_hdf5_module(tmp_local_or_remote_netcdf)
- with h5.File(tmp_local_or_remote_netcdf) as f:
+ with h5.File(tmp_local_or_remote_netcdf, 'r') as f:
assert '_NCProperties' not in f.attrs
@@ -665,20 +819,21 @@
def test_invalid_netcdf_okay(tmp_local_or_remote_netcdf):
if tmp_local_or_remote_netcdf.startswith(remote_h5):
pytest.skip('h5pyd does not support NumPy complex dtype yet')
- with h5netcdf.File(tmp_local_or_remote_netcdf, invalid_netcdf=True) as f:
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'w',
+ invalid_netcdf=True) as f:
f.create_variable('lzf_compressed', data=[1], dimensions=('x'),
compression='lzf')
f.create_variable('complex', data=1j)
f.attrs['complex_attr'] = 1j
f.create_variable('scaleoffset', data=[1], dimensions=('x',),
scaleoffset=0)
- with h5netcdf.File(tmp_local_or_remote_netcdf) as f:
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'r') as f:
np.testing.assert_equal(f['lzf_compressed'][:], [1])
assert f['complex'][...] == 1j
assert f.attrs['complex_attr'] == 1j
np.testing.assert_equal(f['scaleoffset'][:], [1])
h5 = get_hdf5_module(tmp_local_or_remote_netcdf)
- with h5.File(tmp_local_or_remote_netcdf) as f:
+ with h5.File(tmp_local_or_remote_netcdf, 'r') as f:
assert '_NCProperties' not in f.attrs
@@ -693,18 +848,18 @@
def test_invalid_then_valid_no_ncproperties(tmp_local_or_remote_netcdf):
- with h5netcdf.File(tmp_local_or_remote_netcdf, invalid_netcdf=True):
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'w', invalid_netcdf=True):
pass
- with h5netcdf.File(tmp_local_or_remote_netcdf):
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'r'):
pass
h5 = get_hdf5_module(tmp_local_or_remote_netcdf)
- with h5.File(tmp_local_or_remote_netcdf) as f:
+ with h5.File(tmp_local_or_remote_netcdf, 'r') as f:
# still not a valid netcdf file
assert '_NCProperties' not in f.attrs
def
test_creating_and_resizing_unlimited_dimensions(tmp_local_or_remote_netcdf):
- with h5netcdf.File(tmp_local_or_remote_netcdf) as f:
+ with h5netcdf.File(tmp_local_or_remote_netcdf, 'w') as f:
f.dimensions['x'] = None
f.dimensions['y'] = 15
f.dimensions['z'] = None
@@ -717,7 +872,7 @@
h5 = get_hdf5_module(tmp_local_or_remote_netcdf)
# Assert some behavior observed by using the C netCDF bindings.
- with h5.File(tmp_local_or_remote_netcdf) as f:
+ with h5.File(tmp_local_or_remote_netcdf, 'r') as f:
assert f["x"].shape == (0,)
assert f["x"].maxshape == (None,)
assert f["y"].shape == (15,)
@@ -880,3 +1035,12 @@
assert f.dimensions == {'x': None}
f = h5netcdf.File(tmp_local_or_remote_netcdf, 'r')
+
+
+def test_reading_special_datatype_created_with_c_api(tmp_local_netcdf):
+ "Test reading a file with unsupported Datatype"
+ with netCDF4.Dataset(tmp_local_netcdf, "w") as f:
+ complex128 = np.dtype([('real', np.float64), ('imag', np.float64)])
+ f.createCompoundType(complex128, 'complex128')
+ with h5netcdf.File(tmp_local_netcdf) as f:
+ pass
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/h5netcdf-0.7.4/h5netcdf/utils.py
new/h5netcdf-0.8.0/h5netcdf/utils.py
--- old/h5netcdf-0.7.4/h5netcdf/utils.py 2018-04-11 17:28:39.000000000
+0200
+++ new/h5netcdf-0.8.0/h5netcdf/utils.py 2020-02-04 18:28:06.000000000
+0100
@@ -1,4 +1,4 @@
-from collections import Mapping
+from collections.abc import Mapping
import numpy as np
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/h5netcdf-0.7.4/h5netcdf.egg-info/PKG-INFO
new/h5netcdf-0.8.0/h5netcdf.egg-info/PKG-INFO
--- old/h5netcdf-0.7.4/h5netcdf.egg-info/PKG-INFO 2019-06-01
23:42:49.000000000 +0200
+++ new/h5netcdf-0.8.0/h5netcdf.egg-info/PKG-INFO 2020-02-04
18:29:32.000000000 +0100
@@ -1,6 +1,6 @@
-Metadata-Version: 1.1
+Metadata-Version: 1.2
Name: h5netcdf
-Version: 0.7.4
+Version: 0.8.0
Summary: netCDF4 via h5py
Home-page: https://github.com/shoyer/h5netcdf
Author: Stephan Hoyer
@@ -176,8 +176,42 @@
we will raise ``h5netcdf.CompatibilityError``. Use
``invalid_netcdf=False`` to switch to the new behavior now.
+ Datasets with missing dimension scales
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+ By default [*]_ h5netcdf raises a ``ValueError`` if variables with no
dimension
+ scale associated with one of their axes are accessed.
+ You can set ``phony_dims='sort'`` when opening a file to let h5netcdf
invent
+ phony dimensions according to `netCDF`_ behaviour.
+
+ .. code-block:: python
+
+ # mimic netCDF-behaviour for non-netcdf files
+ f = h5netcdf.File('mydata.h5', mode='r', phony_dims='sort')
+ ...
+
+ Note, that this iterates once over the whole group-hierarchy. This has
affects
+ on performance in case you rely on lazyness of group access.
+ You can set ``phony_dims='access'`` instead to defer phony dimension
creation
+ to group access time. The created phony dimension naming will differ
from
+ `netCDF`_ behaviour.
+
+ .. code-block:: python
+
+ f = h5netcdf.File('mydata.h5', mode='r', phony_dims='access')
+ ...
+
+ .. _netCDF:
https://www.unidata.ucar.edu/software/netcdf/docs/interoperability_hdf5.html
+ .. [*] Keyword default setting ``phony_dims=None`` for backwards
compatibility.
+
Change Log
----------
+ Version 0.8.0 (February 4, 2020):
+
+ - Support for reading Datasets with missing dimension scales.
+ By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_.
+ - Fixed a bug where ``Datatype`` objects were treated as ``Datasets``.
+ - Fixed several issues with upstream deprecations.
Version 0.7.4 (June 1, 2019):
@@ -279,11 +313,9 @@
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 2
-Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.4
-Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.6
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/h5netcdf-0.7.4/setup.cfg new/h5netcdf-0.8.0/setup.cfg
--- old/h5netcdf-0.7.4/setup.cfg 2019-06-01 23:42:49.000000000 +0200
+++ new/h5netcdf-0.8.0/setup.cfg 2020-02-04 18:29:33.000000000 +0100
@@ -1,4 +1,4 @@
-[wheel]
+[bdist_wheel]
universal = 1
[egg_info]
diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn'
'--exclude=.svnignore' old/h5netcdf-0.7.4/setup.py new/h5netcdf-0.8.0/setup.py
--- old/h5netcdf-0.7.4/setup.py 2019-06-01 23:41:14.000000000 +0200
+++ new/h5netcdf-0.8.0/setup.py 2020-02-04 18:28:50.000000000 +0100
@@ -1,6 +1,9 @@
import os
from setuptools import setup, find_packages
+import sys
+if sys.version_info[:2] < (3, 6):
+ raise RuntimeError('Python version >= 3.6 required.')
CLASSIFIERS = [
'Development Status :: 4 - Beta',
@@ -8,13 +11,10 @@
'Operating System :: OS Independent',
'Intended Audience :: Science/Research',
'Programming Language :: Python',
- 'Programming Language :: Python :: 2',
- 'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
- 'Programming Language :: Python :: 3.4',
- 'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
+ 'Programming Language :: Python :: 3.8',
'Topic :: Scientific/Engineering',
]
@@ -24,12 +24,13 @@
long_description=(open('README.rst').read()
if os.path.exists('README.rst')
else ''),
- version='0.7.4',
+ version='0.8.0',
license='BSD',
classifiers=CLASSIFIERS,
author='Stephan Hoyer',
author_email='[email protected]',
url='https://github.com/shoyer/h5netcdf',
+ python_requires='>=3.6',
install_requires=['h5py'],
tests_require=['netCDF4', 'pytest'],
packages=find_packages())
