Request to join the team
Hi, I would like to join the Python team, to help update and maintain the Python packages in Debian as a daily Debian and Python user, to learn how Debian works, and to contribute to the community. I have read and accept the team's policy. My salsa login is noctis. Thank you! -- Regards, Blair Noctis OpenPGP_signature Description: OpenPGP digital signature
Re: dxtbx pytest issue
For the record, I found it..., the upstream modify the HDF5_PLUGIN_PATH when
loading the dxtbx module.
they guess that they are using conda and override the path. All this is useless
on Debian since the plugin are system installed properly.
Cheers
Fred
# Ensures that HDF5 has the conda_base plugin path configured.
#
# Ideally this will be properly configured by the conda environment.
# However, currently the dials-installer will not install a path-correct
# conda_base folder, so it needs to be updated manually.
_hdf5_plugin_path = libtbx.env.under_base(os.path.join("lib", "hdf5", "plugin"))
# Inject via the environment if h5py not used yet, or else use h5py
if "h5py" not in sys.modules:
os.environ["HDF5_PLUGIN_PATH"] = (
_hdf5_plugin_path + os.pathsep + os.getenv("HDF5_PLUGIN_PATH", "")
)
else:
# We've already loaded h5py, so setting the environment variable won't work
# We need to use the h5py API to add a plugin path
import h5py
h5_plugin_paths = [h5py.h5pl.get(i).decode() for i in
range(h5py.h5pl.size())]
if _hdf5_plugin_path not in h5_plugin_paths:
h5py.h5pl.prepend(_hdf5_plugin_path.encode())
Re: dxtbx pytest issue
Hello Neil > Looks like you need a -v option to see more detail. thanks for the advices, I found by removing files one by one that the failling behavious is due to the import of the library itself. the failing test PASS by himself, but if I add a useless import dxtbx inside, it failes. so there is some interaction between the dxtbx package and the h5py library, I will dig it. Cheers Fred
Re: dxtbx pytest issue
On Thu, 1 Sep 2022 13:32:36 +0200 (CEST)
PICCA Frederic-Emmanuel
wrote:
> Hello,
>
> I am still trying to package[2] dxtbx[1], and now I end up with
> something strange. When I run the test suite during the build I have
> a failure like this
>
> tests/test_dataset_as_flex.py ..F..F..F
>[ 2%]
Looks like you need a -v option to see more detail.
> I put the error message bellow, it is quite long
>
> now If I execute by hand only the failing test like this, it works
>
>
> $ pytest-3 tests/test_dataset_as_flex.py
The 2% indicates that other tests have executed before the test series
gets to test_dataset_as_flex.py. When you run it directly, no other
tests are run. This could indicate that there is interference between
tests (an upstream problem) in that the tests are not sufficiently
discrete and are affected by which other tests are executed before the
failing test. pytest has support to solve these problems, if the test
suite is written correctly.
It could also be a version problem with any of the build-dependencies
(like h5py), so check for various requirements*.txt and see if upstream
have a .gitlabci.yml or .github config which clarifies exactly what
versions are used.
You may need to exclude certain tests in case, for example, h5py in
Debian is built with options that dxtbx cannot handle.
> before investigating further, I would like your advice in order to
> debug this sort of issue.
>
> first what is the difference between
>
> pytest and pytest
pytest scans the directory tree to collect all the tests it can find.
pytest only collects the tests from that file.
Check for a pytest.ini in the upstream source tree and also setup.cfg
Use the -v option to see more of what is going on and the
--collect-only option is useful too.
See pytest --help as other filtering options are available as well.
You'll need to have a local build environment (fakeroot debian/rules
build etc.) & dig into the Python unit test methods and see what is
going on.
> thanks for your help
>
> Frederic
>
> [1] https://github.com/cctbx/dxtbx
> [2] https://salsa.debian.org/science-team/dxtbx
>
>
>
> full error message
>
> ___ test_dataset_as_flex[int-dataset_as_flex_int-bshuf_lz4]
>
>
> type_name = 'int', creator =
> converter =
>
> @pytest.mark.parametrize(
> "creator",
> [
> uncompressed,
> gzip,
> bshuf_lz4,
> ],
> )
> @pytest.mark.parametrize(
> "type_name,converter",
> [
> ("int", dataset_as_flex_int),
> ("float", dataset_as_flex_float),
> ("double", dataset_as_flex_double),
> ],
> )
> def test_dataset_as_flex(type_name, creator, converter):
> # Create an in-memory HDF5 dataset with unique name
> f = h5py.File(type_name + ".h5", "w", driver="core",
> backing_store=False)
> shape = (20, 20, 20)
> > dataset = creator(f, shape, type_name)
>
> tests/test_dataset_as_flex.py:64:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ tests/test_dataset_as_flex.py:34: in bshuf_lz4
> return file.create_dataset(
> /usr/lib/python3/dist-packages/h5py/_debian_h5py_serial/_hl/group.py:161:
> in create_dataset dsid = dataset.make_new_dset(group, shape, dtype,
> data, name, **kwds)
> /usr/lib/python3/dist-packages/h5py/_debian_h5py_serial/_hl/dataset.py:106:
> in make_new_dset dcpl = filters.fill_dcpl( _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>
> plist = 0x7fccdbbd9760> shape = (20, 20, 20), dtype = dtype('int64'), chunks
> 0x7fccdbbd9760> = (10, 10, 10)
> compression = 32008, compression_opts = (0, 2), shuffle = None
> fletcher32 = None, maxshape = None, scaleoffset = None, external = []
> allow_unknown_filter = False
>
> def fill_dcpl(plist, shape, dtype, chunks, compression,
> compression_opts, shuffle, fletcher32, maxshape, scaleoffset,
> external, allow_unknown_filter=False):
> """ Generate a dataset creation property list.
>
> Undocumented and subject to change without warning.
> """
>
> if shape is None or shape == ():
> shapetype = 'Empty' if shape is None else 'Scalar'
> if any((chunks, compression, compression_opts, shuffle,
> fletcher32, scaleoffset is not None)):
> raise TypeError(
> f"{shapetype} datasets don't support chunk/filter
> options" )
> if maxshape and maxshape != ():
> raise TypeError(f"{shapetype} datasets cannot be
> extended") return h5p.create(h5p.DATASET_CREATE)
>
> def rq_tuple(tpl, name):
> """ Check if chunks/maxshape match dataset rank """
> if tpl in (None, True):
> return
> try:
> tpl = tuple(tpl)
> except TypeError:
> raise TypeError('"%s" argument must be None or a
dxtbx pytest issue
Hello,
I am still trying to package[2] dxtbx[1], and now I end up with something
strange. When I run the test suite during the build I have a failure like this
tests/test_dataset_as_flex.py ..F..F..F [ 2%]
I put the error message bellow, it is quite long
now If I execute by hand only the failing test like this, it works
$ pytest-3 tests/test_dataset_as_flex.py
=
test session starts
=
platform linux -- Python 3.10.6, pytest-7.1.2, pluggy-1.0.0+repack
rootdir: /home/picca/debian/science-team/dxtbx, configfile: pytest.ini
plugins: requests-mock-1.9.3, forked-1.4.0, xdist-2.5.0, mock-3.8.2,
dials-data-2.4.0
collected 9 items
tests/test_dataset_as_flex.py .
[100%]
==
9 passed in 0.61s
==
before investigating further, I would like your advice in order to debug this
sort of issue.
first what is the difference between
pytest and pytest
thanks for your help
Frederic
[1] https://github.com/cctbx/dxtbx
[2] https://salsa.debian.org/science-team/dxtbx
full error message
___ test_dataset_as_flex[int-dataset_as_flex_int-bshuf_lz4]
type_name = 'int', creator =
converter =
@pytest.mark.parametrize(
"creator",
[
uncompressed,
gzip,
bshuf_lz4,
],
)
@pytest.mark.parametrize(
"type_name,converter",
[
("int", dataset_as_flex_int),
("float", dataset_as_flex_float),
("double", dataset_as_flex_double),
],
)
def test_dataset_as_flex(type_name, creator, converter):
# Create an in-memory HDF5 dataset with unique name
f = h5py.File(type_name + ".h5", "w", driver="core",
backing_store=False)
shape = (20, 20, 20)
> dataset = creator(f, shape, type_name)
tests/test_dataset_as_flex.py:64:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dataset_as_flex.py:34: in bshuf_lz4
return file.create_dataset(
/usr/lib/python3/dist-packages/h5py/_debian_h5py_serial/_hl/group.py:161: in
create_dataset
dsid = dataset.make_new_dset(group, shape, dtype, data, name, **kwds)
/usr/lib/python3/dist-packages/h5py/_debian_h5py_serial/_hl/dataset.py:106: in
make_new_dset
dcpl = filters.fill_dcpl(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
plist =
shape = (20, 20, 20), dtype = dtype('int64'), chunks = (10, 10, 10)
compression = 32008, compression_opts = (0, 2), shuffle = None
fletcher32 = None, maxshape = None, scaleoffset = None, external = []
allow_unknown_filter = False
def fill_dcpl(plist, shape, dtype, chunks, compression, compression_opts,
shuffle, fletcher32, maxshape, scaleoffset, external,
allow_unknown_filter=False):
""" Generate a dataset creation property list.
Undocumented and subject to change without warning.
"""
if shape is None or shape == ():
shapetype = 'Empty' if shape is None else 'Scalar'
if any((chunks, compression, compression_opts, shuffle, fletcher32,
scaleoffset is not None)):
raise TypeError(
f"{shapetype} datasets don't support chunk/filter options"
)
if maxshape and maxshape != ():
raise TypeError(f"{shapetype} datasets cannot be extended")
return h5p.create(h5p.DATASET_CREATE)
def rq_tuple(tpl, name):
""" Check if chunks/maxshape match dataset rank """
if tpl in (None, True):
return
try:
tpl = tuple(tpl)
except TypeError:
raise TypeError('"%s" argument must be None or a sequence
object' % name)
if len(tpl) != len(shape):
raise ValueError('"%s" must have same rank as dataset shape' %
name)
rq_tuple(chunks,
Re: RFS: tpm2-pytss
Hi Carsten, On 2022-09-01 10:56, Carsten Schoenert wrote: Hello Claudius, I can take care on this, might take time until the weekend. Thanks! regards, Claudius
Re: RFS: tpm2-pytss
Hello Claudius, I can take care on this, might take time until the weekend. Am 01.09.22 um 08:47 schrieb Claudius Heine: Hi, would someone be willing to sponsor the tpm2-pytss package? https://salsa.debian.org/python-team/packages/tpm2-pytss This package is a dependencies of newer versions of tpm2-pkcs11 (>= 1.8.0). regards, Claudius -- Regards Carsten
RFS: tpm2-pytss
Hi, would someone be willing to sponsor the tpm2-pytss package? https://salsa.debian.org/python-team/packages/tpm2-pytss This package is a dependencies of newer versions of tpm2-pkcs11 (>= 1.8.0). regards, Claudius
