Bug#1032997: ITP: pylsl -- Python bindings for liblsl
Package: wnpp Severity: wishlist Owner: Enrico Zini X-Debbugs-Cc: debian-de...@lists.debian.org, debian-science@lists.debian.org * Package name: pylsl Version : 1.16.1 Upstream Contact: Christian A. Kothe * URL : https://github.com/labstreaminglayer/pylsl * License : MIT Programming Lang: Python Description : Python bindings for liblsl This is the Python interface to the Lab Streaming Layer (LSL). LSL is an overlay network for real-time exchange of time series between applications, most often used in research environments. LSL has clients for many other languages and platforms that are compatible with each other. - - - LSL seems to be a standard for taking data from devices like EEG sensors and making them availale to software that analyses them. I've packaged this for a personal project, and it would be straightforward to upload it to Debian. I can't take responsibility for supporting people with more professional needs besides trying to deal with FTBFS kind of issues, but after asking in the Debian Science Team mailing list, it can be maintained under the Science Team umbrella.
Bug#1032996: ITP: liblsl -- lsl library for multi-modal time-synched data transmission over the local network
Package: wnpp Severity: wishlist Owner: Enrico Zini X-Debbugs-Cc: debian-de...@lists.debian.org, debian-science@lists.debian.org * Package name: liblsl Version : 4.6.0 Upstream Contact: Christian A. Kothe * URL : https://github.com/sccn/liblsl * License : MIT Programming Lang: C++ Description : lsl library for multi-modal time-synched data transmission over the local network The lab streaming layer is a simple all-in-one approach to streaming experiment data between applications in a lab, e.g. instrument time series, event markers, audio, and so on LSL seems to be a standard for taking data from devices like EEG sensors and making them availale to software that analyses them. I've packaged this for a personal project, and it would be straightforward to upload it to Debian. I can't take responsibility for supporting people with more professional needs besides trying to deal with FTBFS kind of issues, but after asking in the Debian Science Team mailing list, it can be maintained under the Science Team umbrella.
Packaging liblsl
Hello, I've been needing to use liblsl[1] for a personal project, I noticed that it's easily packaged, and it would be straightforward to upload it to Debian. However, the package seems to have a much vaster reach and field of application than a hobby project, and I fear I can't take responsibility for supporting people on the issue tracker besides FTBFS kind of issues. With these premises, would it be ok if I uploaded it under the hat of the Debian Science Team and let it become a sort of shared good? [1] https://github.com/sccn/liblsl Enrico -- GPG key: 4096R/634F4BD1E7AD5568 2009-05-08 Enrico Zini
Re: Numba not in testing due to lots of autopkgtest errors - how to proceed
On Mon, Feb 06, 2023 at 11:23:47AM -0800, Diane Trout wrote: > I pushed numba 0.56.4+dfsg-2 to unstable, and it should be vastly > better than -1. I quickly counted there's about 13 test failures for > what functionality we can support, those failing tests are currently > getting skipped by pythons unittest expectedFailure and skip test > decorators, so autopkgtest will thinks it's clean. > > They have tests for cuda that are ignored, and I had to disable the > onetbb support since they haven't updated to the versions we are > shipping. > > I hope that helps out with hyperspy I confirm it is vastly better than 0.56.4+dfsg-1, thank you so much for your work! On amd64 at least everything works swimmingly, and hyperspy's test suite succeeds \o/ Enrico -- GPG key: 4096R/634F4BD1E7AD5568 2009-05-08 Enrico Zini signature.asc Description: PGP signature
Re: Numba not in testing due to lots of autopkgtest errors - how to proceed
On Wed, Feb 01, 2023 at 02:21:08PM -0800, Diane Trout wrote: > Ok what I've been doing so far was I did manage to backport all of > upstreams python 3.11 patches. > > Though I did make a mistake in one of them that generated a great deal > of errors (and more importantly I did find, fix, and pushed the fix for > that error to salsa) Thank you deeply for that! >From the context of hyperspy[1], if it's not too much trouble to at some point upload what you have to experimental it may help testing how things are improving for reverse dependencies [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028548 If you're at a stead point in importing packagse, if it can help I can try building a new numba package out of git, run the hyperspy test suite with it, and let you know if things changed since 0.56.4+dfsg-1 Enrico -- GPG key: 4096R/634F4BD1E7AD5568 2009-05-08 Enrico Zini
Help with hyperspy's test suite and numba
Hello, I'm trying to get hyperspy in shape for testing[1]. Diane's upload of numba and Andreas' upload of dask unblocked the issues with its reverse dependencies, and I'm now left with failing tests on its test suite[2] All tests are in this form, and the full test suite output can be found at [2]: FAILED hyperspy/tests/utils/test_roi.py::TestInteractive::test_lazy_interactive_snap[False] - numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend) No implementation of function Function() found for signature: >>> where(bool, float64, float64) There are 4 candidate implementations: - Of which 4 did not match due to: Overload in function '_OverloadWrapper._build..ol_generated': File: numba/core/overload_glue.py: Line 129. With argument(s): '(bool, float64, float64)': Rejected as the implementation raised a specific error: TypeError: code() argument 13 must be str, not int raised from /usr/lib/python3/dist-packages/numba/core/overload_glue.py:64 During: resolving callee type: Function() During: typing of call at /builds/science-team/hyperspy/debian/output/source_dir/.pybuild/cpython3_3.11_hyperspy/build/hyperspy/misc/array_tools.py (557) File "hyperspy/misc/array_tools.py", line 557: def round_half_towards_zero(array, decimals=0): # pragma: no cover return np.where(array >= 0, ^ Unfortunately, I don't know enough of numba to debug this further. Can someone help me figure out a next step to deal with this? [1] https://bugs.debian.org/1028548 [2] https://salsa.debian.org/science-team/hyperspy/-/jobs/3810806 Thanks, Enrico -- GPG key: 4096R/634F4BD1E7AD5568 2009-05-08 Enrico Zini signature.asc Description: PGP signature
HDF5 filter plugin packaging guidelines
Hello,
I've tried to condense what I proposed in my previous mail on this
topic[1] into documentation maintained in a git repository:
https://salsa.debian.org/science-team/h5f-packaging-guidelines
Since I didn't get replies to that mail I didn't feel there was
consensus enough to call it mini-policy, so I went for "guidelines".
Anyone can edit it to contribute their own bits of experience.
I'm pasting below the (short) content of the guidelines. I hope it can
be helpful to people packaging and using HDF5 filter plugins.
[1] https://lists.debian.org/debian-science/2022/12/msg00016.html
The current content of the repository is this:
- - - - - - - - - - -
# HDF5 filter plugin packaging guidelines
# Naming conventions
A good default name for a package containing an HDF5 filter plugin is
`hdf5-filter-plugin-*-{serial,openmpi}` which matches how HDF5 documentation
refers to them.
For packages that ship multiple plugins, or that ship plugins and other code,
you can `Provide:` one or more `hdf5-filter-plugin-*-{serial,openmpi}` virtual
package names, so that packages requiring the plugin can depend on it, however
it is being distributed.
# openmpi versions of plugins
You are free to package the two versions in two different packages, or in a
single one.
If you bundle them in a single package, you can use virtual package names to
show that it contains both versions.
# testing openmpi versions of plugins
This is a simple way of testing that the openmpi version of a plugin loads
correctly:
```
H5PY_ALWAYS_USE_MPI=1 python3
>>> import h5py
>>> h5py.h5z.filter_avail()
```
You can use it in `debian/tests/control` to have autopkgtests check that
plugins load correctly:
```
Test-Command: python3 -c 'import sys, h5py; sys.exit(not
h5py.h5z.filter_avail(32008))'
Depends: python3, python3-h5py, @, @recommends!
Restrictions: allow-stderr
Test-Command: H5PY_ALWAYS_USE_MPI=1 python3 -c 'import sys, h5py; sys.exit(not
h5py.h5z.filter_avail(32008))'
Depends: python3, python3-h5py, @, @recommends!
Restrictions: allow-stderr
```
This example uses 32008 as an example filter plugin ID, which is the ID for
bitshuffle: remember to change it with the one(s) for your own package!
- - - - - - - - - - -
Enrico
--
GPG key: 4096R/634F4BD1E7AD5568 2009-05-08 Enrico Zini
signature.asc
Description: PGP signature
RFC: Organizing HDF5 filter plugin packages
Hello,
I've been working on packaging some HDF5 filter plugins. There start to
be several of them in Debian, and it seems an area in need of some
organization. Here's a first proposal: my intention is to kickstart a
discussion with already some practical options on the table.
These are the packages containing filter plugins at the moment:
- hdf5-plugin-lzf: LZF (openmpi + serial)
- hdf5-filter-plugin: LZ4, BZip2, Bitshuffle (serial only)
- bitshuffle: LZF, Bitshuffle (openmpi + serial)
- hdf5-filter-plugin-blosc: Blosc (serial only)
- hdf5-plugin-zfp (serial only)
# Naming conventions
At the moment we have "hdf5-plugin-*", "hdf5-filter-plugin[-*]", and
plugins packaged as part of another binary (bitshuffle).
We could standardize on one. I'd open the discussion proposing
"hdf5-filter-plugin-*-{serial,openmpi}" as it seems to match how HDF5
documentation refers to them.
It can also make sense to use virtual package names to declare that a
plugin is being provided, for packages like bitshuffle that doesn't only
package a plugin, or hdf5-filter-plugin that packages multiple ones.
Note: I checked what Fedora is doing to try not to reinvent the wheel,
and I didn't recognize packages that specifically ship hdf5 plugins at
all.
# Plugins packaged multiple times
- The LZF filter plugin is currently packaged both in hdf5-plugin-lzf
and in bitshuffle. They use differing names liblzf_filter vs libH5LZF
though. Are they the same implementation?
- The bitshuffle plugin is currently packaged in hdf5-filter-plugin and
in bitshuffle. This currently causes an unpack error when coinstalling
them.
I could try to locate the reference versions of those plugins, and file
bugs for stripping them when they are bundled, in favour of the
reference version.
# openmpi versions of plugins
Should the two versions of plugins be packaged in the same package or as
separete packages?
hdf5 itself uses separate packages; bitshuffle packages both plugins,
and therefore pulls in both hdf5 libraries).
This can be solved by using *-serial/*-openmpi package names (real or
virtual), unless the extra complexity is not needed and users rather
expect to always have both plugins provided.
# testing openmpi versions of plugins
Is this a good enough way to test if the openmpi version of a plugin
works?
```
H5PY_ALWAYS_USE_MPI=1 python3
>>> import h5py
>>> h5py.h5z.filter_avail()
```
If so, I'd document it, possibly also with ready made autopkgtest
recipes, along the lines of:
```
# Test serial:
python3 -c 'import h5py; import sys; sys.exit(0 if
h5py.h5z.filter_avail() else 1)'
# Test openmpi:
H5PY_ALWAYS_USE_MPI=1 python3 -c 'import h5py; import sys; sys.exit(0 if
h5py.h5z.filter_avail() else 1)'
```
# Packaging the plugins as libraries
HDF5 filters can both be dynamically loaded as plugins, or directly
linked in code.
With hdf5-plugin-lzf I tried to package both plugin and library, for
both serial and openmpi, but couldn't figure out install locations for
the serial and openmpi versions of headers and static library that
wouldn't conflict.
Is this a problem that needs solving at all, or can we just skip
packaging headers and static libraries for plugins for now?
# A HDF5 filter plugin mini-policy?
I offer to distill all that comes ouf of this thread in a mini-policy.
Thoughts? Does this all make sense?
Enrico
--
GPG key: 4096R/634F4BD1E7AD5568 2009-05-08 Enrico Zini
signature.asc
Description: PGP signature
Re: Coinstallability of Fortran libraries built with different compilers
On Sat, Oct 22, 2011 at 04:24:23PM -0700, Steve Langasek wrote: One point to think of is how this works with multiarch, which is being introduced in Debian. Instead of 'ifort' should we use the architecture triplet, eg. i386-linux-intel instead ? Then the libraries go in i386-linux-intel rather than i386-linux-gnu for gfortran; ditto for the .mod files in /usr/include/i386-linux-intel I'm not familiar with this i386-linux-intel triplet. Is this a triplet targeted by the toolchain? Does software built for this target not use GNU libc? (I guess I can't presume that it uses any libc at all, since we're speaking specifically of fortran here.) I'm not sure about libc dependencies of fortran binaries, I'll leave Alastair to answer that bit. My understanding on library use and ABI compatibilities is that the critical point are .mod files in /usr/include, whereas .a or .so files are perfectly reusable across compilers. That means that fortran binaries compiled with any compiler are free to depend on C libraries built with any compiler. For example, /usr/lib/libnetcdff.so links with curl, libm, libc, libhdf5, and plenty others according to ldd. Ideally one would want to have parallel, per-compiler versions of fortran libraries, because of the different .mod file formats, and then share all the chain of C dependencies. Ciao, Enrico -- GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini enr...@enricozini.org signature.asc Description: Digital signature
gforgran module ABI
Hello, if I run ls /usr/include/*.mod | xargs -L 1 head -n 1 I see fortran modules installed that have different versions. At one weather centre they need to build fortran programs that need both grib_api (version 6) and netcdf (version 0), and there seems to be no version of gfortran that is happy to use both. How do you deal with the incompatibility of fortran modules across compiler versions? Should we regularly ask for binNMUs of packages shipping .mod files every time the default gfortran version changes, or is something better (like, a standard .mod file format) about to come? (netcdf seems to get hit by this quite often: #618967, #643894, #630564, #576968 and #543871, for example) Ciao, Enrico -- GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini enr...@enricozini.org signature.asc Description: Digital signature
Added tags science::calculation and science::visualisation
Hello, I've added two more tags for debian-science, after a discussion we had at Debconf: Tag: science::calculation Description: Calculation Tag: science::visualisation Description: Visualization They should be available in the tagging interface in half an hour or so. Ciao, Enrico -- GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini enr...@enricozini.org signature.asc Description: Digital signature
Re: Debtags for science
are specialised in sea waves, some in radar processing, some in satellite imagery, and they probably could go on categorising themselves until they end up having more categories than people, and I'm sure it's the same in pretty much every branch of science. Therefore, don't pick a category that describes what YOU do, because it will never be specific enough (except, of course, you name your category, for example Chris Walker or Enrico Zini, but then they wouldn't be a very useful classification system, or at least they wouldn't solve the problem that you're trying to solve here). Rather than describe what you do, describe what everyone in your faculty does: that probably gives you a good level of abstraction from which to look at things (note: this isn't exactly my idea, and came out of a conversation with Chris). Now that I put the big red boring verbose warning sign about the possible pitfalls, I can make a practical proposal to start a discussion as well as some work. Talking with Chris we came out almost by accident with this possible new facet, which I rather like: Facet: science Description: Science Tag: science::modelling Description: Modelling Tag: science::data-acquisition Description: Data acquisition Tag: science::plotting Description: Plotting Tag: science::bibliogaphy Description: Bibliography Tag: science::publishing Description: Publishing This more or less models the point of view of how that package can be useful for research work from the level of abstraction of don't say what you do, say what every scholar in your faculty does, and it consequently gives a facet that is potentially useful to everyone in your faculty, and therefore a big win. I propose we give this facet a look, se if there's anything wrong (missing things can always be added later) and give it a go. Then, since changing a tool also changes the nature of the work done with the tool, after we are satisfied that these tags are properly put into use, we can restart the discussion and see where it leads. Ciao, Enrico -- GPG key: 1024D/797EBFAB 2000-12-05 Enrico Zini [EMAIL PROTECTED] signature.asc Description: Digital signature
Re: Debian and meteorology
On Fri, Jun 08, 2007 at 07:10:14PM -0500, Jordi Gutierrez Hermoso wrote: On 08/06/07, Enrico Zini [EMAIL PROTECTED] wrote: In the last 2 years I've been working in a weather centre and I got to know, write and maintain a bit of meteorology-oriented software in Debian. I work very tangentially in meteorology (more than anything, I just solve PDEs that are inspired by meteorology), and I find your post very interesting. Which software do you maintain? dballe[1], provami[1], msat[2] and related libraries. I'm polishing for release a tool to import data from geostationary and polar satellite images into DB-All.e and I'm about to start working on a GRIB and BUFR archiver. Btw, do you know how can I have access to real meteorological data? What free data is out there? It's not that I don't want to pay; it's that I want to make a point about working only with free software and free data. Right now, I mostly use libnoise to generate various levels of coherent noise as initial conditions for my PDEs, but I would like to be able to work with real data. It depends what kind of data you need. For gridded data, ECMWF are shamefully not publishing their data (ehy, I pay it with my taxes!) but they'd give you a limited amout of free data for research only (http://www.ecmwf.int/services/archive/). NOAA is much better, and you can access their GRIB archive at http://nomads.ncdc.noaa.gov/ I don't know much about where to get radar maps, geostationary satellite images in useable formats, polar satellite strides and point observations. I know that you can get realtime METAR reports for free from any airport because the gnome weather applet uses that, but I have no idea about where to get historical data. From my experience, NOAA is publishing much more data and code than the ECMWF is, so you may want to start looking there. But I work more with creating software that decodes, converts and makes sense of the data rather than with acquiring the data. So if you find a data source and the files are in a data format that you can't easily read[3], do let me know and I might be able to point you at some (free) software that opens the thing. Ciao, Enrico [1] http://www.smr.arpa.emr.it/software/DBalle.html (look for the poster link) [2] http://www.smr.arpa.emr.it/software/msat/msat.html [3] there are many such specialised formats, the most popular being BUFR, GRIB, HRIT and undocumented dialects of NetCDF. -- GPG key: 1024D/797EBFAB 2000-12-05 Enrico Zini [EMAIL PROTECTED] signature.asc Description: Digital signature
Re: Bug#389773: ITP: cnf -- library for C and Fortran mixed programming
On Wed, Sep 27, 2006 at 06:09:03PM +0200, Enrico Zini wrote: The original distribution features an own packaging system, which was good when it was made, but now it's a bit tricky to work with. To make it easy package CNF for Debian and Fedora, I skipped part of the original packaging and created a new makefile which wraps the old one providing a more usual behaviour. I now reread this mail [1] from Nick Barkas and realised that some autotools packaging effort was started. However, browsing at the website I get the impression that those are nightly builds from CVS, and the stable versions are still packaged with the old system. And, still as I understand it, the project was terminated before the autotools-based code was released, and that work has been frozen as a nightly build. I can now do one of three things: 1) go on with my package, which has the stable version with a somehow fixed build system; 2) backport their autotools-based build system to the stable version; 3) package the CVS nightly build. At the moment my preferred option would be to go with my package, for two reasons: 1. because it's already done and ready to be uploaded and I'm lazy :) and 2. because it's a way to package the stable version with as little changes as possible. Someone with more insight on starlink's situation can still easily change my mind, though :) Ciao, Enrico [1] http://lists.debian.org/debian-science/2006/02/msg00020.html [2] http://dev.starlink.ac.uk/build/DEBIAN-3.0r3_i386/dist/ -- GPG key: 1024D/797EBFAB 2000-12-05 Enrico Zini [EMAIL PROTECTED] signature.asc Description: Digital signature
Re: Bug#389773: ITP: cnf -- library for C and Fortran mixed programming
On Wed, Sep 27, 2006 at 10:29:05AM -0700, Kevin B. McCarty wrote: Wondering -- what does this do differently than cfortran? Note, I'm *not* trying to imply that because cfortran is already in Debian, this is redundant. Just asking out of curiosity in the hope I learn something :-) [...] To be honest, I can't really say: for me it has been a job requirement, and all together it has worked rather well. It has a way to deal with almost any weird function passing and value return convention, as it requires you to use its own macros for all function declarations, passed parameters and return types you export from C to Fortran. This is the chapter on calling C from Fortran: http://www.starlink.rl.ac.uk/star/docs/sun209.htx/node18.html You may want to have a cursory look to the documentation, which I have to say is rather well done and not too long: I've been comfortable with it: http://www.starlink.rl.ac.uk/star/docs/sun209.htx/sun209.html I don't think it mentions COMPLEX: at least I didn't see it mentioned in the list of available macros: http://www.starlink.rl.ac.uk/star/docs/sun209.htx/node65.html and grepping the header file I actually get a definitive answer: /* Define macros for all the Fortran data types (except COMPLEX, * which is not handled by this package). */ I don't know if/how it handles gfortran, though, as gfortran is quite recent and CNF is quite old. But after a cursory look at the documentation (mainly at the way you define function arguments), the header is very straightforward to read. Ciao, Enrico -- GPG key: 1024D/797EBFAB 2000-12-05 Enrico Zini [EMAIL PROTECTED] signature.asc Description: Digital signature
Packaging Starlink CNF (highly portable fortran-C bridge)
Hello, For my work I developed quite some useful code (in meteorological environments) and it's licensed GPL. It now works, and it's time to think about distribution. All the dependencies are fine except CNF: http://www.starlink.ac.uk/static_www/soft_further_CNF.html CNF is a highly portable bridge between C and Fortran. It completely hides arch and compiler issues behind a convenient set of macros, and provides functions to common conversion tasks (like converting strings between Fortran and C). So far I found it superior to any other C/Fortran bridging layer, especially for portability. Now, CNF is licensed under the terms of GPL, but the installer's fairly weird (http://www.starlink.ac.uk/store/store.html): - Get the package from http://www.starlink.ac.uk/cgi-store/ftpform1?CNF - Unpack the shell archive - Go through an interactive process to configure and build and install The interactive process is also the process that generates the CNF macros according to the architecture and compiler in use. Is anyone interested in packaging this wild beast and would like to work on it together with me? Ciao, Enrico -- GPG key: 1024D/797EBFAB 2000-12-05 Enrico Zini [EMAIL PROTECTED] signature.asc Description: Digital signature
