Re: Packaging Open Porous Media (OPM) software suite
Hi, I have worked on the packages quite a bit. They are now all added to the CI on salsa and are green. See the list https://salsa.debian.org/search?utf8=%E2%9C%93=opm-_id=_id==false_ref=_source=navbar There have some struggles and I had to work a bit to get it work or deactivate some stuff On Wed, May 26, 2021 at 03:32:54PM +0200, Markus Blatt wrote: - "build i386" as the upstream tests fail on this platform. One of the reasons is probably the Y2k38 problem on this architecture. Can I restrict Architecture in debian/control to 64bit only? It turned out that even with fixes for i386 the packages would be rather useless. Hence the work was stopped. Instead we now only lit 64bit architectures in debian/control and intsruct salsa-ci to skip i386 builds via variables: SALSA_CI_DISABLE_BUILD_PACKAGE_I386: 1 in debian/salsa-ci.yml -reprotest times out. I increased the timeout of the CI to 2 hours. At least for opm-common some of the reprocibility bugs have been fixed. Unfortunately for the other modules reprotest had to be deactivated (SALSA_CI_DISABLE_REPROTEST: 1) as the test runs as root and some of our tests are run using MPI (which refuses to run as root). See https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/160 autopkgtests: I have added autopkgtests to all modules. Unfortunately, I had to deactivate running them on all modules but opm-common. The problem is that they need the packages from other OPM modules. That works for the build (see below) but autopkgtest has a problem with ca-certificates when using external repositories, see https://salsa.debian.org/salsa-ci-team/pipeline/-/issues/201#note_244674 For opm-simulators the shared CI on gitlab.com ran out of RAM and I had to deactivate parallel builds in the CI (SALSA_CI_DPKG_BUILDPACKAGE_ARGS: --jobs=1) - Please consider reducing the number of source packages. Uploading/sponsoring 7 source packages, especially if some of them should go through NEW queue. It is pain. Each of the source packages has its own repository upstream. [...] I always assumed that Debian source package correspond to official upstream tarballs/repositories. Indeed and I feel with you. When I added the CI to all packages I already had to deal with that. As we need the OPM packages we depend on, I used the following approach - Activated aptly for salsa-ci: (SALSA_CI_DISABLE_APTLY: 0) - Created a gpg keypair for the aptly repostories - In the Setting->CI/CD->Variables - Set variable (type variable) SALSA_CI_APTLY_GPG_KEY to the private key (same key for all repos) - Set variable (type variable) SALSA_CI_APTLY_GPG_PASSPHRASE to the passphrase for the key - Set variable (type File) SALSA_CI_EXTRA_REPOSITORY to a recent one of aptly, e.g. deb https://salsa.debian.org/science-team/opm-common/-/jobs/1734127/artifacts/raw/aptly unstable main - Set variable (type File) SALSA_CI_EXTRA_REPOSITORY_KEY to the public (same key for all repos) I wonder how other multi-repository projects deal with these build dependencies. Any hint/insight would be highly appreciated Would be cool if somebody would take another look? Is there anything I need to do because of the freeze? CI is using unstable. Cheers, Markus
Re: Packaging Open Porous Media (OPM) software suite
Hi Andrius, Thanks alot for your kind help. Sorry, about late reply but I wanted to address the comments of Anton in the opm-common package first. That turned out to be quite some work. On Thu, Apr 29, 2021 at 12:39:31PM +0300, Andrius Merkys wrote: On 2021-04-28 16:17, Markus Blatt wrote: - For the library packages the SONAME will change with each release, as the ABI is quite unstable. The version is not part of the library package name, which lintian would warn about. But we are overwriting the warning currently. This lintian warning is quite important. If the ABI is unstable, I would suggest making these libraries private by putting them under /usr/lib//opm (for example) and shipping them in the same binary package as the main executable(s). Not sure what you mean by private. The libraries are/will/can/be used by other packages, e.g. the binaries in lib-opm-simulators-bin need the libraries from all the lib-opm-* packages. Also opm-grid is a DUNE modules and a grid along the DUNE interface. Hence the library from lib-opm-grid could be used in user code and future Debian packages. Should I still put them beneath /usr/lib//opm? It seems a bit uncommon but I might be missing something. Thanks and Cheers, Markus
Re: Packaging Open Porous Media (OPM) software suite
Hi Markus, On 2021-04-28 16:17, Markus Blatt wrote: > I have recently posted an ITP (bug )for this software > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=987381 Thanks a lot an interesting ITP, and welcome to the team! Just a couple of comments on points not addressed by Anton: > - For the library packages the SONAME will change with each release, as > the ABI is quite unstable. The version is not part of the library > package name, which lintian would warn about. But we are overwriting > the warning currently. This lintian warning is quite important. If the ABI is unstable, I would suggest making these libraries private by putting them under /usr/lib//opm (for example) and shipping them in the same binary package as the main executable(s). > - Does the top-level directory in the tarballs need to have a special > name (like opm-common-2021.04 for version 2021.04 of opm-common)? > The reason for asking is that upstream tags final versions as > release//final, which will make uuscan us funny looking > opm-common-release-2021.04-final. If it does upstream needs to use the > more common tagging scheme v, or we need to create the > tarballs ourselves and use gbp import-orig . The suffix '-final' can be removed by uscan's option 'uversionmangle'. Best, Andrius
Re: Packaging Open Porous Media (OPM) software suite
Hi Markus, Welcome on board! I have added you to the salsa group of the Debian Science Team. Feel free to push your projects there. I did not have a deep look into the packaging, but have some notes: - Try to use the newer compat-version. 11 is a little bit outdated. - Please add autopkgtests for packages. - Try to add Gitlab-CI for every package. It definitely improves the package quality and makes the package maintenance easier. - Please consider reducing the number of source packages. Uploading/sponsoring 7 source packages, especially if some of them should go through NEW queue. It is pain. > - In the future we imight want to be able to install several versions I would recommend not to do it. At least in the official version of the Debian package. That just means that every new upload should have another binary name, NEW queue, sponsoring from DD etc. It can only have sense for very big and very popular packages like boost for example. But for small scientific packages it is just overhead. > Does the top-level directory in the tarballs need to have a special No. If you do "dpkg-source -x AAA.dsc" the top level directory will be overwritten. Just a general note. Make the simple things not too complicated. Otherwise the package will not work and is very difficult in maintenance. Best regards Anton Am Mi., 28. Apr. 2021 um 15:40 Uhr schrieb Markus Blatt : > Hi, > > I have recently posted an ITP (bug )for this software > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=987381 > > > > * Package name: opm-common, opm-material, opm-grid, opm-models, opm- > simulators, opm-upscaling > Version : 2021.04 > Upstream Author : OPM > * URL : https://www.opm-project.org/ > * License : GPL2+/GPL3+ > Programming Lang: C++, Python > Description : Open Porous Media (OPM) software suite > . > The Open Porous Media (OPM) software suite provides libraries and > tools for modeling and simulation of porous media processes, especially > for simulating CO2 sequestration and improved and enhanced oil > recovery. > > > > I am part of the OPM development team. We are prepared to maintain the > packages, but as nobody of us is a Debian developer, we would of course > need help for uploading. We have not done so, but intend to ask Ansgar > Burchardt whether he has time for this. > > I have an account on salsa and the current version of the packaging > effort can be found there. https://salsa.debian.org/blattms/opm-common > https://salsa.debian.org/blattms/opm-material > https://salsa.debian.org/blattms/opm-models > https://salsa.debian.org/blattms/opm-grid > https://salsa.debian.org/blattms/opm-simulators > https://salsa.debian.org/blattms/opm-upscaling To us as recreational > packagers this looks good now, but chances are that we have missed > stuff. > > I also requested being added to the Debian Science team on Salsa, but > that does not seem to have happened yet (or I missed it). Once that > happens I will gladly push to https://salsa.debian.org/science-team. > > Some notes on the packaging: > > - Packaging is based on the branch point (git commit) for the upcoming > 2021.04 release. Once that is released we will add the new version. > - We use git-buildpackage with the default layout and pristine-tar > (there is a debian/dbp.conf file that activate pristine-tar, and tells > it which files to strip from upstream) > - We have stripped some sources for the Debian packaging (jenkins, > travis, embedded source code, upstream packaging for Redhat, Debian, > OpenSuse) to satisfy the Debian Policy and prevent unnecessary changes > from appearing in the Repositories of the Debian packaging. Note that > these are also stripped from the tarballs and listed in > debian/copyright (Excluded-Files) and debian/gbp.conf for ease of > maintenance. > - Apart from opm-material and opm-models, which are header-only > libraries and missing lib.deb because of this, most > repositories create Packages lib-dev.deb, lib.deb, > lib-doc.deb and (for opm-common, opm-grid, opm-simulators) > lib-bin.deb. There are also packages with python bindings: > python3-opm-common.deb, python3-opm-simulators.deb > - For the library packages the SONAME will change with each release, as > the ABI is quite unstable. The version is not part of the library > package name, which lintian would warn about. But we are overwriting > the warning currently. > - opm-upscaling is missing manpages for the binaries. This will be fixed > upstream and backported soon. > > Some questions: > > - In the future we imight want to be able to install several versions > of libopm-simulators-bin.deb concurrently (which would need to depend > on different versions of the library packages lib.deb, and > probably install binaries into different versioned directories and use > update-alternatives. Is there a proposed policy/approach to do this? I > would assume that this
Packaging Open Porous Media (OPM) software suite
Hi, I have recently posted an ITP (bug )for this software https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=987381 * Package name: opm-common, opm-material, opm-grid, opm-models, opm- simulators, opm-upscaling Version : 2021.04 Upstream Author : OPM * URL : https://www.opm-project.org/ * License : GPL2+/GPL3+ Programming Lang: C++, Python Description : Open Porous Media (OPM) software suite . The Open Porous Media (OPM) software suite provides libraries and tools for modeling and simulation of porous media processes, especially for simulating CO2 sequestration and improved and enhanced oil recovery. I am part of the OPM development team. We are prepared to maintain the packages, but as nobody of us is a Debian developer, we would of course need help for uploading. We have not done so, but intend to ask Ansgar Burchardt whether he has time for this. I have an account on salsa and the current version of the packaging effort can be found there. https://salsa.debian.org/blattms/opm-common https://salsa.debian.org/blattms/opm-material https://salsa.debian.org/blattms/opm-models https://salsa.debian.org/blattms/opm-grid https://salsa.debian.org/blattms/opm-simulators https://salsa.debian.org/blattms/opm-upscaling To us as recreational packagers this looks good now, but chances are that we have missed stuff. I also requested being added to the Debian Science team on Salsa, but that does not seem to have happened yet (or I missed it). Once that happens I will gladly push to https://salsa.debian.org/science-team. Some notes on the packaging: - Packaging is based on the branch point (git commit) for the upcoming 2021.04 release. Once that is released we will add the new version. - We use git-buildpackage with the default layout and pristine-tar (there is a debian/dbp.conf file that activate pristine-tar, and tells it which files to strip from upstream) - We have stripped some sources for the Debian packaging (jenkins, travis, embedded source code, upstream packaging for Redhat, Debian, OpenSuse) to satisfy the Debian Policy and prevent unnecessary changes from appearing in the Repositories of the Debian packaging. Note that these are also stripped from the tarballs and listed in debian/copyright (Excluded-Files) and debian/gbp.conf for ease of maintenance. - Apart from opm-material and opm-models, which are header-only libraries and missing lib.deb because of this, most repositories create Packages lib-dev.deb, lib.deb, lib-doc.deb and (for opm-common, opm-grid, opm-simulators) lib-bin.deb. There are also packages with python bindings: python3-opm-common.deb, python3-opm-simulators.deb - For the library packages the SONAME will change with each release, as the ABI is quite unstable. The version is not part of the library package name, which lintian would warn about. But we are overwriting the warning currently. - opm-upscaling is missing manpages for the binaries. This will be fixed upstream and backported soon. Some questions: - In the future we imight want to be able to install several versions of libopm-simulators-bin.deb concurrently (which would need to depend on different versions of the library packages lib.deb, and probably install binaries into different versioned directories and use update-alternatives. Is there a proposed policy/approach to do this? I would assume that this needs to be prepared now by putting some kind of versioning into the Debian packages for the libraries (e.g. like Petsc). - Does the top-level directory in the tarballs need to have a special name (like opm-common-2021.04 for version 2021.04 of opm-common)? The reason for asking is that upstream tags final versions as release//final, which will make uuscan us funny looking opm-common-release-2021.04-final. If it does upstream needs to use the more common tagging scheme v, or we need to create the tarballs ourselves and use gbp import-orig . Thanks a lot for your kind help. Cheers, Markus -- Dr. Markus Blatt - HPC-Simulation-Software & Services http://www.dr-blatt.de Pedettistr. 38, 85072 Eichstätt, Germany, USt-Id: DE279960836 Tel.: +49 (0) 160 97590858
