Re: Bug#679905: RFP: cctbx -- Computational Crystallography Toolbox
Hi, thanks for caring about Debian packaging efforts! This one is an old one, it must have been 2012 or 2013, and salsa was called alioth :-) What I remember was a rather inflexible packaging system, that forced us to duplicate existing code. And an effort that, well, slowly lost steam. Then I lost track of it due to real life commitments (my first child was born 2014). I also gratefully remember that Luc Bourhis was very helpful from the upstream side. I wouldn't want to bother him again, though, until we have something concrete to put on the table. Best regards, Baptiste Le 06/10/2021 à 10:13, Andreas Tille a écrit : > Hi, > > I've got a hint about cctbx and realised that there is an RFP (degraded > from ITP) as well as a salsa repository[1]. I took the freedom to > inject the latest upstream source and updated *parts* of the packaging > to recent standards. I have no idea whether this might give some new > inspiration for those who want to catch up with this package, which is > interesting for several teams (in CC). So I simply make some noise for > anybody who might want to continue from here. > > Kind regards > > Andreas. > > [1] https://salsa.debian.org/science-team/cctbx >
Re: DFSG status of petsc
Hello, just a small remark (not CC-ing -legal): Le 19/09/2015 10:46, Drew Parsons a écrit : > The win32 binary is irrelevant to us, so it no more violates dfsg than > a png image file would. Change its bits and distribute the altered exe > if you want to, the licence doesn't prohibit that. Hey, why not edit it > into the format of a png image file? If you ever distribute win32 binaries, make sure you check them for viruses (unless you know for sure that upstream does so). It happened to me once that my ClamAV spotted a virus in a DLL inside a published tarball of multiplatform Free Software; probably an infected computer had been used for the build. It would be embarrassing for Debian to convey such a virus, though it most probably wouldn't harm anyone. Baptiste
Re: cctbx debian package [HEADS UP]
Hi Radi, hi Frédéric, I'm going to push my changes tomorrow, if the nightly build test runs well. [HEADS UP] I made important changes to the files setup.py and sconsutils.py. If you have any changes pending on those two files, it might be good to push them now, so that I can handle the merge. I made the following additions: * a build_py command that can add the __future__ import in the python files that need it * a clean command, that is automatically called by dh, and also cleans the scons build dir * a test command, that runs the tests just after build. This last part is still a bit buggy (see below) * an install_data command that knows what to do with libtbx_env (using Radi's modify_libtbx_env function), so that we don't need a temporary file any more I also made the following architectural changes: * I moved the scons build dir inside the distutils temporary dir (I was unconfortable with the idea of setup.py automatically cleaning a directory it doesn't own) * I moved the computation of PREFIX and BUILDDIR to be done at run time in sconsutils.py (cleaner, and needed for the previous point) I also made code changes for platform independence and consistency, and fixed an embarassing bug in find_packages_and_extensions which prevented the resulting python modules from working (inclusion of boost_python_meta_ext.so). Yeah, lots of work at once, but I worked on it in complete disorder, so I couldn't push smaller parts. The testing strategy as of now, is to test the modified python files in build/lib.linux-x86_64-2.6 and the just-built executables in the scons build dir. Problems with the tests remain: * some of the tests (written either in Python or in Shell), which are designed to be run from the source directory, are not part of a python package. So they don't get installed to build/lib.linux-x86_64-2.6. This leads to No such file or directory errors when the test routine tries to run them. * some of the tests are really very long to run (at least on my machine) * I don't know yet how to assess the success or failure of the tests. I'm afraid we might have to grep the output * the setup.py test command is not standard, therefore it doesn't get automatically called by dh For these reasons, the tests are not run yet from debian/rules. Cheers, Baptiste -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/jus0r3$n03$1...@dough.gmane.org
Re: RE : cctbx debian package [HEADS UP]
Le 26/07/2012 20:45, PICCA Frédéric-Emmanuel a écrit : maybe in that case it would be better to rebase on top of the changes currently in the git repository git rebase origin/master is your friend. That way we should have a linear history. I find it much convenient that way. But it is only me :) Don't worry, I'm using gbp-pq now, so the only thing I have to merge is the patch (0010-adding-setup_py.patch). how do you decide if a package need this __future__ or not ? I am wondering also if we miss some __future__ features. We already have divide, what else ? This *is* the divide thing. I don't know of another __future__ problem. The source of this problem is really only the fact that upstream develop and test with -Qnew, which silently does the same thing as the __future__ import. So I add a from __future__ import division line to all non-empty files that don't have it already. I do not know about this, can you explain clearly what is the problem with thoses libtbx_env files. Maybe document this it in the wiki, so It will be easier the day we will discuss with the upstream. Radi can probably explain this better than me. Basically, at runtime, libtbx makes use of a serialized copy of the build environment. We need to modify this environment to correct the modules install paths. * I moved the scons build dir inside the distutils temporary dir (I was unconfortable with the idea of setup.py automatically cleaning a directory it doesn't own) you mean under the build directory ? I mean 'build/temp.linux-x86_64-2.6', which is the standard distutils place for build artefacts. I also made code changes for platform independence and consistency, and fixed an embarassing bug in find_packages_and_extensions which prevented the resulting python modules from working (inclusion of boost_python_meta_ext.so). for upstream ? No, it was just forgotten by my previous code and not included in the packages, which broke them. Can you put your reflections about the test suite in the wiki. I like this wiki too much :). I'll try to do that tomorrow. this is not an issue in that case, a simple should do the trick override_dh_auto_test: # for each version of python python setup.py test I didn't suspect that it was so easy :-) Still, I won't include it yet, as the tests take really too long. Cheers, Baptiste -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/jus64e$2vl$1...@dough.gmane.org
Re: cctbx debian package
Le 20/07/2012 23:54, Radostan Riedel a écrit : On Fri, 20. Jul 20:04, Frédéric-Emmanuel Picca wrote: I do not understand can you explain more precisely ? I think what he is means: How to setup a local apt repository and use it with pbuilder. Some nice infos can also be found at the Ubuntu wiki[1] [1] https://wiki.ubuntu.com/PbuilderHowto Exactly :-) And indeed, documentation can be found on the web. I was just wondering if there was interest in gathering such documentation in a single place on the wiki. As for the cdebootstrap bug, I must admit I didn't investigate it fully yet. There are several bug reports where pbuilder --create fails in the debootstrap stage, but I didn't recognise the exact same error messages. Sure thing is: this is solved in unstable. Cheers, Baptiste P.S.: I had no time to work on cctbx today, I hope I can push something tomorrow -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/jujvsu$rjs$1...@dough.gmane.org
Re: cctbx debian package
Le 19/07/2012 23:58, Radostan Riedel a écrit : On Thu, 19. Jul 22:14, Baptiste Carvello wrote: To be honnest, I would have better understood the patches if I had actually applied them, not just read them ;-) Learning the tools is a large part of the work as a new contributor: I just learned to use quilt. Welcome to the club :). I'm not so good myself. Fred can tell a story about how I annoyed him about how to use gbp-pq. Now I can't live without it. I really have to say that in the last 2 weeks I learned a lot and you will too by just doing. That's true. By the way, I could write down some of it for the wiki (especially the part about setting up cowbuilder, which I found to be a little tricky). Is DebianScience/Tools/InstallingCowbuilder an acceptable page name? I've been playing with gbp-pq today, and I'm trying to build your package right now. Is using -j 8 for building a standard Debian practice? That almost killed my computer, and finally grinded to a halt with 100% memory used, and 0% CPU :,-( So I had to change it to -j 2. I won't be working in the weekend, so I'll try to get some development done on monday. Cheers, Baptiste -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/juc9m9$dgd$1...@dough.gmane.org
Re: RE : cctbx debian package
Le 20/07/2012 21:14, PICCA Frédéric-Emmanuel a écrit : I do not understand, which part of cowbuilder installation was tricky. mostly two things: * there's a bug in squeeze's cdebootstrap, that gets triggered by cowbuilder --create, so I had to upgrade cdebootstrap to sid * it was not immediately obvious to me how to make home-built packages for the dependencies available to the chroot Cheers, Baptiste -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/jucdco$ani$1...@dough.gmane.org
Re: cctbx debian package
Le 19/07/2012 11:01, Picca Frédéric-Emmanuel a écrit : On Wed, 18 Jul 2012 16:59:13 +0200 Radostan Riedel raybu...@googlemail.com wrote: Hello Have you checked out my commit? I changed a little bit your scripts and I think this should already work. If you don't like my changes then tell me or fix it. just to understand what changes you have done to his scripts, it would have be nice to split your commit in two parts. First one remove the unneeded pycbf.py file, then update the scripts. It is easyer to review small modifications, if they are not lost in the middle of a 7000 lines commit :). To be honnest, I would have better understood the patches if I had actually applied them, not just read them ;-) Learning the tools is a large part of the work as a new contributor: I just learned to use quilt. * use distutils to add a from __future__ import division line to the files that lack it. Nice! Can you also think of a way to get a nice clean target. I guess the build2.7 directories are not cleaned. yes normally a package should be buildable twice in a row. So the clean target must restore the source package to it's initial state. Sure, but who must clean those dirs? setup.py, or dh? I'll tell you when this is done, so you can update your patch. Or you can push your changes in right away into the git so you be acknowledged ;). Yes I agreed with Radi, it is important to be acknowledged for your work. So please commit by yourself on the repository. Another toolset task ;-) I created an alioth account (bcarvello-guest), setup ssh and asked to be added to the project. So I should be able to push shortly. I'm not sure yet what the workflow is. Do I guess right, that you (Frédéric) are using gbp-pq, and Radi is using plain quilt? Cheers, Baptiste -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/ju9psi$lbt$1...@dough.gmane.org
Re: cctbx debian package
Hi all, Le 18/07/2012 12:11, Radostan Riedel a écrit : On Wed, 18. Jul 10:51, Picca Frédéric-Emmanuel wrote: can you explaine a little bit what is going on, the name of the patch says adding setup.py but there is this massif cctbx_sources/pycbf.py I moved a file that normally lies in BUILDIR/lib. And I also moved stdlib.py. I don't know why but upstream is spreading files around directories and sets PYTHONPATH because they want to do: import pycbf I thought it's the best way to just move that file to the top directory. Did I do it wrong? I simply cp that file to the top directory. I first try to move that file in the setup.py with shutil but I found it a little bit too hacky. I don't know how to do that with distutils but probably Baptiste will figure it out ;). I had overlooked this pycbf.py file, and thought only .so extensions existed in lib. Now, it looks like this file is automatically generated, so it should definitely not be included in the patch. I'll see how to move it with distutils, I don't think it will be difficult. There are also 2 directories that should be moved (boost_adaptbx/boost -- boost, clipper_adaptbx/clipper -- clipper) I solved this with distutils but we can consider moving them when we do a repack and remove clipper, boost directories. OK, this is the boost question I asked in a previous e-mail. To answer Frédéric's question: unless I'm mistaken, the code stems from the cctbx project. The Boost project don't seem to distribute a python package for their own project (as opposed to python packages built *with* Boost), so it might be acceptable to hijack the name, even though I'm not 100% confortable with it. Same goes for clipper. Radi, a question for you about this: moving boost_adaptbx/boost to boost means it can be imported with import boost, but no more with from boost_adaptbx import boost. Unless I'm mistaken, the upstream packaging allows both. Should we not do the same (a .pth file allows that)? Further questions: * I had understood that we would create several debian packages for the python interface. This is the reason why I created several setup-xx.py scripts. Do you think we should rather build a single python-cctbx package, or is it just a temporary fix. * your setup.py uses extra_path = 'cctbx'. Will this not force users to write (for example) import cctbx.cctbx.sgtbx instead of import cctbx.sgtbx? Or is there something I overlooked? What I'll try to do until the end of the week: * fix the pycbf.py and stdlib.py problems * move the .so extensions back to the root of the package, so that they are directly importable (needed for scitbx.array_family.flex) * use distutils to add a from __future__ import division line to the files that lack it. I'll tell you when this is done, so you can update your patch. Cheers, Baptiste -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/ju6i7q$46k$1...@dough.gmane.org
Re: cctbx debian package
Le 15/07/2012 10:27, PICCA Frédéric-Emmanuel a écrit : Yes it would be better to build using the scons provided by Debian. At the end of the process, cctbx should be repackage and removing scons is on the list :) OK, the problem was that the upstream build system insists on running with the same python version that was configured. So when just calling the system scons command, which uses the system default python, I can't build both versions. But I just thought of an easy way to work around this: I'll add /usr/lib/scons to the PYTHONPATH and directly call SCons.Script.main() with the python version I need. Can you provide more information about this runtime problem ? Nothing really severe, just 2 small things: 1) contrary to Radi's setup.py, mine installs packages directly into the main python path. That means you would import scitbx with import scitbx, not import cctbx.scitbx. I think it is better because both the upstream python files and user scripts use the first idiom. However, I was hesitating about boost_adaptbx: is it acceptable to install it as just boost in the main namespace? 2) upstream installs the .so extensions into the main python path, but almost all of them are only imported from one place inside the packages. So I moved them there. There is however one exception: scitbx_array_family_flex_ext.so gets imported from several places, so I need to either keep it global, or write an import stub. Cheers, Baptiste -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/ju0gkp$orb$1...@dough.gmane.org
Re: RE : RE : RE : RE : RE : cctbx debian package
Hi all, Le 10/07/2012 18:48, Radostan Riedel a écrit : On Tue, 10. Jul 16:26, PICCA Frédéric-Emmanuel wrote: if we got a working setup.py all this comes for free ;) Am I right assuming that: 1) packages, extensions and data files need to be correctly classified 2) only the install command need to actually use the standard distutils? ok when overriding the auto_build target we can step1: build the shared libraries with scons step2 use dh_python2 with setup.py to build all extensions. not sure about step 2: the extensions are built from many object files, so we would have to duplicate a large part of the build system. By the way, scons has some nice introspection features: you can inspect the dependencies in the build with a command like: bin/libtbx.scons -n --tree=all,prune lib/cctbx_sgtbx_ext.so | less so if we goes with the setup.py things we need to extract the extension information from their tbx build system. if not, we need to build byhand for each version of python supported by Debian like you already did in the rules file. I personnaly prefer the setup.py because this is the standard for python but if this is to much work. Let's build all extensions from their build system. what is your opinion ? I'm afraid that this could be too much. They are chaining SConscripts together with a lot of targets and checks (with boost threads, with openmp etc.) In my opinion the scons build works good now. We get all extension in ./lib and all shlibs in a subdir of ./lib. Seperation of the extension or python modules is another problem but you said it's possible so we go that way. When I cp all py-ext to cctbx_sources DIR I can easily call the current setup.py and I already have a python package. Beware: this setup.py does not correctly classify the extensions. I think they would be treated as package data, which would prevent dh_python2 from moving them to the right place. The question is now if we should just call their scons build system from setup.py or trigger setup.py from SConstruct? I personally would prefer a pure setup.py if upstream is on board but we need to build the shlibs too and this works good now with scons and the build of py-ext works also upstream native so it'll be easier to get upstream on board to our changes. I think like Radi: duplicating the build system is not just a one-time cost, but a long term maintenance cost. Plus it would probably annoy the upstream developers. I'd rather propose another solution: step1: build the shared libraries with scons _either_ step1.5a: build the extensions with a custom setup.py build command that calls scons under the hood _or_ step1.5b: build the extensions with scons and copy the results to the correct distutils build directory step2: use dh_python2 with setup.py install to install the extensions at the right place Both step1.5a and step1.5b have their problems (1.5a implies digging into distutils internals, 1.5b might be a little bit brittle), but I think that's the best strategy. If nobody objects, I will experiment with this until the begining of next week and report back if it works. Cheers, Baptiste -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4ffe8202.1000...@baptiste-carvello.net
Re: RE : RE : RE : cctbx debian package
Hi, Le 09/07/2012 23:04, Radostan Riedel a écrit : Did you checkout my latest commits. I fixed the build errors. Indeed, it now builds, but the install step fails with: dh_install --fail-missing --package=cctbx-bundle build2.7-omp/bin/* \ usr/bin dh_install: cctbx-bundle missing files (build2.7-omp/bin/*), aborting make: *** [install.cctbx-bundle] Error 2 (tested with 0a5fb8d) I thought so too. My intension with this setup.py file was only to get a clean installation of the modules. If just cp the *ext.so from ./lib to cctbx_sources and call: python setup.py install --root=/whatever/path we already get a clean install. One small difficulty though: the extensions need to be built for all the supported python versions. So we'll need to call the build system several times. And to install each version of the extension at the right place. By the way, I had a look at the import wrapper they use for loading extensions (boost.python.import_ext) and wrote a few lines about it in the wiki. As for now all Shlibs will go into ./lib/usr/lib. So they can be copied then too. I guess the best would be to specify a prefix during the build. But I'm not entirely sure yet. The problem is there are so much shlibs build. A few of them should go into seperate binary packages (libcctbx0, libsctbx0...) I can easily do this with regex but I'd like a more elegant method. There is a dh_install helper, which can be used to split files into separate packages. I'm not sure if it does what you want. It also uses lists and regexes :-) Cheers, Baptiste -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/jth966$om$1...@dough.gmane.org
Re: RE : RE : RE : cctbx debian package
Le 09/07/2012 12:53, Radostan Riedel a écrit : Baptiste can you tell us your opinion. Hi Radi, hi Frédéric, I'm still around, but I've been fighting with the tools (cowbuilder mostly, it now works, and I get the same build error as Frédéric). Regarding the python packages, I don't think we can rely much on the upstream build system, as they just set PYTHONPATH and use the modules in-place in the source tree. So we'll have to copy the .py files, and maybe a few data files too (not sure, I found some, but they seem to only be used in tests). As far as the setup.py is concerned, my proposition would be to write a fake 'build' stage that executes after the upstream build system and only copies the files to the appropriate places, so that the distutils 'install' stage can find them. Cheers, Baptiste -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/jtff7f$v9f$1...@dough.gmane.org
Packaging cctbx in Debian: request for comments
Hallo, I'm part of a group in debian-science who are investigating the packaging of cctbx in Debian (ITP is Bug#679905, discussion happens on the debian-science list). As you maintain objcryst-fox, which builds against cctbx, I though you might have some knowledge that could help us: tips or warnings against some pitfalls... I also would like your input regarding the C++ interface. In your opinion, is the ABI stable enough that we could provide a library package with SOs, or should we stick to static libraries? Finally, in a more or less distant future, would you be interested in patches to build objcryst-fox against a system cctbx package? Cheers, Baptiste -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4ff54a79.8010...@baptiste-carvello.net
Re: RE : Bug#679990: ITP: clipper -- object oriented development kit for crystallographic computing
Le 05/07/2012 00:33, Radostan Riedel a écrit : [...] And I don't wanna link these modules statically, waste disk space and loose the c++ compatibility. I was also wondering about the C++ interface. While static libraries waste disk space, they could be provided in a -devel package without having to commit to ABI compatibility (if I understand Policy 8.3 well). On the other hand, with a real lib package, we would have to maintain the SONAMEs. If, as I fear, upstream won't help with this task, and this beeing C++, I'm afraid it could be a difficult task. I asked Carlo Segre for his input, as he is a consumer of this C++ interface. About that control file: I'm still not sure how many python packages to build. Maybe I could get some suggestions here. Since cctbx consists of a lot of python modules for example smtbx, cctbx, mmtbx etc. which do not depend on each other, maybe it would be best to split that a little bit up. What is the logical choice here? Every module has a file called libtbx_config which describes what dependencies it has. Personnally, I always used cctbx in homegrown scripts (mostly sgtbx and uctbx). For that kind of use, a big package is fine IMHO, as you don't want to spend time cherry-picking the parts you need. Cheers, Baptiste -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/jt3j29$k3p$1...@dough.gmane.org
Re: Bug#679990: ITP: clipper -- object oriented development kit for crystallographic computing
I am a little bit concern by the list of libraries which are not int the graph. Well the graph is a very crude way to find dependencies. It only uses the libtbx_config, which only exist for the python modules. Radostand are you sure of the list you gave me ? example : libsmtbx_refinement_constraints.so I confirm Radostan's list, but you must use the same configure line as he does. With the default build, you don't have all those libraries. I even have 2 more libraries in my cctbx_build/lib: libann.so libclipper.so Cheers, Baptiste -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/jt4m83$iq5$1...@dough.gmane.org
Re: Bug#679990: ITP: clipper -- object oriented development kit for crystallographic computing
Le 03/07/2012 17:05, Picca Frédéric-Emmanuel a écrit : Yes we decided to work on thoses packages, because of cctbx which is used in bio crystallography. but also in some software like pyNX[1] used for nano material science :). mostly diffraction which is quite related to crystallography. maybe you chould be the MOM of july under my responsability what do you think about this Andreas and you of course Radi ? See you Frederic [1] http://sourceforge.net/apps/mediawiki/pynx/index.php?title=Main_Page Hello, I would be interested in helping package cctbx, which I have used in the past for its Python interface. Is there already an ITP for that? Speaking of which: * some of the design choices of the upstream authors are rather annoying for packaging, such as the eavy reliance on environment variables, or the use of python -Qnew. Has somebody been in contact with them? * it seems that some part of cctbx is already used in Debian in the package objcryst-fox. Maybe its maintainer has some knowledge to share... Cheers, Baptiste P.S.: I'm new to the list, but I met some of you at the ESRF workshop in Grenoble last week. -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/jt1kg8$4mi$1...@dough.gmane.org
Re: Bug#679990: ITP: clipper -- object oriented development kit for crystallographic computing
Hello, thanks to both of you for your answers. I subscribed to the ITP bug. I will write to Carlo Segre tomorrow. I also cloned the git repo. From a quick look at the commit messages, it looks like some of the original patches don't apply on latest cctbx. If it is so, I will first try to remedy that (a good way to understand those patches ;-) Cheers, Baptiste -- To UNSUBSCRIBE, email to debian-science-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/jt2695$ghh$1...@dough.gmane.org
