Hi Anton, thanks for your quick reply.
On Mon, Feb 17, 2014 at 07:12:40PM +0400, Anton Korobeynikov wrote: > > > I now wanted to give SPAdes a try and did the packaging in SVN[2]. The > > most important thing you might like to know is that I followed the > > Debian philosophy to use the just packaged code and thus I skipped the > > code copies of boos, python-yaml and python-joblib. > This may be wrong. Unfortunately, pyyaml and joblib are not > Python3-friendly. So we had to patch them to make sure one can run > spades.py under Python 3. We're packaging these patched versions along > with SPAdes and select the proper variant (Python 2 or Python3) on > fly. Have Debian patched these packaged to be able to run on both > Python 2 and Python 3? Have you checked SPAdes functionality with > Python 3? spades.py itself is Python 3 friendly. Well, in Debian the according modules are selected automatically depending from the Python version. So if you run python3 the Python modules path is automatically set to the Python 3 modules (which are in the packages python3-yaml and python3-joblib. If you confirm that SPAdes works "better" with Python 3 I will adapt the according script in /usr/bin/spades to force python3 as interpreter. (The patch will remain the same.) > > You also might like to notice that Debian always delivers an executable > > under /usr/bin which > > is in this case named spades and dispades consisting of a wrapper to > > spades.py which finds the main spades binary under /usr/lib/spades/bin > This is pretty bad. Because it definitely creates the command line > difference wrt the released version. We would strongly suggest to keep > spades.py and dipspades.py as the main runner scripts. Otherwise we > expect that the user experience from SPAdes may be broken: > > 1. Debian user who used to run just "spades" will run bare SPAdes > binary on other systems with (sometimes) cryptic error messages about > missed config, etc. > 2. non-Debian user who used to run "spades.py" will have hard time to > find the proper way to run SPAdes on Debian. This way is not > documented in the official SPAdes documentation. This is a never lasting discussion on our lists. In Debian language extensions are disapproved by policy and there are good reasons for this. I know that there is an ever lasting discussion specifically also in our Debian Med team since the very reasons you are mentioning above: Upstream authors of scientific software seem to to like expressing in what software a script is written for whatever reason. I would be happy if you would take your time to read one example of this discussion where I tried to collect links to very good reason to not use these language extensions: https://lists.debian.org/debian-med/2012/04/msg00103.html For sure I do not intend to create an incompatibility with any non-Debian instances of SPAdes and it is definitely not in my interest to create packages you as developers do not like. However, specifically in the case of spades.py I feel confused as a user since this seems to be basically a wrapper around "spades.c" code. Could you please elaborate about the reasons why you are "advertising" the script language in a way which may be needs to reverted in case at some time in time you consider rewriting the wrapper into Ruby, Haskell or whatever cool language (not that I personally would consider this sensible, just finding some examples). > > but these are technical details. > I looked over the patches and it seems like all of them are invalid. > Let me provide the feedback: > > 1. adapt_to_debian_pathes.patch > spades.py supports two directory layouts: > - the pretty standard /usr/bin + /usr/share layout > - all-in-one used mainly for development (using src, etc.) > > Basically you removed the developer's way of running SPAdes which is > pretty useless. If you really need this - then maybe you forgot to do > "make install" ? The debian/rules file was calling `make install` automatically but in this case (if I remember correctly) a lot of files went to /usr/bin but the user should actually only call spades.py and dipspades.py. I have seen no point in providing (ion)hammer, bwa-spades and even spades (the compiled binary) in /usr/bin/ since the user should not really touch these (if I'm correct - please correct me if I'm wrong). So after the installation the files location were rearranged to enable a file system layout which at least to my possibly naive look at things seems more clean from a users perspective. And yes, I did not regarded the developer's way of running SPAdes. The Debian package (for the moment) supports only the users who do not intend to rebuild from code. Do you think it would make sense to support a development library? > 2. do_not_install_third_party_libs.patch > See above about Python 2 vs Python 3. Please make sure that you can > run SPAdes with all supported Python versions (2.4, 2.5, 2.6, 2.7, 3.2 > and 3.3). On Debian Systems the default Python interpreter is currently Python 2.7. If you want to use Python 3 you need to call python3 (and you get Python 3.3 (for the moment). IMHO the advantage of the wrapper I was choosing is that we will be able to force python3 via this wrapper if you would prefer this. Can you tell me any reason for a user who simply wants to do genome assemblies to pick from a set of Python interpreters? > 3. use_debian-packaged_boost.patch > No need to patch. Just run cmake with -DSPADES_BOOST_ROOT=/usr/include Yep, I confirm the patch was a quick-n-(to)-dirty approach. I will drop this. > 4. use_debian_packaged_python-yaml.patch > See above > > So, it looks like that at least 2 of patches are useless and may be > simply removed. For other two ones you need to make sure you have not > broken Python 3 support :) As I said: Python 3 support is definitely not brocken: $ python Python 2.7.6 (default, Jan 11 2014, 14:34:26) [GCC 4.8.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import yaml as pyyaml >>> $ python3 Python 3.3.4rc1 (default, Jan 27 2014, 11:28:31) [GCC 4.8.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import yaml as pyyaml >>> Just the renamed versions were not working on a Debian system properly if you want to use the packaged versions instead of the code copies (which usually recieve a veto from our security team). > > The problem I have is that I have access to some other computer on which > > a copy of SPAdes 2.5 is installed (I assume from the prebuilded binary > > but I can not say for sure). When running the very same test I get > > different *.fasta/*.fastq results. What I'm wondering now: What is the > > reason for the differences in the test between version 2.5 and 3.0. I > > think it would make sense to provide some kind of test suite to ensure > > users that their results have good chances to be reproducible. > SPAdes 3.0 contains many improvements, so the difference in the output > is expected. SPAdes does include the testsuite, however it's not > pretty-much user-friendly. Does this mean there is a test suite included I just not detected? The point is that it does not need to be user friendly because we try to run it on behalf of our users a) at package build time (if possible without network access) b) in a persiodically scheduled cron job over the whole Debian archive So if this test suite is scriptable I would be happy to learn how I could do this. > Also, please make sure that you specified all the necessary > dependencies (gcc 4.7, cmake 2.8.8). That's granted inside the target release. $ gcc --version gcc (Debian 4.8.2-14) 4.8.2 $ cmake --version cmake version 2.8.12.1 For the sake of backports to (very) old Debian systems this might be a problem but it is not even for the latest released version (Debian "Wheezy" 7.x). > Below is the list of additional components with different licenses > which are missed from 'copyright' file (actually, everything from ext > dir): > > ext/include/btree - Apache 2.0 license > ext/src/jemalloc - BSDlike license. Note that this is a *modified* > version of the jemalloc > ext/src/yaml-cpp - MIT license. I believe this version is unmodified, > but I do not recall offhand > ext/include/kseq - MIT license > ext/include/libcxx - MIT license Thanks for these hints. I did not yet finished the debian/copyright file but this list actually helps. > ext/tools/bwa - GPLv3. Slightly modified to make sure it compiles with > modern GCC's. My plan is to replace this by the Debian packaged version: $ apt-cache show bwa | grep ^Version Version: 0.7.5a-2 which definitely compiles with latest gcc (and we actually provided patches to upstream in the past to let this happen). Do you think it is a bad idea to rather use bwa 0.7.5a than the code copy you are providing inside SPAdes? > If you have any questions or suggestions - feel free to ask and I will > be happy to answer and provide help, if needed. As I said I'm very happy about any comments of yours (depite potentially contradicting opinions about language extensions). I would really like to get some result out in Debian you as upstream developers are happy about. Currently it is work in progress and any input of yours is really helpful. Kind regards Andreas. -- http://fam-tille.de -- To UNSUBSCRIBE, email to [email protected] with a subject of "unsubscribe". Trouble? Contact [email protected] Archive: http://lists.debian.org/[email protected]
