Dear EasyBuilders and friends (and both :-P), As you perhaps are aware, upstream software providers are not always keen with providing older versions of their software packages. This happens independently of if the software in question is open source, semi-open or closed.
(warning: longer email follows) ## semi open source For instance, if you try to build DOLFIN from zero, you will quickly realize that that is doomed to fail, since source for MTL4/4.0.8878 is no longer avail. There are work-arounds that someone can do (eg. I now symlink the newer version, faking it to be older) but, these are all inferior to having a copy of the tarball. ## OSS Open source packages are also not immune to this, eg. finding the authoritative source and latest version for mpi-BLAST will keep you wondering for a while. (if you have that btw, how would you re-obtain it after all? :-P ) Even complete repos with history on github, can vanish from one day to the next. I certainly don't have a good feeling for the sources of the bioinformatics codes! ## closed source Finally, closed source packages are not handled any better. Especially compilers are a critical dependency as regards reproducibility. I have open two tickets on Intel, asking for their support on older versions and what is their stance: https://premier.intel.com/premier/IssueDetail.aspx?IssueID=691555 # icc/11.1.073 https://premier.intel.com/premier/IssueDetail.aspx?IssueID=691558 # impi/4.0.0.028 (can you read these issues with your own access, btw?) >From the later ticket (IMPI), I received yesterday the following response: > Our official policy is to support two major versions back. At present, that > includes Version 3.2 and Version 4.0, along with their corresponding updates. This is actually bad news, because if the upstream provider deprives you of the licensing (this particular issue was really about that!) it flushes away all the reproducibility argument, at least for new-coming HPC sites (we wouldn't have access to old versions). Yeah, you can always rebuild from scratch with a newer version, yada, yada... OK. # proposal Now, I understand that not everything will be possible but, I would really like we had a mirroring solution, at least for the open source software codes. Along with it, we would like to work on the SHA1/MD5 hashing business (ie. ensure that the codes are the ones we claim they are). a) Have you heard of any other kind of "open source" registry project, perhaps something that we could ride on registering specific tarballs? b) What technology do you think should be deployed? What are your preferences? (http, ftp, git, rsync, zsync ... whatever you think should be offered) c) Is your preference to integrate this to easybuild or, perhaps, keep it orthogonal? (eg. someone could bootstrap .local/easybuild/source with zsync & then let it go) ps. We are going to implement something anyway on our end (git+http+zsync looks attractive), HPCBIOS should cater for this, so the more that get interested in it, the merrier. ps2. Eventually we could come with some kind of solution for the non-OSS codes also, yet, I can safely predict that vendor licensing may put limits on what is doable. (github would not be the spot for that kind of stuff, in any case, btw) ps3. Ubuntu's "LTS" lineage is a good example of a commendable vendor retention policy. Somehow, not everybody around understands the universal need for LTS style solutions... thanks for looking into this, Fotis
