Re: Huge data files in Debian

2015-07-21 Thread Andreas Tille
On Mon, Jul 20, 2015 at 06:03:41PM -0400, Yaroslav Halchenko wrote: On Sat, 18 Jul 2015, Ole Streicher wrote: I'd support this -- we could also discuss the distributed filesystem approach there. Could this happen between Friday (DebCamp) and Tuesday? I have to leave on Wednesday...

Re: Huge data files in Debian

2015-07-20 Thread Yaroslav Halchenko
On Sat, 18 Jul 2015, Ole Streicher wrote: On Fri, Jul 17, 2015 at 09:58:32PM -0400, Yaroslav Halchenko wrote: +1 Would you register a large data set BoF in Summit? something like https://summit.debconf.org/debconf15/meeting/333/bof-big-data-packages/ Yes. ? not sure if I would

Re: Huge data files in Debian

2015-07-20 Thread Yaroslav Halchenko
On Sat, 18 Jul 2015, PICCA Frederic-Emmanuel wrote: Hi Wouldn't a p2p system scale better than any server based solution? Also in regards to cost... gittorrent[1] would be great for this. [1] http://blog.printf.net/articles/2015/05/29/announcing-gittorrent-a-decentralized-github/

Re: Huge data files in Debian

2015-07-18 Thread Philip Hands
Troy Benjegerdes ho...@hozed.org writes: On Fri, Jul 17, 2015 at 09:38:06PM +0200, Jakub Wilk wrote: * Ole Streicher oleb...@debian.org, 2015-07-17, 10:34: But: These packages sum up to ~25 GB, with the maximal package size of 3.5 GB. Well, that's a lot. Just as data points: * The

Re: Huge data files in Debian

2015-07-18 Thread Pedro Larroy
Hi Wouldn't a p2p system scale better than any server based solution? Also in regards to cost... Pedro. On Sat, Jul 18, 2015 at 9:42 AM, Philip Hands p...@hands.com wrote: Troy Benjegerdes ho...@hozed.org writes: On Fri, Jul 17, 2015 at 09:38:06PM +0200, Jakub Wilk wrote: * Ole

Re: Huge data files in Debian

2015-07-18 Thread Ole Streicher
Pedro Larroy pedro.larroy.li...@gmail.com writes: Wouldn't a p2p system scale better than any server based solution? Also in regards to cost... I am a bit afraid about the long-term availability of any non-standard, new solution: Anything we implement now (and used for Debian 9) should be

Re: Huge data files in Debian

2015-07-18 Thread Andreas Tille
Hi Ole, On Sat, Jul 18, 2015 at 10:36:56AM +0200, Ole Streicher wrote: ? not sure if I would actually like to be The Speaker ;) Why not. Your mail contained someconstructove details that could kickstart a discussion. There is not more needed in a BoF. I'd support this -- we could

Re: Huge data files in Debian

2015-07-18 Thread Ole Streicher
Andreas Tille andr...@an3as.eu writes: On Fri, Jul 17, 2015 at 09:58:32PM -0400, Yaroslav Halchenko wrote: +1 Would you register a large data set BoF in Summit? something like https://summit.debconf.org/debconf15/meeting/333/bof-big-data-packages/ Yes. ? not sure if I would actually

Re: Huge data files in Debian

2015-07-18 Thread Ole Streicher
Andreas Tille andr...@an3as.eu writes: I doubt that we can or should reproduce the catalogs from the original exposures in Debian -- this would require disk space, computing power and man power which we obviously don't have. May be ftpmaster might give some input here - if we are lucky one

Re: Huge data files in Debian

2015-07-18 Thread Troy Benjegerdes
On Sat, Jul 18, 2015 at 10:41:44AM +0200, Ole Streicher wrote: Pedro Larroy pedro.larroy.li...@gmail.com writes: Wouldn't a p2p system scale better than any server based solution? Also in regards to cost... I am a bit afraid about the long-term availability of any non-standard, new

Huge data files in Debian

2015-07-17 Thread Ole Streicher
Hi all, again a question where I find it difficult to put it into one single box. However, please reply to debian-science. I am trying to get the package astrometry.net into Debian. This package exists for Ubuntu [1], but (with some minor changes) could be uploaded to Debian as well. I already

Re: Huge data files in Debian

2015-07-17 Thread Yaroslav Halchenko
On Fri, 17 Jul 2015, Ole Streicher wrote: Hi all, again a question where I find it difficult to put it into one single box. However, please reply to debian-science. I am trying to get the package astrometry.net into Debian. This package exists for Ubuntu [1], but (with some minor changes)

Re: Huge data files in Debian

2015-07-17 Thread Andreas Tille
On Fri, Jul 17, 2015 at 12:03:36PM -0400, Yaroslav Halchenko wrote: Going to debconf15? may be we could have a BoF or just a lunch chat to discuss this eye-to-eye? ;) +1 Would you register a large data set BoF in Summit? Kind regards Andreas. -- http://fam-tille.de -- To

Re: Huge data files in Debian

2015-07-17 Thread Jakub Wilk
* Ole Streicher oleb...@debian.org, 2015-07-17, 10:34: But: These packages sum up to ~25 GB, with the maximal package size of 3.5 GB. Well, that's a lot. Just as data points: * The biggest binary package currently in the archive, ns3-doc_3.17+dfsg-1_all.deb, is only ~1GB. * The biggest

Re: Huge data files in Debian

2015-07-17 Thread Yaroslav Halchenko
On Fri, 17 Jul 2015, Andreas Tille wrote: Going to debconf15? may be we could have a BoF or just a lunch chat to discuss this eye-to-eye? ;) +1 Would you register a large data set BoF in Summit? something like https://summit.debconf.org/debconf15/meeting/333/bof-big-data-packages/ ? not

Re: Huge data files in Debian

2015-07-17 Thread Guillem Jover
Hi! On Fri, 2015-07-17 at 12:03:36 -0400, Yaroslav Halchenko wrote: On Fri, 17 Jul 2015, Ole Streicher wrote: But: These packages sum up to ~25 GB, with the maximal package size of 3.5 GB. What is the best way to deal with them? Loosely following the discussion about the Icedove icons, it

Re: Huge data files in Debian

2015-07-17 Thread Troy Benjegerdes
On Fri, Jul 17, 2015 at 09:38:06PM +0200, Jakub Wilk wrote: * Ole Streicher oleb...@debian.org, 2015-07-17, 10:34: But: These packages sum up to ~25 GB, with the maximal package size of 3.5 GB. Well, that's a lot. Just as data points: * The biggest binary package currently in the archive,

Re: Huge data files in Debian

2015-07-17 Thread Andreas Tille
Hi Yarislav, On Fri, Jul 17, 2015 at 09:58:32PM -0400, Yaroslav Halchenko wrote: +1 Would you register a large data set BoF in Summit? something like https://summit.debconf.org/debconf15/meeting/333/bof-big-data-packages/ Yes. ? not sure if I would actually like to be The Speaker ;)

Re: Huge data files in Debian

2015-07-17 Thread Yaroslav Halchenko
On Sat, 18 Jul 2015, Guillem Jover wrote: The ultimate solution we are aiming for (see http://datalad.org for more information) is to utilize git-annex and ship either mere pointers to git-annex sources or lean (without data) git-annex repositories which fetch data from original (or