Re: compressed fixture support
On Fri, Sep 19, 2008 at 6:05 PM, Bas van Oostveen <[EMAIL PROTECTED]> wrote: > > And would be as easy to implement, the possible usecases i've got is > with a big fixture of a organism/protein database. This can be > compressed very well and is only applied one, so no need to keep a 50+ > MB fixture around (and in version control) when it can be compressed in > a few mb with gzip. > > Anyways sorry to bump this thread ;) Bump away. However, nothing is going to happen until someone does some work on #4924 :-) The last time I looked at the patch on that ticket, it wasn't miles away from being ready for trunk. The basic implementation was sound - it just needed to be updated, slightly improved to include zip as a supported backend, and have tests and docs written. As you noted, this isn't a particularly difficult task - it just needs someone to do the work. I promise that if the patch gets some love and attention, I'll put it into trunk. Yours, Russ Magee %-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: compressed fixture support
I agree with Russ Magee's feeling. For compression you should just use pipes, you already pipe the output to a file, you should also pipe it through a compress app. An -c,--compress option does not make any sense without an accompanying -o,--output-file option as well. Though i'm +1 on having transparent support for gzip and bzip2 in commands where we actually specify the filename. ( ./manage.py loaddata somefile.json.gz ) And would be as easy to implement, the possible usecases i've got is with a big fixture of a organism/protein database. This can be compressed very well and is only applied one, so no need to keep a 50+ MB fixture around (and in version control) when it can be compressed in a few mb with gzip. Anyways sorry to bump this thread ;) On Sat, 2008-09-13 at 19:43 +0800, Russell Keith-Magee wrote: > On Sat, Sep 13, 2008 at 1:35 PM, Jeremy Dunck <[EMAIL PROTECTED]> wrote: > > > > I'd like to add support for fixture load/dump to deal with compressed > > (gzip) files transparently. > > Something like #4924, perhaps? :-) > > I'm not completely sold on the --compress option to dumpdata - there > are many options for compression other than gz, and you can use shell > pipes to compress the output of dumpdata using whatever scheme you > desire. > > However, loading compressed fixtures is a reasonable suggestion > (although, like Jacob, I don't have much need for them myself). > > Yours, > Russ Magee %-) > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: compressed fixture support
I have a use case for compressed fixtures too. > Jeremy Dunck wrote: >> I'd like to add support for fixture load/dump to deal with compressed >> (gzip) files transparently. Jacob Kaplan-Moss wrote: > I don't see a good reason *not* to do this, but I also don't see the > space requirements as a big deal, either; disk space is very cheap. Well, if fixtures are changed often, and the VCS does not employ binary diffs, the occupied storage space may actually be greater when compressing files. Subversion does binary diffs, Mercurial does not. I don't know about git and Bazaar. > How big are your fixtures, anyway? I have a number of files ranging up to 30MB. When storing them uncompressed in the repo, used storage increases much less, as if Mercurial stores them compressed anyway. However, commands are then slowed down significantly, as if Mercurial had to decompress the files in memory each time. We ended up storing the fixtures in explictly compressed form, to regain acceptable operating speed. -- Nicola Larosa - http://www.tekNico.net/ Like any other entrenched, complex, and often closeted industry, things in IT don't really work the way many people think they do. I'm guessing the Vatican is a bit like that, too. - Robert X. Cringely, May 2008 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: compressed fixture support
On Sat, Sep 13, 2008 at 1:35 PM, Jeremy Dunck <[EMAIL PROTECTED]> wrote: > > I'd like to add support for fixture load/dump to deal with compressed > (gzip) files transparently. Something like #4924, perhaps? :-) I'm not completely sold on the --compress option to dumpdata - there are many options for compression other than gz, and you can use shell pipes to compress the output of dumpdata using whatever scheme you desire. However, loading compressed fixtures is a reasonable suggestion (although, like Jacob, I don't have much need for them myself). Yours, Russ Magee %-) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: compressed fixture support
On Sat, Sep 13, 2008 at 6:35 AM, Jeremy Dunck <[EMAIL PROTECTED]> wrote: > I'd like to add support for fixture load/dump to deal with compressed > (gzip) files transparently. I don't see a good reason *not* to do this, but I also don't see the space requirements as a big deal, either; disk space is very cheap. How big are your fixtures, anyway? > If there's pushback on this, do others keep their fixtures in source > control? If so, doesn't it bloat your repo? Of course, keeping compressed fixtures in source control means you lose the ability to intelligently diff them... Really, though, seems like a fine feature, and one that'd be pretty easy to add. No objections here. Jacob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
