Optimizing before applying migrations?
Hi everyone, We're working on upgrading our product from Django 1.6 to 1.8, and one of the biggest tasks is moving from Django Evolution to Django's migrations. Please correct me if I'm missing something, but from my testing and investigation into the code, it seems that the migrate command will always apply each migration file one-at-a-time, individually, rather than pre-optimizing all the changes that need to be made across all pending migrations. This results in a lot of unnecessary, potentially expensive SQL operations. I'm wondering if there are plans to optimize this to reduce migration time. For example, say we have a Book model. Over the course of 5 migrations, we've added 3 fields, changed one field, and removed one more. (Maybe there are other migrations for other models being applied in-between these migrations.) Now let's say either a new database is being created for the first time, or an existing database is being upgraded from a state prior to the creation of the Book model. My expectation is that the database would only need to issue a single CREATE TABLE statement (and possibly follow-up ALTER TABLE ADD CONSTRAINTs). However, it's looking like it's instead executing a CREATE TABLE and then multiple ALTER TABLEs (even multiple ones per field within one migration). While not too terrible for a brand new table, it is a problem for an existing table with a lot of data. If you have 5 new field changes to an existing table containing a million rows and you're running on MySQL (for instance), you'll end up with 5 ALTER TABLEs instead of one, causing MySQL to dump data/alter table/import data 5 times. Django Evolution used to do this as well, and it made for very slow database migrations for tables with a lot of data. We had complaints of multi-day-long upgrades from some big users. Nowadays, it contains an optimizer (https://github.com/beanbaginc/django-evolution/blob/master/django_evolution/mutators.py#L377) that first gathers all the operations that need to be performed, consolidates them whenever possible (such as rolling up any follow-up ALTER TABLEs into an earlier CREATE TABLE, filters out anything table-related for a table that will just be deleted later, and consolidates ALTER TABLEs into as few statements as possible.). This has done wonders for database migration time. I know there's squashmigrations, but that's only really useful if you're in full control over all affected code and databases. (The documentation states, "if you’re a third-party project, just ensure your users upgrade releases in order without skipping any," which isn't feasible in shipping products.). It seems that the same optimization work could be done at application time though, couldn't it? Are there already plans to add an optimizer for the full migration? What are the big issues blocking it? Thanks, Christian -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6e91e997-4e17-48fe-9ca4-3f34accfca41%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Pre-DEP: community support of unmaintained versions of Django
On Mon, Aug 24, 2015 at 9:32 PM, Ryan Hiebert wrote: > > > On Aug 24, 2015, at 5:37 PM, Carl Meyer wrote: > > > >> Any ideas on alternative ways to tackle this? I'm officially stuck. > > > > I'm afraid I don't have any solution to offer, other than embracing the > > "abstract vs concrete" dependencies distinction, and accepting the fact > > that users of your wheels/eggs are choosing to be more "on their own" > > when it comes to their chosen dependency versions. > > I definitely agree. setup.py files, for the reason you mentioned, are not > intended for concrete dependencies, while requirements.txt files are. They > are the right tool for the job. If I was you, that's what I would do. That sort of setup works fine when it's a standalone install on a server that someone experienced with the Python world is maintaining, or a developer working against a set of modules. It's not really an option for a full-blown application meant to be easily installed by people used to being able to run 'apt-get install ' or 'yum install ' As a more complex webapp, our packaging requires the compilation/minification of .less/.js files, which means any user using the sdist would need a suitable build environment. That raises a huge barrier to installation for a significant chunk of our userbase. Our goal is to keep installation simple for everyone, treating it much like any typical application, without requiring a whole bunch of pre-setup. Aside from this particular issue with the backported Django builds, we've been able to do this for years. It's basically the difference between being able to download/install a setup.exe for OpenOffice as opposed to needing a build environment and source to install it. It's starting to sound like the best bet for us is to move away from pip/easy_install for end-user consumption of the software and onto more of a dedicated installer that can manage dependencies more explicitly, maybe taking care of the building/consumption of a requirements.txt for them. Thanks for your thoughts on this, everyone :) Christian -- Christian Hammond - chip...@chipx86.com Review Board - https://www.reviewboard.org Beanbag, Inc. - https://www.beanbaginc.com > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Django developers (Contributions to Django itself)" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/django-developers/69fOquu8v-U/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/486DA079-7F97-4597-A7FB-CD179DFB54DC%40ryanhiebert.com > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAHfcSbxkQaUW6HvLGrTpcZEyrAZGAvgidHHELHOK8-XgCje0zg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Pre-DEP: community support of unmaintained versions of Django
On Wed, Aug 19, 2015 at 4:43 PM, Markus Holtermann wrote:
> Hey Christian,
>
> On Wed, Aug 19, 2015 at 02:41:49PM -0700, Christian Hammond wrote:
>
>> Regarding the version number, from what I've read, there's still a
>> reported
>> compatibility issue if we use the +localidentifer part of a version number
>> with older versions of setuptools. I need to do some testing on this. If I
>> don't hit any issues, I'm very happy to put an identifier on there. I
>> would
>> like to keep it at 1.6.11.x, rather than 1.6.12+, so that it's very clear
>> we're based on upstream Django 1.6.11.
>>
>
> Thanks, that sounds good to me. 1.6.11.x+unofficial if possible,
> otherwise 1.6.11.x
Looks like it's going to have to be 1.6.11.x.
However, I've hit a pretty major stumbling block here.
pip's removed support for dependency_links, requiring that all package
dependencies be made available on PyPI. This means there's no way we can
host a build of this ourselves, have a project depend on it, and build the
project as a package. It simply will ignore our builds and fail with a
dependency error (assuming a version range of >=1.6.11.1,<1.7). (If
installing from source, it looks like requirements.txt can still specify
links, but that doesn't help when building/distributing wheels/eggs.)
I tried a hacky workaround of creating a sdist package that invoked
setuptools to install using dependency_links, and depended on that package
instead. Older versions of pip prior to 7.0 would execute our setup.py and
in turn install our build of Django. However, since pip now converts sdists
to wheels prior to installing, it just skips over dependency_links,
resulting in the original problem.
The only other option I can think of is to actually upload the 1.6.11.x
build to PyPI under a different name ("beanbag-django16", for instance),
but keep the top-level module as "django". This will inevitably cause
conflicts with the officially installed Django package, though. Not a good
option.
So at this point, it seems that while we can maintain security backports of
Django 1.6.11, we can't realistically depend on them or trigger upgrading
to them. Short of the official Django package on PyPI either hosting our
backports or registering URLs to them, there might not be a clean way to do
this. I'm pretty sure is not the route you guys want to go for this, though.
Any ideas on alternative ways to tackle this? I'm officially stuck.
Christian
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CAHfcSbx9nwya7Aw3rnmECBZjE%3DwGW%2BSgzMT1tYo9xfXQgiHYjw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Re: Pre-DEP: community support of unmaintained versions of Django
Hey everyone, Thanks for the quick feedback on this :) Regarding the version number, from what I've read, there's still a reported compatibility issue if we use the +localidentifer part of a version number with older versions of setuptools. I need to do some testing on this. If I don't hit any issues, I'm very happy to put an identifier on there. I would like to keep it at 1.6.11.x, rather than 1.6.12+, so that it's very clear we're based on upstream Django 1.6.11. As for the downloads page, we're hosting everything on S3, as opposed to a dedicated host. This is so that our deliverables will be replicated and still be up if either Amazon or we have some outage. We *can* get HTTPS on this, using CloudFront, but that uses SNI, meaning only the latest Python 2.7.x releases (and 3.x) would be able to download files from there. We're looking into an approach that spins up a couple VMs that essentially wrap S3 with an HTTPS endpoint, but it increases our costs a bit, and we're just not ready to do that yet. I want HTTPS on this pretty badly, though, so we might try that approach when we can get to it. To attempt to compensate for this, though, we sign all the builds and include checksums in both the links and in dedicated .sha256sum files. Not the best solution, but it's something at least :/ Christian -- Christian Hammond - chip...@chipx86.com Review Board - https://www.reviewboard.org Beanbag, Inc. - https://www.beanbaginc.com On Wed, Aug 19, 2015 at 9:08 AM, Carl Meyer wrote: > Hi Christian, > > On 08/18/2015 08:01 PM, Christian Hammond wrote: > [snip] > > Also as a status update, we've started a fork and applied for the > > pre-notification list. I've backported all current security fixes to a > > branch, ensured the test suite passes with flying colors, and have added > > a README detailing everything you've requested. This is all up at > > https://github.com/beanbaginc/django. Let me know if there's anything > > you'd like changed. > > One thing was pointed out to me just now -- the downloads page _really_ > should use TLS. Any chance you could make that happen? > > Carl > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Django developers (Contributions to Django itself)" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/django-developers/69fOquu8v-U/unsubscribe > . > To unsubscribe from this group and all its topics, send an email to > django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/55D4AA04.3090403%40oddbird.net > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAHfcSbz%3DN9%2BFduAcSODtTVLEVmbD42Nu_CsiJHAQiXxtU70kCg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Pre-DEP: community support of unmaintained versions of Django
Hi Carl, I know it's been a while since we discussed this, but today's security release is the first one that's really affecting our product and we've finally got things in shape to be able to start distributing unofficial Django security releases (we've also just been swamped since our discussion earlier this year). I wanted to dig into the versioning scheme just a bit. We tried going with local version identifiers, but it turns out these are fairly useless. When specifying a dependency with a local version identifier, your only option is direct equality (==1.6.11+extended.1). You can't use any form of range (>=1.6.11+extended.4), as these get interpreted as >=1.6.11. This is the case with both easy_install and pip, and is documented in the PEP. That means consumers of the builds either need to specify an explicit version (preventing other packages or administrators from choosing a higher version when a new build is released), or need to depend on the official 1.6.11 and rely on administrators to keep up-to-date with the latest releases and manually install them. Not to mention that this all falls apart if using setuptools < 8, which doesn't support these identifiers. Given that, what would your feelings be on allowing for 1.6.11.x releases? This way we wouldn't take up any possible future versioning slots (unlikely as they may be), while being compatible with all versions of setuptools, and compatible with specifying version ranges. This would still appear solely on our own downloads page and announcements, with an appropriate note on them being unofficial builds. Also as a status update, we've started a fork and applied for the pre-notification list. I've backported all current security fixes to a branch, ensured the test suite passes with flying colors, and have added a README detailing everything you've requested. This is all up at https://github.com/beanbaginc/django. Let me know if there's anything you'd like changed. Thanks, Christian On Friday, March 27, 2015 at 9:25:20 AM UTC-7, Carl Meyer wrote: > > Hi John and Christian, > > On 03/27/2015 09:46 AM, John Paulett wrote: > > James, thanks for putting this together. > > > > Christian, I am in a similar position, supporting 2.6 for the next 6-12 > > months. I had planned to keep a personal fork of Django 1.6, > > backporting security patches as needed, but I would be happy to > > contribute to a common effort in this regard. > > > > As mentioned by others, keeping the effort unofficial may be ideal. A > > planed end-of-life of this secondary support may also be helpful to > > still convince people to migrate forward. > > > > For a personal fork this wasn't going to be needed, but I wonder how > > package distribution should occur. I doubt that publishing unofficial > > distributable under Django's PyPI project is advisable. > > I agree. > > I'll outline my idea of how this could work; consider it an alternative > proposal (or proposed revision of) James' DEP. (I don't think a DEP is > needed for this, but if other core team members feel that it is, I am > willing to work this up into DEP format). > > - The community support work happens in a third-party GitHub fork. If > you and Christian want to work together on it for 1.6, one of you can > create a repo and grant the other commit access. (This is of course > something that you have every right to do regardless of what the core > team thinks of it.) > > - One of you applies for security pre-notifications (under the > "distributor of Django" category -- essentially you are applying to be > equivalent to e.g. a Debian or Fedora packager of Django, who > could/might provide the same kind of extended support you are planning > to.) > > - I think from the core team's perspective, the ideal situation would be > that these releases not be on PyPI at all, but that your users would > download them directly from the "releases" section of your github repo, > or perhaps pip-install them from a PyPI-compatible index that you > publish somewhere on static file hosting. If this is acceptable, it > would allow you to use PEP 440 "local version identifiers" [1] in your > versions, which I think is the most semantically correct for this > situation. So your versions would look something like > "1.6.11+extended-1" (choice of actual local-version label up to you). > If not-on-PyPI is a total dealbreaker for your users, it might be an > option to release under a different PyPI project name, such as > "Django-16-Community-Support" or similar, but this is less than ideal. > For one, it would require the actual package to have a name other than > "Django", and it would require the use of something like "1.6.11.post1" > instead of the local version identifier, since PyPI does not accept > local version identifiers. > > - A preamble should be added to the README of the project on GitHub to > indicate that this is unofficial community-based e
Re: Pre-DEP: community support of unmaintained versions of Django
Hi Carl, James, Sorry for the late reply on this. I've been unavailable the past week. I know you guys are still sorting out how you want to run this, but I wanted to let you know that, given our current dependence on Python 2.6, I'm willing to do what's needed to maintain security backports for either an official or unofficial fork. I'd love to know some further thoughts on the logistics for this. Things like how versioning would work, what you'd need from me, what kind of timeframes we'd have to work with, any legalese needed to be notified of security issues, etc. Thanks! Christian On Friday, March 20, 2015 at 10:02:40 AM UTC-7, Carl Meyer wrote: > > Hi James, > > Thanks for taking the time to write this up carefully, research the > history, etc. I think some form of extended community-based support > could work, but I have some concerns about your specific proposal; > mostly, that it places too much of the responsibility with the core team. > > My feeling is that we don't really need a DEP or this much process > specified at this point; it'd be plenty for now to simply express > guarded openness to the idea of extended community maintenance for 1.6 > and/or 1.4 and see if anyone actually steps up to do the work. > > I'm also curious how this sounds to Christian and Stephen, who asked for > the extended support on django-users. Since they're the only ones who > have asked, if a proposal along these lines won't work for their cases, > then there's not much point in going forward with it. > > Comments interspersed: > > On 03/19/2015 04:59 PM, James Bennett wrote: > > There's been some recent discussion in a django-users thread[1] about > > the impending end-of-life of Django 1.6, which in turn will mean > > end-of-life for Django supported on Python 2.6. In addition, the > > end-of-life of the current LTS release, Django 1.4, will mean > > end-of-life for Django supported on Python 2.5. The following is a > > draft of a proposal based on ideas raised in that thread; it should be > > considered pre-DEP status, to be elevated to a DEP submission if there > > is sufficient support. > > > > > > Abstract > > > > > > The support process for older versions of Django is changed, as > > follows: > > > > * Official support for older versions of Django from the Django core > > team will continue to end at the scheduled times (i.e., upon release > > of a new minor version for non-LTS releases, at end of advertised > > support lifetime for LTS releases). > > > > * Unofficial support, by persons willing to volunteer their time, will > > be permitted through a secondary code repository existing solely to > > provide this support. > > "Permitted" sounds odd to me, since regardless of this DEP I don't think > the core team is in a position to forbid anyone else from continuing to > patch older Django releases if they want to. (I suppose if we really > wanted to we could prevent them from using the Django trademark to > describe what they are doing, but IANAL.) > > I'm also not convinced we should provide a special repo under the > django/ org for this. That doesn't bring any technical benefits, just > social ones (perception of "officialness"). And I'm not clear that > perception of officialness is a good thing - after all, this support is > unofficial. So I would lean towards saying that this concept should be > well-proven in a third-party repo (created by whoever is doing the work) > before we even consider a django/ repo for it. > > > * As the official end-of-life of each Django version approaches, the > > Django core team will solicit proposals and volunteers to provide > > unofficial support for that version past its official end-of-life, > > and select maintainer(s) from amongst those volunteers. This support > > will be known as "community maintenance". > > This is my strongest objection to the proposal. This whole scheme will > only work if the community maintainer is highly motivated, and there's a > strong need for the extended community maintenance for a specific > release. One indicator of both is that the initiative comes unsolicited > from the community. I think it makes sense to clarify our general > openness to such initiative, but I'm opposed to adding an additional > responsibility to the core team's plate to "solicit proposals and > volunteers" for community maintenance of every release that hits EOL. > > > Rationale > > - > > > > For the past five years, Django has been in the process of deprecating > > support for older 2.x release of Python, as part of a plan to move > > Django toward, first, compatibility with both Python 2 and Python 3 > > (completed as of Django 1.6), and ultimately with only Python 3 (in a > > future, to-be-determined, release). > > > > The timeline for this was as follows: > > > > * May 2010: Django 1.2 instituted a minimum Python version of 2.4. > > > >
Re: Enabling context access in simple_tag
On Dec 14, 12:02 am, Julien Phalip wrote: > On Dec 13, 10:16 am, Tai Lee wrote: > > -snip- > > > One suggestion from #1105 was to split out this functionality into > > individual decorators, @takes_context, @takes_block. I'm not sure how > > easy this would be technically to implement, but I think it would > > solve the `takes_context_plus` sink problem Malcolm describes as we > > potentially add more special case tag types (simple, inclusion, > > assignment, etc.) > > The djblets (created by the guys at reviewboard.org) contain two nifty > decorators for exactly this purpose: > @basictag:https://github.com/djblets/djblets/blob/master/djblets/util/decorator... > @blocktag:https://github.com/djblets/djblets/blob/master/djblets/util/decorator... > > This now seems to me like the perfect compromise. It would generally > allow for more versatility and to keep simple_tag (and the future > assignment_tag) free of takes_xxx cruft. > > Any chance these two decorators could be considered for inclusion in > Django core? For what it's worth, these two decorators have seriously cut down on our development and maintenance burdens. Whether they'd be sufficient as-is, I don't know (though you're welcome to have the code), but I'd also love to see equivalent functionality in Django. If I can help in some way to get these in shape (assuming you'd want to go that direction), just let me know. Christian -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
