Re: Ideas for a new DEP on long-term Django API compatibility

[Pascal Chambon]

Hello,

Just for the record, we spent a good time discussion this change (to bring
> inheritance of Admin Actions in line with Python’s expected inheritance
> rules). We reviewed the entire history of the feature. We explored an
> alternative approach, which would have maintained BC. We put the discussion
> to the mailing list, where further people reviewed it, and we came to a
> decision, as a group, without any objections being raised.


That's good and reassuring news; if so, that's precisely another occurrence
of "those reasons really need to be very good, and spelled out and
explained". Most Django users will never see farther than the release
notes, so if abrupt in-places changes occur without a rationale behind, it
*appears* reckless (whetever really happened in the background). If release
notes are considered too long already, wouldn't html tooltips/collapsibles
be useful, so that one may know more about the rational behind changes?

Tech debt is a very real thing.  Keeping it under control is vital to the
> pace of improvement and updates for any project, large, middling or small.
> Any project requires maintanence at minimum, just like owning a car or
> house.  If you buy a car and run it every day for five years, only filling
> up the gas tank and adding a splash of oil now and then,  you don't have
> any excuse to be upset at the mechanic or manufacturer for the size of the
> repair bill when things start breaking.  At least with software, its alot
> easier to get fixes out than with hardware.  You make a persuasive argument
> and have great passion, but the overall goal of enabling out of date and
> unsupported releases to remain around longer than they should be makes me
> feel really uncomfortable.  A spftware project is not buy once and forget,
> neither is a car.  And your package seems to encourage that lax practice to
> a level that I would be highly resistant at working at a company if I
> couldn't persuade management to prioritize keeping up to date and tech debt
> at a manageable level.
>

We agree on the necessity of updating code, and indeed DCP was never meant
to encourage slacking but to solve external dependency conflicts. The
latter - and the additional costs involved in solving them - are precisely
used by management to refrain from upgrading codebases.

As we're getting to the end of useful arguments here, I'd like to ask you
> to step back for a minute and take a calm look at what you're writing.
> Imagine you were receiving such messages rather than sending them. Would
> you want to spend more time collaborating with the person addressing you
> that way and to receive more such messages? Perhaps this is another reason
> why you aren't getting as much support as you'd hope.
>

My initial article, as well as most of this ML discussion, is not small
feature/bugfix request, but a rant on the whole deprecation process. The
aim is to raise awarenesss, to shake minds, to push points as deep as
possible, and to not let any falsehood or diversion slip by. As thus, it
uses all kinds of strong words, of emphases, of rhetorical figures, of
denounciations, to deliver the argumentation. I guess a rant will always
look aggressive, even if it only states the most scientific facts.

So I'm sorry if it sounded insulting, but personally I don't mind facing
adamant insistence, blunt statements, criticisms of any current state of
affairs (whatever the amount of effort one might have put into getting
there), and heated discussions, as long as they don't turn into ad hominem
attacks; in the excepts that you pointed out, I was mainly stating my frank
opinion on the (visibly incomplete) evidence that I was shown, and I would
only have sounded hypocritical by masking the seriousness these issues have
for me. Maybe some feel that Django as already having admirable commitment
regarding API stability, alas some other languages and frameworks have set
the bar much higher (farther than semantic versioning), and *they* are to
be used as references (although too extended compatibility is considered
harmful by some here).

Surely some of my words could have been better chosen ("punishing" shall be
understood as "putting at a disadvantage", and "little version" as "minor
version"), my bad, like some of yours ("throwing your toys" sounds
condescending, though I couldn't care less); but I feel that politeness
euphemisms and tone policing destroy the productiveness of crucial
controversies like this one, instead of helping them. I'd rather be called
names, but not see already-debunked untruths reappear in the conversation
("current behavior [...] *can't be fixed* without a hard compatibility
break", "*always* take the time to create deprecation paths"...); this
intellectual scrambling is what makes *me* stomach-sick, much more than old
unmaintained packages.

For example, Luke was open to amending the deprecation policy to better
> reflect reality. You should have suggested some concrete changes, erring on
> 

Re: Django Async DEP

[Pascal Chambon]

Hello,

There is something a little scary for me, in changing all the core of
Django to async, when this really helps only, imho, a tiny fraction of
users : websocket/long polling services, and reddit-like sites with
thousands+ hits per second. For most webpages and webservices, async
artillery sounds quite overkill.

Are cpython threads inefficient ? As far as I know they are only kernel
threads constrained by the Gil, so they shouldnt wake up when they are
blocked on io syscalls/mutexes (or do they?), and context switches remain
acceptable compared to the slowness of python itself.

We used to provide provisioning and automatic authentication for 20 million
users, with partner webservices tar-pitting us for sometimes 1mn. The
nightmare scenario. But with 2 machines, 1 process by core, and 800 threads
by process, it did the job, enough for us to answer millions of hits a day.
Without even relying on other no-recoding optimizations like pypy or gevent.

Async would certainly have been a relevant techno if we had known in
advance that our partners would be so slow, but avoiding the extra
complexity burden of this style (where a single buggy dependency can block
all requests in a process, where all modules have to be recoded for it) was
also a huge benefit. And the limited thread pool also protected our DB from
unbearable loads.

It's very nice if a proper async ecosystem emerges in python, but I fear
lots of people are currently jumping into it without a need for such
performance, and at the expense of lots of much more important matters like
robust ess, correctness, compatibility... like it happened for docker and
microservices, transforming into fragile bloatwares simple intranets, which
just needed a single django codebase deployed in a single container.

A few days ago I audited a well used django module, the current user was
stored in a global variable (!!!). People might eventually fix that ticket,
use threadlocals, and then switch to a future django-async without
realizing that the security issue has come back due to the way async works.

Still I hope I'm wrong, that the performance gains will prove worth the
software fragmentation and complexity brought by asyncio, but I still dont
understand them for 99% users... Especially as long as key-in-hand
solutions like greenlets exist for power users.

Regards,
Pascal





Le ven. 7 juin 2019 à 19:41, Andrew Godwin  a écrit :

>
>
> On Fri, Jun 7, 2019 at 9:19 AM John Obelenus  wrote:
>
>> I wonder about the end-result payoff of this approach. In general,
>> Django/Python code is not going to be I/O bound, which is where
>> asynchronous approaches are going to get the bang for your buck. Even when
>> it comes to DB access—the DB is a lot faster than the python and django
>> code running against the result set. And too much context-switching (as you
>> noted) has painful ramifications for performance.
>>
>
> To the contrary, I have found that as you scale up, a large amount of your
> time becomes I/O (either HTTP calls to other components/hosted serviecs or
> database calls). Our APM at work shows me that it's around 80% of request
> time.
>
> Obviously we don't design Django just for large use cases, which is why
> it's not going to be the default, but with the massive growth of hosted
> services, I suspect this trend will continue to trickle down to smaller
> deploys too. And ultimately, for smaller deploys performance is rarely a
> concern anyway.
>
>
>>
>> I can absolutely see why creating a layer that handles asgi, websockets,
>> and http requests asynchronously is going to pay off. Bit time. But I'm
>> less certain that the ORM access will benefit from an asyncio approach. Do
>> we have anything that approaches a hard number that would tell us re-doing
>> the ORM layer in asyncio would get us X% performance benefit?
>>
>> I'm basing my thoughts off this well-reasoned look at performance:
>> https://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/
>>
>>
> I do not personally have hard numbers that I am allowed to share,
> unfortunately, but I would encourage you to look at results on benchmarks
> that include database access - like this one (
> https://twitter.com/_tomchristie/status/1005001902092967936) using Python
> asyncio/ASGI - and see that it does make a difference. Obviously it doesn't
> matter for all deploys, but I believe it matters for the majority of site
> architectures as they scale up.
>
> Andrew
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/5CVsR9FSqmg/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at 

Re: Presenting DCP, a compatibility layer for Django (feedback welcome)

[Pascal Chambon]

Hi Raphaël ,

the goal of DCP is to "unite" the behaviour of different django versions,
so that all the code needed by a project (misc. pypi dependencies, as well
as custom project modules) may work together, whatever the exact django
version each of these module actually targets. That's closer to your 3).

At the moment, available fixers are all for backwards compatibility, so the
idea is to install the latest django version (with latest security
features), and be able to run all kinds of dependencies with it, even if
they haven't yet been updated. If fixers for forward compatibility got
contributed, people could at the contrary remain on an old django version,
yet begin to use new APIs and features. As far as I've seen, most changes
can be patched in a way which allows both the old and new behaviour to work
at the same time (that's what happens during the PendingDeprecation period
of a change, by the way).

However, note that DCP is mainly aimed at *project maintainers*, eg. a
specific website or saas. When developing a pluggable apps or library, you
can't expect all your users to install and activate DCP in their own
project, so a compatibility toolkit like "django-compat" (
https://github.com/arteria/django-compat) is more relevant - at least as
long as no DCP-like system is built into django.

regards,
Pascal




2017-01-23 13:37 GMT+01:00 Raphaël Barrois <raphael.barr...@gmail.com>:

> Hi Pascal,
>
> I'm unsure as to what problem you're trying to solve; if I understand
> correctly, this project aims at making a "new"
> Django behave like an "old" one.
>
>
> I see a few possible use cases in the conversation:
>
> * Running a legacy application (without new development) beyond the
> upstream support schedule (namely Django)
> * Developing libraries (or reusable apps) that need to support several
> Django versions at once
> * Using libraries that don't support (yet/anymore) your chosen Django
> version
>
> In my opinion, the Django project has no reason to try to support the
> first one: those projects are basically
> requesting longer security support from both Django and a set of
> third-party libraries; this requires time from the
> maintainers of those projects.
> If the project has decided to *not* provided longer-term-support, it is
> often for good reason, mostly "nobody is paying
> developers enough for that additional support to be viable — if developers
> are paid at all".
>
>
> For the second part, as a library developer, I'm indeed interested in a
> compatibility layer that would allow me to use
> the *new* Django APIs in older versions; not the other way around.
> This way, I can benefit immediately from improvements in the Django
> codebase and prepare my code for future upgrades —
> akin to Python's ``from __future__ import ...```.
> However, such a feature would need to provide a *wrapper* around Django's
> APIs and not modify it: other code in
> the project is likely to rely on the documented behaviour.
>
>
> For the third part, I'm unsure how you'd handle various libraries having
> different Django target versions?
>
>
>
>
>
> On Mon, 23 Jan 2017 13:09:35 +0100
> Pascal Chambon <chambon.pas...@gmail.com> wrote:
>
> > Hi Dmitry,
> >
> > thanks for the alternative way, however it seems more a *complement
> *than a
> > replacement of DCP.
> >
> > Indeed, your "rewriting" approach updates a codebase to support the
> latest
> > django API, but it raises a number of issues :
> >
> > - how could it be applied to django reusable apps, installed via pip ?
> Does
> > it mean a package hook should be used to automatically transform code on
> > "pip install" for django-related modules, as setuptools allows it for the
> > 2to3 case ? Or a special import hook for django ?
> > - it doesn't ensure backwards compatibility of the updated module, so
> > unless every django-app maintainers use that system to stick to the
> latest
> > django version, conflicts would remain when handling big projects.
> > - imo it can only handle trivial changes (renames mostly) ; when features
> > disappear, or are replaced by much different approaches, unless this
> > rewritter itself injects big shims, it wouldn't work. For example, the
> > SortedDict, in which keys could be inserted at arbitrary positions, has
> > been replaced by stdlib OrderedDict which is "readonly", so the code
> around
> > needs to be rethought accordingly.
> >
> > That being said, I guess everyone would love it, if they could upgrade
> > THEIR codebase semi-automatically, instead of doing mass regex
> > search/replaces.
> > There are plenty of A

Re: Presenting DCP, a compatibility layer for Django (feedback welcome)

[Pascal Chambon]

Hi Dmitry,

thanks for the alternative way, however it seems more a *complement *than a
replacement of DCP.

Indeed, your "rewriting" approach updates a codebase to support the latest
django API, but it raises a number of issues :

- how could it be applied to django reusable apps, installed via pip ? Does
it mean a package hook should be used to automatically transform code on
"pip install" for django-related modules, as setuptools allows it for the
2to3 case ? Or a special import hook for django ?
- it doesn't ensure backwards compatibility of the updated module, so
unless every django-app maintainers use that system to stick to the latest
django version, conflicts would remain when handling big projects.
- imo it can only handle trivial changes (renames mostly) ; when features
disappear, or are replaced by much different approaches, unless this
rewritter itself injects big shims, it wouldn't work. For example, the
SortedDict, in which keys could be inserted at arbitrary positions, has
been replaced by stdlib OrderedDict which is "readonly", so the code around
needs to be rethought accordingly.

That being said, I guess everyone would love it, if they could upgrade
THEIR codebase semi-automatically, instead of doing mass regex
search/replaces.
There are plenty of AST modifiers for python (
https://github.com/berkerpeksag/astor, or others, pytest does some nifty
ast hacking too...), else a regex based django-command would already be
nice too.

regards,
Pascal Chambon





2017-01-22 22:38 GMT+01:00 Dmitry Gladkov <dmitry.glad...@gmail.com>:

> Hi,
>
> Making Django upgrades less painful is a great goal, but I believe the
> patching Django and restoring removed functionality is not the right
> solution. JavaScript world has the same problem with far more frequent
> major compatibility breakages and they solve it with automatic codebase
> upgrade tools like jscodeshift [1]. This approach uses AST transformation
> similar to what 2to3 does, but with configurable transformation rules.
> There's also another project written in Python [2] that implements simpler
> and more general, but less reliable regex-based approach.
>
> I believe adding codebase upgrade rules for each major Django release and
> giving users ability to apply them by running something like  ./manage.py
> upgrade_from 1.7 after Django upgrade will greatly simplify upgrading of
> large Django projects.
>
> [1] https://github.com/facebook/jscodeshift
> [2] https://github.com/facebook/codemod
>
> --
> Best wishes,
> Dmitry Gladkov
>
> On 22 January 2017 at 20:53, Pascal Chambon <chambon.pas...@gmail.com>
> wrote:
>
>> Hi,
>>
>> @James Pc - thanks for the support, if you happen to miss some fixers in
>> DCP and don't have the opportunity to contribute them, please open an issue
>> so that I have a look
>>
>> @Tim Graham & James James Bennett - from what I sum up, the new policy
>> simply extends the delay between breaking changes, and so the overall life
>> expectancy of django reusable apps, but other limitations remain as is.
>> I've detailed the misc benefits that a compatibility layer would bring, I
>> just hope not too many people needing big compatibility will fail to learn
>> about django-compat or DCP, and complicate their life with handmade shims
>> or useless forks.
>>
>> regards,
>> Pascal
>>
>> 2017-01-16 10:48 GMT+01:00 James Bennett <ubernost...@gmail.com>:
>>
>>> On Sun, Jan 15, 2017 at 1:09 PM, Pkl <chambon.pas...@gmail.com> wrote:
>>>
>>>> My bad, if people are guaranteed 2 x 24-month cycles before a feature
>>>> gets removed, it's already much better. However, the same pattern as
>>>> previously appears in docs : "each feature release will continue to have a
>>>> few documented backwards incompatibilities where a deprecation path isn’t
>>>> possible or not worth the cost.". I might be paranoid, but I foresee lots
>>>> of dependency breakages in the future, if incompatibilities continue to be
>>>> introduced at developer's will History proved even seemingly harmless
>>>> django modifications (ex. import aliases removed) broke external code,
>>>> sometimes forcing "commit reverts" in django code.
>>>>
>>>
>>> Just to clarify, the future plan -- beginning with Django 2.0, the next
>>> release after 1.11, which is about to feature-freeze -- is:
>>>
>>> 1. Releases go in a cycle of X.0, X.1, X.2, (X+1).0. So, for example,
>>> 2.0, then 2.1, then 2.2, then 3.0.
>>> 2. Each X.2 is an LTS.
>>> 3. Code which runs with no deprecation warnings on an LT

Re: Presenting DCP, a compatibility layer for Django (feedback welcome)

[Pascal Chambon]

Hi,

@James Pc - thanks for the support, if you happen to miss some fixers in
DCP and don't have the opportunity to contribute them, please open an issue
so that I have a look

@Tim Graham & James James Bennett - from what I sum up, the new policy
simply extends the delay between breaking changes, and so the overall life
expectancy of django reusable apps, but other limitations remain as is.
I've detailed the misc benefits that a compatibility layer would bring, I
just hope not too many people needing big compatibility will fail to learn
about django-compat or DCP, and complicate their life with handmade shims
or useless forks.

regards,
Pascal

2017-01-16 10:48 GMT+01:00 James Bennett :

> On Sun, Jan 15, 2017 at 1:09 PM, Pkl  wrote:
>
>> My bad, if people are guaranteed 2 x 24-month cycles before a feature
>> gets removed, it's already much better. However, the same pattern as
>> previously appears in docs : "each feature release will continue to have a
>> few documented backwards incompatibilities where a deprecation path isn’t
>> possible or not worth the cost.". I might be paranoid, but I foresee lots
>> of dependency breakages in the future, if incompatibilities continue to be
>> introduced at developer's will History proved even seemingly harmless
>> django modifications (ex. import aliases removed) broke external code,
>> sometimes forcing "commit reverts" in django code.
>>
>
> Just to clarify, the future plan -- beginning with Django 2.0, the next
> release after 1.11, which is about to feature-freeze -- is:
>
> 1. Releases go in a cycle of X.0, X.1, X.2, (X+1).0. So, for example, 2.0,
> then 2.1, then 2.2, then 3.0.
> 2. Each X.2 is an LTS.
> 3. Code which runs with no deprecation warnings on an LTS will also run
> (though may raise deprecation warnings) on the next LTS.
> 4. Thus, any time you run on an LTS with no deprecation warnings, you know
> you're safe to upgrade to the next LTS. And once you clear any new
> deprecation warnings on the new LTS, you know you're clear to upgrade to
> the one after that.
>
> Regarding backwards-incompatible changes in general: they do happen, but
> they also follow the guidelines in the API stability document. When they
> occur, it's because there's a security issue or larger bug being solved.
> Additionally, many of the seemingly-large list of such changes each release
> are, if you dig into them, changes to private/internal or at least
> undocumented APIs not covered by the stability policy, but we've learned
> people still rely on those APIs and so we document changes to them even if
> we don't guarantee stability in them.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/django-developers/dKeLB-qR7lY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/CAL13Cg8_0BRpu1b_zz2Dx_
> YXcLaXGXw8Qw0jfA_uDhaxt%3DD_%3Dw%40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CA%2BXNUS35Au2um1an3bV-P8q%3DivfOM067r3Qay4jntd%2BCHADkPg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Presenting DCP, a compatibility layer for Django (feedback welcome)

[Pascal Chambon]

broken every two LTS releases, i.e every 16
>> months. For a big software ecosystem, this is still a really short
>> lifespan. And since not every developers cares about deprecation warnings
>> (when they are visible), it means any pypi package for django which hasn't
>> been updated for 6 or 12 months would quite likely be broken. This means
>> lots of forks without any added value, without evolutions nor bugfixes,
>> just "so that it keeps working" .
>>
>> Many major languages or frameworks have strong commitments on
>> retrocompatibility, and have succeeded on this commitment for years
>> (python2 being one of them) ; I remember a blogpost where a Qt developer
>> explained at great lengths how they carefully crafted every function
>> signature, so that it could evolve without breaking older code. If they can
>> achieve this C++, how comes major python frameworks would do less ?
>>
>> My opinion is that the "walk or die" approach of Django is harmful to its
>> ecosystem, and furthermore unnecessary, since a layer like DCP can ensure
>> BOTH a wart-free & fast-moving codebase, and a strong retrocompatibility.
>>
>> About the current tests of deprecation warnings, to me this is a wrong
>> "separation of concerns". Checking that a web framework feature works, and
>> checking that it properly triggers specific warnings, are aspects which
>> should not be mixed with each other.
>> Why ? Because it prevents advanced uses of test suites. For example, how
>> do you check that Django version X.Y hasn't broken code written for
>> previous releases ? Just because it passes its own test suite ? Just
>> because "committers have been careful" ? I don't think it says much. What
>> I'd do, is rather run the unit-tests of the previous django version
>> (excluding some specifically marked "short-lived" tests), against the
>> newest code. This would mean so much more.
>>
>> Here is my view on why using a compatibility layer in Django (or any
>> framework, actually) would be a good idea :
>>
>> PROs:
>> - cleaner django codebase
>> - less work to deprecate features
>> - automated checks of compatibility between django version
>> - easily achieve long-term compatibility
>> - more control given to django users regarding compatibility choices
>>
>> CONs:
>> - small additional delay at django startup ?
>>
>> Am I missing something there ?
>>
>> regards,
>> Pascal Chambon
>>
>> Le mardi 10 janvier 2017 01:29:41 UTC+1, Tim Graham a écrit :
>>>
>>> Our new deprecation policy was designed to eliminate the need for this
>>> type of compatibility library. If you want to support more Django versions
>>> than what our guidelines recommend (which may involve supporting versions
>>> that no longer receive security updates), I guess this library could be
>>> useful, but I wouldn't give it an endorsement in the Django docs.
>>>
>>> Perhaps you could elaborate on "Most importantly, this way, django main
>>> tests suites  would stop making asserts on the presence or absence of
>>> deprecation warnings (which is a way of mixing unrelated aspects of the
>>> framework)." I don't understand your complaint about the current way we're
>>> testing deprecation warnings.
>>>
>>> On Saturday, January 7, 2017 at 3:17:06 PM UTC-5, Pkl wrote:
>>>>
>>>>
>>>> Hello,
>>>>
>>>> I've *at last* had some time to experiment with the idea I had thrown
>>>> years again, of a system to workaround the recurring breakages introduced
>>>> by new django releases, breakages which sometimes lead to unreconcilable
>>>> requirements between django apps.
>>>>
>>>> After a sprint at PyconFR, with a colleague (R. Gomès), and some
>>>> cleanups and improvements, here it is:
>>>>
>>>> https://github.com/pakal/django-compat-patcher  (also known as DCP)
>>>> https://pypi.python.org/pypi/django-compat-patcher
>>>>
>>>> The philosophy of the module is that, with extremly agile languages
>>>> such as python, one doesn't have to choose between a fast-moving project,
>>>> and a compatibility-friendly project.
>>>> So, in an approach reminding "aspect oriented programming", DCP
>>>> separates the Django codebase, which may move forward and be left mostly
>>>> free of compatibility shims, from retrocompatibility shims ; and changing
>>>>

Re: Fate of sql* management commands

[Pascal Chambon]

Hello,

sorry for the delay (I'm moving home),
thanks for the detailed explanations, I've learned a lot about the new
status of migrations, and I guess every django users might benefit from
this "shift of canonical representation", that's why there shouldbe room
for a proper introduction in django docs, explaining all that and
especially discarding the belief (which I guess is true, for example, in
Sqlalchemy) that ORM classes ARE the source of all other schema
representations.

I should draft something like that, but before that, we should advance on
the points raised previously.

Migrations are a very complex subject (to me at least), and there is a risk
of incoherence between the SQL DB schema (as altered by migrations), and
what the ORM sees via ORM classes. So having public debugging tools, as
proposed by Marc and Aymeric, would be very appreciable. It could be:
- an SQL dump of the structure of ORM classes (we would just have to
explain that it's an INCOMPLETE view of what might be in real DB, due to
RUNSQL/RUNPYTHON and the like) ; a separate command would be better I
think, so that people understand it's only "hints".
- an autocheck, which ensures for example that at least simple selects() on
all known columns fields don't crash, and if possible a column-type check
too ; I added somethings like this to an SQLAlchemy-based soft, dunno how
hard it'd be in Django, but it saved my life on numerous occasions.

Do these propositions sound acceptable ?

regards,
Pascal




2015-04-02 23:03 GMT+02:00 Marc Tamlyn :

> As far as I'm aware, we have some code paths which still work in 1.9 which
> generate tables directly from the models. We use this when running Django's
> own test suite, and it is also used now by djangobench. I haven't looked
> into exactly how to turn this into logged SQL rather than run SQL but it
> should be possible.
>
> I think we should document such a tool as a debugging tool rather than a
> development tool, but it would be useful nonetheless. Something like
> `sqlmigrate --from-clean` might be appropriate.
>
> Marc
> On 2 Apr 2015 21:50, "Aymeric Augustin" <
> aymeric.augus...@polytechnique.org> wrote:
>
>> On 30 mars 2015, at 23:10, Carl Meyer  wrote:
>>
>> > So it is not true that the Python models are the canonical
>> > representation of your schema, and the SQL DDL is just a translation of
>> > them. In fact, your migrations are the (only) canonical representation
>> > of your schema, which may include things (specialized indices, views,
>> > SQL functions, triggers...) that don't have any direct representation in
>> > the Python models at all. The Python models just need to be in-sync
>> > enough to generate the correct SQL for queries at run-time, but they
>> > often will not fully represent your schema.
>>
>> While I understand this argument in theory, I'm not sure we can stop
>> there in
>> practice. I know that anecdote != data but here's my story and what I
>> think it
>> means for Django.
>>
>> I started a moderately complex Django-based e-commerce project about one
>> year
>> ago. We switched to Django 1.7 right after it was released and went live
>> a few
>> days later. We made the most basic use of migrations: add a model here,
>> run
>> makemigrations, commit; change a field there, run makemigrations, commit.
>>
>> We did only one complicated thing. We moved our custom user model because
>> of a
>> circular import problem. (Pro-tip #1: put your custom user model in an app
>> that doesn't depend on anything else. Perhaps we should document that.) We
>> papered over the resulting mess by squashing migrations and life was good.
>> (Pro-tip #2: if you change AUTH_USER_MODEL, throw your migrations history
>> away
>> and regenerate it. It's easy. We should definitely document how to do
>> that.)
>>
>> A few weeks ago, we ran into issues I can't describe exactly when
>> generating
>> or applying new migrations. I'm sorry, I don't remember the details
>> because
>> I'm not monitoring development very closely. Judging by how much time had
>> been
>> spent on the issue and how messy it looked, I decided to simply scratch
>> our
>> migrations history, which we should have done earlier. (See pro-tip #2.)
>>
>> I wanted to check that the resulting schema matched what we had in
>> production
>> in order to make sure that I wouldn't introduce inconsistencies. On my
>> machine, I emptied all migrations folders, ran makemigrations, created a
>> fresh
>> database, ran migrate, and dumped the schema. Then I grabbed a dump of the
>> production schema. The dumps were about 11 000 lines long. The diff
>> produced
>> by apgdiff -- a great tool -- was 2 000 lines long.
>>
>> Much of the diff was noise created by non-deterministic index and
>> constraint
>> names. I'm almost sure we fixed that in 1.8. I hacked a script to
>> renormalize
>> hashes and was still left with significant differences. Oops.
>>
>> These differences happen when Django produces 

Re: Fate of sql* management commands

[Pascal Chambon]

Thanks for your answers,

l'm confused nonetheless, because their are two notions mixing up, here,
that we should probably separate:

- the HISTORY of SQL schemas, aka "django/south migrations"
- the CURRENT STATE of SQL schemas, that I'll call "ORM Models Dump"

Let's leave the SQL data (and its migrations) out of the equation please,
here i'm only caring about the structure of data, the schemas, and their
corresponding SQL DDL.

I completely understand and agree that migrations are the way to go to
upgrade between app version, not only because it allows adding/removing
columns (which syncdb didn't do, previously), but also doing all sorts of
on-the-fly custom conversions.

However, what I am seeking, is not a dump of successive migrations (even
aggregated), it's an "ORM models dump", i.e a "transliteration" of django
ORM classes to their corresponding SQL CREATE statements. Just like a
"mysqldump --no-data", except that it's the logical, theoretical schema
that I want here, not the real, SQL-side one.

I really do not believe that we need all the history of schema changes to
get this schema dump ; AFAIK, it's the django ORM models which define the
logical schema of data, not the migrations ; if a class MyUser defines a
field "my_adress", and migrations have no track of it, it's the migrations
that are wrong/incomplete, not the python code. Both "current models" and
"the sum of all migrations" are supposed to be in sync at any time, arent
they ? Custom SQL  code is thus not a problem, since we don't care about
the path taken, just were we are currently.

So if django was probably able to dump all the DDL for its ORM models in
the past, why couldn't it continue to do it ?

As for the question:
*"why do you want it"? manage.py migration *should* be handling all your
table creation and migration functionality, so the need to manually diff a
schema shouldn't exist*

My answer is : for the same reason that I need all other debug tools
available, from django-debug-toolbar to lowest-level cpython debuggers. ^^

I see tons of ways how stuffs could go wrong, from an accidental
interruption of migrations (DDL instructions often having the flaw of being
non-rollbackable), to bugs in migrations' code, without forgetting
unorthodoxes events which may have happend to a project's conf (eg.
switching from a USER_MODEL to another, and thus having to fix stuffs the
hard way, with "--fake" migrations if necessary), or the handling of apps
that have no support for migrations.

Sure, people encountering troubles might make like I did - start a brand
new project and mysqldump it after whole migration - but if we can expose
that information in a straightforward way, like it was before, it's all
benefits for everyone, isn't it ?

regards,
Pascal







2015-03-30 5:59 GMT+02:00 Andrew Godwin :

> I must also point out that the sqlmigrate command will still get you SQL
> from migrations, though it sadly lacks the ability to take a range of
> migrations, optimise them down, and output the SQL for _that_ - that's
> probably the best thing for us to aim towards, and will almost entirely
> recreate the functionality of sqlall without actually being wrong (sqlall
> doesn't understand things like data migrations or custom SQL sections of
> migrations, and so will sadly be wrong in if you have anything like a
> moderately complex migration set).
>
> The plus side is that it would be possible to do this sort of thing as a
> third-party command/backport if we wanted to have it usable on 1.8 and even
> 1.7 (the 1.8 feature set is frozen now, or I'd suggest quickly squeezing it
> in).
>
> Andrew
>
> On Sun, Mar 29, 2015 at 4:57 PM, Russell Keith-Magee <
> russ...@keith-magee.com> wrote:
>
>>
>> On Mon, Mar 30, 2015 at 1:36 AM, Pkl  wrote:
>>
>>> Hello all,
>>>
>>> I've noticed that all sql* commands that were in django core management
>>> have disappeared from the latest sources.
>>>
>>> However, even though Django now has builtin south-style migrations now,
>>> these *sql** commands (especially sql-all) were priceless to debug
>>> differences between the models and the actual DB schema.
>>> Especially as long as some django apps haven't caught up with all these
>>> recent changes - I myself ended up with missing tables after scrupulously
>>> following upgrade instructions, with "manage.py migrate" saying "all is
>>> fine".
>>>
>>> *What is the new way to dump the sql schema of currently installed
>>> django appz ?* It'd maybe be worth that I provide a doc patch to inform
>>> users about it.
>>> *If there is none, is there an agreement to restore at least sqlall and
>>> sqlclear ?*
>>>
>>
>> No, there isn't :-)
>>
>> Also what was the reasoning behind, on the first, place, blocking access
>>> to these utilities in django1.7 (they seemed to work, when I forced them),
>>> and now removing them ? A simple warning about the new "migrations" stuffs
>>> (or a "--force" argument) would most probably