Hello Luke,

thanks for your comments, mine are below

On Thursday, August 1, 2019 at 10:34:22 AM UTC+2, Luke Plant wrote:
>
> Hi Pascal,
>
> I know this is a very late reply, but there have been some things going 
> round my head that I thought I should get down.
>
> *First,* some positive changes I think we can make from your suggestions:
>
> 1. I think in our release notes, for breaking changes that might need some 
> justification, we should link to the tickets and/or django-devs discussions 
> behind them. Adding full reasoning in the notes themselves would likely 
> bloat them far too much, but a link could be helpful for anything which is 
> not obviously necessary.
>

Yes documenting the rationale behind breaking changes, via a ticket link or 
another medium, would be indeed a nice improvement.
 

> 2. We should change the wording in our API stability docs 
> <https://docs.djangoproject.com/en/stable/misc/api-stability/>, which 
> does indeed initially sound like a promise never to break working code that 
> follows docs:
>
> Django promises API stability and forwards-compatibility since version 1.0.
>
> We do have this line later on:
>
> If, for some reason, an API declared stable must be removed or replaced, 
> it will be declared deprecated but will remain in the API for at least two 
> feature releases. Warnings will be issued when the deprecated method is 
> called.
>
> But we do make changes at a rate faster than that document would make you 
> think.
>
> I suggest that as a contrast to the existing text on that page, we also 
> point out that we are continually improving Django, and have a 
> "(eventually) one way to do it" policy regarding how we plan APIs, which 
> means we will remove old things, while always providing better 
> alternatives. I'm happy to write a patch for this.
>

As you imagine, I'd rather have had a change in actual django policy, 
rather than a doc update to reflect the current status; but it'd be better 
than nothing. However, the second sentence "*If, for some reason, an API 
declared stable must be removed or replaced, it will be declared deprecated 
but will remain in the API for at least two feature releases*" is itself 
still misleading, since at least one recent major breakage violates it 
<https://docs.djangoproject.com/en/2.2/releases/2.2/#admin-actions-are-no-longer-collected-from-base-modeladmin-classes>
 
; *A pronouncement of the core dev team would be needed here**: is this 
behaviour change a procedure error? Or is it really the new policy, to 
allow changes which create immediate silent errors, and prevent third-party 
compatibility shims from getting implemented?*

Having "only one obvious way to day it" is a useful and common commitment 
in Python, but please note the "obvious". You won't have "one way to do 
it", ever. If people want to use sys.stdout.write(), or do a libc-cffi call 
to output text to console, instead of just calling print(), they will do 
so. And compatibility shims don't change anything on this matter: the 
"obvious", "standard" way of doing is just the one mentioned in official 
docs and which doesn't trigger DeprecationWarnings.


 

> *Second,* there were things I wanted to say that might be useful to you 
> (Pascal) in understanding why you are not likely to get very far with your 
> current approach.
>
> Your exaggerated tone is not likely to help you. You talk about Django's 
> compatibility breakages as "*ruining the day of thousands of anonymous 
> plugin/website maintainers all over the world*". When we make these kind 
> of changes, we always do it with very clear and detailed upgrade notes, and 
> almost always with several releases emitting deprecation warnings, as per 
> our policy. And no-one is forced to upgrade immediately. Those who value 
> stability above other things can use LTS, which gives them a *3 year 
> release cadence,* not 8 months, which is far from excessively fast 
> compared to other projects.
>


Imho dependency hell is absolutely unrelated to detailed release notes or 
LTS versions. Dependency hell is just the fact that, when each minor 
version breaks compatibility, the app ecosystem inevitably explodes, 
especially in its outskirts (i.e the least popular packages), into a myriad 
fragments, each targeting specific django versions. 
Your project remains on 1.11 LTS ? Nice, but one of your dependency (and 
its latest security bugfixes) require a new feature of Django 2.0; while 
another dependency is stuck on Django 1.10, either because the maintainer 
vanished, or because he is himself trapped by the requirements of a 
subdependency.

That's why Semantic Versioning was invented: so that developers can 
gradually update their own codebase (since each minor version brings 
improvements and deprecation warnings), while still ensuring all 
dependencies keep working under the same major versions (the project just 
has to install the latest minor version). The concept of "lax semantic 
versioning" sounds like a joke, it destroys the very purpose of this 
versioning scheme; would you put a "laxly waterproof watch" under water? 

You mention Mezzanine, and indeed this CMS, being quite monolithic, should 
not have big troubles with the Django compatibility policy. More generally, 
as long as one sticks to a bunch of big packages (Mezzanine, 
Django-rest-framework, Django-extensions...), things work fine. Dependency 
hell occurs when you use modular architectures, and want to cherry pick the 
most relevant modules for your needs, even those who haven not invested 
into custom shims and multi-version testing, or which have not received 
updates for a few years. 
Such modular architectures are precious, they ensure that users don't get 
trapped by feature choices of underlying frameworks; but they need this 
extra bit of carefulness about softare compatibility.

Take for example Django-CMS, a dozen external content type plugins 
(ckeditor, restructuredtext, videos, musics, image galleries...), a blog 
engine and its bridge (Zinnia), some utilities (easy-thumbnails, 
django-filer...); and there you are, struggling to make it all work despite 
large-impact breakages of the Django API. All this, not to do rocket 
science, just to build what we would call a simple website.

Let's note that Mezzanine had to fork its own dependencies, by the way, and 
in big part for what reason?

*"At the time of grappelli_safe's creation, Grappelli was incorrectly 
packaged on PyPI, and had also dropped compatibility with Django 1.1 -- 
grappelli_safe was therefore created to address these specific issuesAt the 
time of filebrowser_safe's creation, FileBrowser was incorrectly packaged 
on PyPI, and had also dropped compatibility with Django 1.1 -- 
filebrowser_safe was therefore created to address these specific issues."*
Nice, more fragmentation.

Now, I can't talk about percentages. I just *estimate *that thousands of 
projects (a part of the total) are relying on modular frameworks like 
django-cms, or must make lots of small django apps play together. That's 
for these people that days are wasted, even if they don't realize how 
unneeded all this work duplication is. For sure, dependency hell sounds 
like a made-up or exaggerated concept, until you have to deal with it; to 
take a harsh metaphor, people of Europe see hippos as peaceful giants, but 
that's not the opinion of the 3000+ people they kill each year in Africa.

If there is an *exageration *somewhere, I bet it's rather on all that 
"*backwards 
compatibility is too hard/costly/dangerous*" speech I'm being served. My 
experiments at the contrary showed that backwards compatibility was easy to 
setup and maintain, provided you have some tooling under the hand (tooling 
that I did implement, like import aliasers). Why be worried about "*Django 
codebase (becoming) a pile of backwards compatibility hacks and cruft*", 
when compat-patchers precisely separate the compatibility concern from the 
rest? 
As for the fear of "non local reasoning" expressed in other posts, I'm 
sorry, this doesn't apply here; compatibility fixers don't do evil things, 
like renaming some objects and removing others. They only ADD shims, they 
are activated on demand, they output logs and warnings to keep users 
informed, and if a shim breaks, you have maybe 99% chances the faulty 
wrapper/utility will appear in the traceback. So there is no backstabbing 
here.

By the way, some points I should have added to my initial article:

   - The most widely breaking changes are, generally, the easiest to fix. 
   Moving a core function/module will break every app, and can be fixed in <10 
   lines of code (tests included). Changing the corner-case behaviour of an 
   ORM query method is hard to maintain, but very few users should be impacted 
   by this change. That's why deprecation periods should be set on a 
   case-by-case basis, and maybe 90% of breakages can be avoided just with 
   some dozens of tiny long-term shims.
   - If the framework code evolves too fast for its own maintainers to cope 
   with compatibility shims, then it probably also evolves too fast for plugin 
   and project maintainers to keep track, too. In this case, it should be 
   called alpha-stage software, and people should wait for it to stabilize.


Thats why I don't buy your "*Quite quickly you get to a point where the 
quantity of cruft, and the need to support all the different legacy ways, 
means that adding new things becomes very difficult, and development would 
have stagnated and quality dropped*". When implementing DCP, I didn't 
experience this nightmare you describe. Shims can be dropped in rare cases, 
when they conflict with new features or introduce security holes, and 
deprecated features don't have to be supported on bugtrackers, so no hands 
are tied.

 

>  *Instead, my code would be just as bad/quirky today as was did back then*, 
> for two reasons:
>
> 1. Without being forced to upgrade, I probably wouldn't have done.
>

I have trouble following you here. Why do so many developers format their 
code, lint it (with pylint, xenon, doc8, mypy...), test it, package it, 
tox-integrate it, document it, internationalize it? Are they compelled to 
do so? Will python refuse to run if one of these best practices is not 
respected? No. 
Developers just keep a codebase clean because they know it's the best way 
to keep it maintainable, and to attract users and contributors (+ the 
happiness feeling of properly done work). 
Fixing (deprecation) warnings is only one of these best practices. Why 
would people skip *this *particular cleanup task, and ignore deprecation 
warnings but not other alerts or test failures? Especially if, as you 
mention, upgrading a single codebase to comply with new django standards is 
quite straightforward.
Note that if the new way of doing things seems less intelligent to your 
users than the old way (no additional clarity, no new features, no new 
bugfixes), there is probably a problem with the development process, and 
twisting user's arms by quickly removing old ways of doing is not a 
solution.
Backwards compatibility shims are not meant as a laziness appeal for active 
projects; but as a solution to dependency hell for modulare projects, and 
as a last recourse for low-maintainance plugins; it avoids the explosion of 
unmergeable forks while a project waits for a new maintainer. 

By the way, your old code is not pretty, but it's understandable; and if 
your project is unmaintained, I'd rather have it not pretty BUT working, 
than not pretty and plain BROKEN.


There are many improvements, for example:
>
>    - Star imports have been removed. There are lots of good reasons why 
>    every Python code linter complains about these. Or would you rather still 
>    have them?
>    - In my editor I can put my cursor on a view function like `index`, do 
>    "jump to definition" and it does the right thing, because it is normal 
>    Python. Do you think dotted strings were better?
>    - My list of URLs is just a list, and can be manipulated with normal 
>    list methods and functions (which I often need to do). Do you think 
>    wrapping everything in `patterns()` would be an improvement? For the sake 
>    of an extensibility we might just use someday (though we didn't in well 
>    over a decade)? 
>    - The addition of `path` is a great help, and has really cleaned up 
>    URL handling a ton, and even more in other areas of my project. I wasn't 
>    forced to make this change, but it's a great improvement. *However, 
>    this is the kind of big addition that the Django developers only 
> considered 
>    and were able to do because the Django codebase was not a pile of 
> backwards 
>    compatibility hacks and cruft already. *And yes, doing so meant we 
>    lost some *undocumented* classes which were implementation details, 
>    which you also seem to be complaining about.
>    
> To clarify, I'm not questioning the interest of individual django changes, 
just asserting that leaving the "old ways" around for a much longer period 
would have harmed no one and helped a lot. (btw, string references are 
still everywhere in Django, and a decent IDE like Pycharm will easily bring 
you from INSTALLED_APPS, MIDDLEWARES or template_name to the proper 
python/html file).

Regarding your last points, I think we all agree that Wordpress is a 
dumpster fire of security holes (with a jaw-dropping database schema 
furthermore), and that AngularJS was an epic failure: an experimental toy 
which was advertised as a next-gen production-ready framework, whereas it 
just didn't scale - and I fell for it too (but at that time, alternatives 
like ReactJS didn't feel so future-proof to me either). 
Django avoided these big traps, and it's very nice. This doesn't impact the 
separate topic of backwards compatibility, in which Django could shine by 
just applying minor cultural changes imho.

*To sum up,* I know that I'm not likely to change the local mindset, on 
this "phobia of shims" as I dare calling it. Personally, I already solved 
my recurring problems, thanks to django-compat-patchers. I'd just like *not 
to get spokes in the wheels*, with non-shimmable changes, hence the 
pronuncement asked above. *Can we all agree at least on this (and 
correcting the documentation as you nicely proposed)?*

It'll still pain me to see so many old but nice plugins mass-forked for 
nothing, when 2 lines of project-level code would have fixed everything; 
but in software world, choosing the longest work path is a recurring 
problem (remember all these people who reimplement half their project in broken 
asyncio <https://veriny.tf/asyncio-a-dumpster-fire-of-bad-design/>, when 
greenthreads would have solved their performance troubles for free until 
further changes are needed, or Trio/Curio 
<https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/>
 
would have at least paved a robust future?)

regards,
Pascal







-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a633762a-81bb-433a-aa7f-37734632c7b4%40googlegroups.com.

Reply via email to