Re: Django LTS support time

[Pkl]

Hello Uri,

my (unsurprising) two cents: this is less a problem with the time span of 
the LTS version, than of the API stability / upgrade ease / django 
compatibility (recently discussed in this thread 
).

In the current state of affairs, I advise you to upgrade your application 
to latest stable Django version, using a compatibility helper like my 
https://github.com/pakal/django-compat-patcher if you have compatibility 
conflicts with your codebase or your dependencies (if some compatibility 
fixers are missing for your own needs, please open an issue or contact me).

best of luck,

regards,
Pascal Chambon




On Friday, August 9, 2019 at 9:31:25 AM UTC+2, Uri wrote:
>
> Django Developers,
>
> I would like to know why Django LTS support time is 3 years. I think with 
> Ubuntu LTS it's at least 5 years. We are using Django 1.11 for Speedy Net, 
> which was released on April 2017, and I'm not sure we will be able to 
> upgrade it before April 2020. There are changes which are not backward 
> compatible, we will have to change some of our code if we upgrade Django, 
> and some packages we are using, such as Django Crispy Forms, we have to use 
> versions which don't support Django versions higher than 1.11 due to bugs 
> in future releases (
> https://github.com/django-crispy-forms/django-crispy-forms/issues/889). 
> Upgrading to the next LTS version (2.2) will take lots of time, and the end 
> of support of 2.1 is December 2019. I want to release Speedy Net and Speedy 
> Match to production soon, and I don't think we will be able to upgrade 
> Django before 2020. It would help if there was 5 years support for each 
> Django LTS version. What do you think?
>
> אורי
> u...@speedy.net 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3a17a325-0433-45dd-ae7f-2db3b3689b8f%40googlegroups.com.

Re: Discussing improvements of django.setup()

[Pkl]

Hello,

any news about this issue?

Concerning the thread-safety of setup(), the PR is here - 
https://github.com/django/django/pull/11440

Concerning the tweakability of setup(), I don't get your argument Aymeric: 
"*One 
more or one less monkey patch isn't going to make a difference to these 
projects*".
That's precisely why we need a hook for custom setup operations. To 
monkey-patch test environments, apply DCP fixers, setup gevent-like 
runtimes... for now, there is no single point where one can inject early 
customization. One needs to change uwsgi.py, maybe manage.py, test 
launchers, potential scripts... 
AppConfig.ready() was introduced so that applications have a single point 
to initialize signals and the likes, this proposal only extends this to 
support early-stage setup customization, which is currently missing. 
Wouldn't an environment variable, pointing to a custom setup function, do 
the job?

thanks,
regards,
Pascal


-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c2bf23f2-ae22-487c-92b9-038e0bcb86cc%40googlegroups.com.

Re: Ideas for a new DEP on long-term Django API compatibility

[Pkl]

Hello Luke,

thanks for your comments, mine are below

On Thursday, August 1, 2019 at 10:34:22 AM UTC+2, Luke Plant wrote:
>
> Hi Pascal,
>
> I know this is a very late reply, but there have been some things going 
> round my head that I thought I should get down.
>
> *First,* some positive changes I think we can make from your suggestions:
>
> 1. I think in our release notes, for breaking changes that might need some 
> justification, we should link to the tickets and/or django-devs discussions 
> behind them. Adding full reasoning in the notes themselves would likely 
> bloat them far too much, but a link could be helpful for anything which is 
> not obviously necessary.
>

Yes documenting the rationale behind breaking changes, via a ticket link or 
another medium, would be indeed a nice improvement.
 

> 2. We should change the wording in our API stability docs 
> , which 
> does indeed initially sound like a promise never to break working code that 
> follows docs:
>
> Django promises API stability and forwards-compatibility since version 1.0.
>
> We do have this line later on:
>
> If, for some reason, an API declared stable must be removed or replaced, 
> it will be declared deprecated but will remain in the API for at least two 
> feature releases. Warnings will be issued when the deprecated method is 
> called.
>
> But we do make changes at a rate faster than that document would make you 
> think.
>
> I suggest that as a contrast to the existing text on that page, we also 
> point out that we are continually improving Django, and have a 
> "(eventually) one way to do it" policy regarding how we plan APIs, which 
> means we will remove old things, while always providing better 
> alternatives. I'm happy to write a patch for this.
>

As you imagine, I'd rather have had a change in actual django policy, 
rather than a doc update to reflect the current status; but it'd be better 
than nothing. However, the second sentence "*If, for some reason, an API 
declared stable must be removed or replaced, it will be declared deprecated 
but will remain in the API for at least two feature releases*" is itself 
still misleading, since at least one recent major breakage violates it 

 
; *A pronouncement of the core dev team would be needed here**: is this 
behaviour change a procedure error? Or is it really the new policy, to 
allow changes which create immediate silent errors, and prevent third-party 
compatibility shims from getting implemented?*

Having "only one obvious way to day it" is a useful and common commitment 
in Python, but please note the "obvious". You won't have "one way to do 
it", ever. If people want to use sys.stdout.write(), or do a libc-cffi call 
to output text to console, instead of just calling print(), they will do 
so. And compatibility shims don't change anything on this matter: the 
"obvious", "standard" way of doing is just the one mentioned in official 
docs and which doesn't trigger DeprecationWarnings.


 

> *Second,* there were things I wanted to say that might be useful to you 
> (Pascal) in understanding why you are not likely to get very far with your 
> current approach.
>
> Your exaggerated tone is not likely to help you. You talk about Django's 
> compatibility breakages as "*ruining the day of thousands of anonymous 
> plugin/website maintainers all over the world*". When we make these kind 
> of changes, we always do it with very clear and detailed upgrade notes, and 
> almost always with several releases emitting deprecation warnings, as per 
> our policy. And no-one is forced to upgrade immediately. Those who value 
> stability above other things can use LTS, which gives them a *3 year 
> release cadence,* not 8 months, which is far from excessively fast 
> compared to other projects.
>


Imho dependency hell is absolutely unrelated to detailed release notes or 
LTS versions. Dependency hell is just the fact that, when each minor 
version breaks compatibility, the app ecosystem inevitably explodes, 
especially in its outskirts (i.e the least popular packages), into a myriad 
fragments, each targeting specific django versions. 
Your project remains on 1.11 LTS ? Nice, but one of your dependency (and 
its latest security bugfixes) require a new feature of Django 2.0; while 
another dependency is stuck on Django 1.10, either because the maintainer 
vanished, or because he is himself trapped by the requirements of a 
subdependency.

That's why Semantic Versioning was invented: so that developers can 
gradually update their own codebase (since each minor version brings 
improvements and deprecation warnings), while still ensuring all 
dependencies keep working under the same major versions (the project just 
has to install the latest minor version). The concept of "lax semantic 
versioning" 

Re: Ideas for a new DEP on long-term Django API compatibility

[Pkl]

Hello everyone,


(apologies in advance for the long post, but there are lots of aspects to 
dig here, it's not for the pleasure of writing but a necessary to progress 
on the relevance of a DEP)

*"As you can see, there's more than "blinding self-absorption" and "harmful 
psychological bias" here. Your freecodecamp article makes valid points. It 
also misses completely what's hard in maintaining backwards compatibility. 
I wish you hadn't found it useful to insult the work put into managing 
backwards-compatibility over the years and the people who did it."*

I'd rather address that immediately: as you read in my article, "blinding 
self-absorption", "harmful psychological bias" and other strong words were 
all aimed at something very specific, the "feeling of return to purity" 
that a person could have when destroying compatibility shims. I persist and 
sign regarding this one: ruining the day of thousands of anonymous 
plugin/website maintainers all over the world should *always* be made under 
compulsion of heavy constraints, and with regrets; never lightheartedly. 

Now, maybe my attempt at explaining the roots of nowadays' mass breakages 
by a cultural problem are all wrong (my bad), and there are other reasons 
behind the *apparently* reckless compatibility destructions that I witness 
left and right. But what are they? Since it's not technical inability, 
since it's not laziness, since it's not ideology, what makes that (as far 
as I'm concerned) the Django ecosystem fails on that aspect, contrary to 
loads of others "big" frameworks, runtimes, libraries etc? 

If it's "we don't have enough resources", it's all understandable, but at 
least let's make it clear. The weirdest, in the whole story, is the global 
idea that everything is fine, that compatibility is already a "major 
concern" and treated so, that maybe it can be improved a little bit but no 
need to push it. It's not that way I see it. And a problem can't be 
addressed if it's not first recognized as so.

Now, I know that it requires a good amount of diplomacy to criticize a 
behaviour, or here a whole policy, without concerned people feeling 
attacked. I don't have this kind of talent (and probably won't ever), so 
all I can do is be honest with my view on the situation, and bring attempts 
at solutions. It's precisely because Django is a great piece of software 
(with its omnipresent lazy-objects, it's automatic admin and model forms, 
its migration system, its focus on security headers...) that having it 
scarifying itself is unbearable. So I apologize in advance for all the 
criticisms that expand below, but without them there'd be no need for any 
DEP or change in the first place.

I also want to make this clear: I didn't mean that breaking changes were 
never justified, just that they had to be the last option, and thoroughly 
explained. When the AngularJS team, after the years of hype, admitted that 
the whole designed was flawed, reset the whole framework as Angular version 
2, and made great efforts to ease part-by-part migration towards the new 
behaviour, I felt unlucky but not angry. When trivial changes ruined tons 
of pluggable Django apps that did one thing and did it well, it was a 
different matter.

Before answering remarks, I think it's worth summarizing what the "problem" 
is. This is not lightly that I used the words "dependency hell". My 
experience with Django, purely regarding compatibility, has been abysmal 
compared to lots of other ecosystems (in C/C++, PHP, Jquery...) that I have 
crossed, and which didn't have the malleability of Python; I see several 
cumulated reasons explaining that :

- The pace of breakages: an arm's long list of new breaking changes 
introduced every 8 months, dropping at the same time all shims older than 
2-3 years, is imo a too fast breakage pace, even by the Web's standard. 
When changes are mainly aesthetical, compatibility shims had better remain 
for very long periods of time, else the benefit/harm balance is just 
indefensible.
- More importantly, the *scope* of breakages: as long as it's only about 
fixing corner cases of the framework, and only a tiny portion of users is 
impacted (those who abused counter-intuitive and unpythonic behaviours), 
one can understand, and just hope not to be shot by the next upgrade; but 
here, we're dealing with changes that deliberately break about every 
existing application, even those which scrupulously followed official 
tutorials: ForeignKey's "on_delete" argument killed most migration files 
when becoming mandatory, moving url-related primitives left and right was 
as destructive as expected...
- The growing "less batteries included" philosophy. Valueable contrib 
modules were outsourced from the codebase, and other pluggable applications 
(likes CMS apps) follow this trend, putting all valuable features into 
external plugins. This "minimal core" philosophy would work fine if main 
applications counter-balanced it with a more delicate 

Re: Discussing improvements of django.setup()

[Pkl]

hello,

AppConfig could provide hooks, but it'd indeed be already quite late in the 
setup process : settings are already loaded, logging already configured, 
script prefix already set, and just accessing AppConfig could already 
trigger the import of advanced submodules (models etc.), which could break 
the early setup.

I guess an environment variable, or a naming convention, could allow early 
hooks similar to those of Gunicorn for example, and unrelated to installed 
apps.

regards,
Pascal Chambon


On Tuesday, July 2, 2019 at 8:40:04 PM UTC+2, Christian González wrote:
>
>
> Am 01.07.19 um 23:24 schrieb Pkl: 
> > 
> > Regarding django.setup() tweakability, the generic need is simply to 
> > have a hook, somewhere for users to put early init code, regardless of 
> > the way django is launched. Code that needs to execute this early are 
> > mostly (if not all) monkey-patching the environnement, a little like a 
> > LD_PRELOAD logic python-style. 
> > [snip] 
> > There are surely lots of ways to provide such a hook, via new 
> > settings, or new environment variables, or modules following precise 
> > naming conventions that django would systematically try to import... I 
> > have no idea how exactly my approach is rated amongst them. 
> > 
> > As for alternatives to these early setup hooks, I'm alas clueless... 
>
> Why not a setup() method in the Apps' AppConfig, like ready()? This way 
> the AppConfig would be used more. Problem here is: when setup runs, it 
> is loading the settings. And THERE INSTALLED_APPS first time tell Django 
> which apps are installed. So before that point, it's not possible to  
> "import modules from apps following a precise naming convention" etc. Or 
> did you mean "modules" outside of apps like e.g. a 
> "django_hook_setup_foo" python module? (Grr.) 
>
> Maybe here could again help to extract the apps list out of settings, 
> and use a descriptive list that is parsed VERY early in the setup 
> process. Then the hook could be placed (with imports from all apps). 
>
> This would help improving my GDAPS plugins system as well, instead of 
> hacking myself into INSTALLED_APPS within settings.py. 
>
> Just my 2 cents. 
>
> Christian 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7a34d34c-0098-4430-b9af-2bf05244755d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Discussing improvements of django.setup()

[Pkl]

Hello Aymeric,

regarding django.setup() idempotence I continued discussion in the PR.

Regarding django.setup() tweakability, the generic need is simply to have a 
hook, somewhere for users to put early init code, regardless of the way 
django is launched. Code that needs to execute this early are mostly (if 
not all) monkey-patching the environnement, a little like a LD_PRELOAD 
logic python-style.

This can be:
- for (of course) the init of django-compat-patcher
- greening of the environment (e.g. as 
https://github.com/erickponce/django-db-geventpool recommends, for Gevent 
or Eventlet)
- early tweaking of some process aspects likes socket timeouts (they broke 
for me when used along with mutiprocessing, but several years ago)
- possibly setup of specific test scaffolding, for which late mock()-style 
things aren't enough (I have no precise scenario in mind for this one, even 
though I know we did suffer with a Jpype bridge to a java LDAP client, 
which required tons of care to not break, including being initialized 
rather early along with process signal handlers)

Without a hook built into django, users that need such early setup have to 
duplicate their efforts for each way Django can be launched, and they must 
call their patching code in manage.py, in the wsgi file, in their custom 
scripts, and in their specific runners likes pytest-django (this last case 
is quite uneasy to handle at the moment, due to its very early setup of 
Django).

I'm afraid I can't provide much code to help the pondering; simply, without 
builtin hook, the user has to battle to ensure the kind of lines like the 
following are always executed before django.setup(). If Django provides a 
hook, it's so much less to worry about.

from psycogreen.gevent import patch_psycopg
patch_psycopg()


There are surely lots of ways to provide such a hook, via new settings, or 
new environment variables, or modules following precise naming conventions 
that django would systematically try to import... I have no idea how 
exactly my approach is rated amongst them. 

As for alternatives to these early setup hooks, I'm alas clueless...

regards,
Pakal









On Sunday, June 30, 2019 at 10:25:15 AM UTC+2, Aymeric Augustin wrote:
>
> Hello Pakal,
>
> *Making django.setup() idempotent*
>
> In https://code.djangoproject.com/ticket/28752, you say:
>
>
> *> I've been bitten numerous times by the impredictable behaviour of 
> django when django.setup() was called numerous times.> In the old days I 
> had exceptions, now it's mainly subtle breakages of logging configuration.*
>
> This is fuzzy for a bug report. Ideally, you should say:
>
> - What you're doing (minimal reproduction instructions, within supported 
> use of Django)
> - What results you're expecting
> - What results you're getting
>
> Anyway it suggests that the problem is about logging. In this comment 
> <https://github.com/django/django/pull/11440#pullrequestreview-250231882> I 
> said that the only possible bug is that `configure_logging()` isn't 
> idempotent. Your answer 
> <https://github.com/django/django/pull/11440#issuecomment-504116261> confirms 
> that the issue is really about logging. Let's fix configure_logging(), then!
>
> Barring new arguments in favor of the proposed approach, which involves 
> nested locks, I'm -1. Nested locks create a possibility of deadlocks. See 
> my comment on the PR for details.
>
> *Making django.setup() tweakable*
>
> Again, the rationale is fuzzy. I'm seeing two use cases:
>
> - pytest-django
> - django-compat-patchers
>
> One more or one less monkey patch isn't going to make a difference to 
> these projects ;-)
>
> I'm ready to accept that there's a need for projects that aren't big hacks 
> but I need to see it explained clearly and specifically. I'd really like to 
> see a feature request in the form of:
>
> - I want to do X, which is a reasonable thing to do and stays within 
> supported use of Django, except I need to override django.setup()
> - Here's what the code looks like with a workaround (monkey-patching, 
> probably)
> - Here's what the code would look like with the improvement I'm asking for 
> (hopefully much better)
> - Here are the alternatives I considered and why I rejected them
> - Other use cases also include Y and Z
>
> I'm asking this because concrete use cases will put us in a better 
> position to find the best solution.
>
> Best regards,
>
> -- 
> Aymeric.
>
>
>
> On 27 Jun 2019, at 23:26, Pkl > wrote:
>
> Hello everyone,
>
> I'm bringing here, for broader review and discussion, the subject of 
> making django setup more "tweakable":
> https://code.djangoproject.com/ticket/30536
> https://github.com/django/django/pull/11435
>
> There is indeed a need for doi

Ideas for a new DEP on long-term Django API compatibility

[Pkl]

Hello everyone,

I'm planning on writing a PEP on long-term API stability for Django.
Most of the rationale behind this kind of commitment is described here : 
https://www.freecodecamp.org/news/api-stability-is-cheap-and-easy/

The idea is that the django ecosystem has suffered a lot, over the past 
decade, from the numerous breakages happening at each (minor) release. I 
could write pages just about what "outsiders" like me endured regarding 
neverending URL patterns changes.

Lots of pluggable apps, and their own plugins and bridges, have become 
incompatible with each other due to sticking to different django versions, 
or to lack of Tox-like cross-version testing. Some undocumented breaking 
changes (like classes replaced by other ones but missing functionality, or 
laziness introduced without autoloading measures for backwards 
compatibility (RequestContext?)) also hindered misc projects, as seen in 
their bugtrackers.

With *less* work than currently, except small changes in procedures, we can 
revive the majority of existing packages (except python2vs3 troubles), 
currently maintained or not, and make upgrades much smoother, which will 
also improve the overall security of django-backed sites (since users won't 
fear upgrades anymore). We can thus reach the same regard for compatibility 
as most industrial-size ecosystems, without sacrificing the cleanness of 
the codebase, nor its maintainability.

For this, imho the best way is just have to include in Django official 
support for a companion library like 
https://github.com/pakal/django-compat-patcher , with which I've had 
immense success

There are several ways of doing this, and for this more thoughts are 
necessary:

- putting compatibility sets in the repository, as a django.compt submodule 
for example?
- putting them in an external repository, but strongly advising them in 
tutorials and otehr docs, like a prime "official project" of django?

There are probably aspects of the issue that I'm missing, as I'm not into 
django "core" development.
So your feedbacks are more than welcome!

Thanks,
regards,
Pakal

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1d2f1399-3310-4b36-ae74-9581a0695002%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

[Pkl]

This society also has linters and such, so that DB migrations are safer 
regarding deployment rollouts - 
https://medium.com/3yourmind/keeping-django-database-migrations-backward-compatible-727820260dbb


On Saturday, June 29, 2019 at 10:36:11 AM UTC+2, Matthieu Rudelle wrote:
>
> Nice find, It's an interesting way to solve it. We might give it a try.
>
> On Thursday, June 27, 2019 at 11:12:16 PM UTC+2, Pkl wrote:
>>
>> Just for reference, here is a package I crossed recently and which has an 
>> approach for that : https://github.com/3YOURMIND/django-deprecate-fields
>>
>> On Monday, June 24, 2019 at 3:15:04 PM UTC+2, Matthieu Rudelle wrote:
>>>
>>> Hi there, 
>>>
>>> I can't find any previous ticket proposing a solution to this problem so 
>>> here are my findings: 
>>>
>>> **Use case**:
>>> When using continuous delivery several versions of the code can be 
>>> running in parallel on se same DB. Say for instance that release 2.42 is in 
>>> production, 2.43 is about to be rolled out and in this release one field 
>>> (say ''MyModel.my_unused_field'') is not used anymore and was removed. 
>>> Before rolling out 2.43 the DB is migrated and column ''my_unused_field'' 
>>> of ''MyModel'' is removed. This makes 2.42 crash saying that one column is 
>>> not found even though 2.42 does not use the field anywhere in the code.
>>>
>>> **Temporary solution**:
>>> Do not makemigrations until de 2.44 release, but it does not scale well 
>>> with many contributors and CI tools (doing their awesome job of making sure 
>>> migrations and models are in sync) will complain.
>>>
>>> **Proposed solution**:
>>> Have a ''disabled'' param on Field. When activated this field is not 
>>> fetched from the DB but replaced by a hardcoded value. 
>>> In our use case, ''disabled'' is added at the 2.42 release, then when 
>>> 2.43 rolls out and migrates the DB no error is thrown.
>>>
>>> **Refs**:
>>> - django-users discussion: 
>>> https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8
>>> - the same problem discussed in this article, but with a slightly 
>>> different solution: 
>>> https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist
>>>  
>>> (third item in the wishlist)
>>> - Previous django-developpers discussion: 
>>> https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ
>>>
>>> What do you guys think?
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/012ce55e-a01d-49dc-bb95-993987725a5d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Discussing improvements of django.setup()

[Pkl]

Hello everyone,

I'm bringing here, for broader review and discussion, the subject of making 
django setup more "tweakable":
https://code.djangoproject.com/ticket/30536
https://github.com/django/django/pull/11435

There is indeed a need for doing pre/post operations at that crucial moment 
of the framework lifetime, especially to apply compatibility shims 
(greening-related, API stability-related...) and setup test scaffolding, 
stubs etc.

A separate PR has been created to handle threadsafety/idempotence 
(https://github.com/django/django/pull/11440) when multiple actors try to 
setup django concurrently, but a discussion remains on how to potentially 
allow users to specify an alternative setup 

I originally did a POC with a new django setting, but if we want to have 
full control, even settings might not be normally loaded at that stage. So 
maybe, as apollo13 suggested, an environment variable, like 
DJANGO_SETUP_CALLABLE, complementing the DJANGO_SETTINGS_MODULE one?
What are your thoughts about this ?

thanks,
regards,
Pakal

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0206c98c-199f-49f9-9ff7-cba736e71224%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Feature Request][Model, ORM] Disabling a field before removal to support continuous delivery

[Pkl]

Just for reference, here is a package I crossed recently and which has an 
approach for that : https://github.com/3YOURMIND/django-deprecate-fields

On Monday, June 24, 2019 at 3:15:04 PM UTC+2, Matthieu Rudelle wrote:
>
> Hi there, 
>
> I can't find any previous ticket proposing a solution to this problem so 
> here are my findings: 
>
> **Use case**:
> When using continuous delivery several versions of the code can be running 
> in parallel on se same DB. Say for instance that release 2.42 is in 
> production, 2.43 is about to be rolled out and in this release one field 
> (say ''MyModel.my_unused_field'') is not used anymore and was removed. 
> Before rolling out 2.43 the DB is migrated and column ''my_unused_field'' 
> of ''MyModel'' is removed. This makes 2.42 crash saying that one column is 
> not found even though 2.42 does not use the field anywhere in the code.
>
> **Temporary solution**:
> Do not makemigrations until de 2.44 release, but it does not scale well 
> with many contributors and CI tools (doing their awesome job of making sure 
> migrations and models are in sync) will complain.
>
> **Proposed solution**:
> Have a ''disabled'' param on Field. When activated this field is not 
> fetched from the DB but replaced by a hardcoded value. 
> In our use case, ''disabled'' is added at the 2.42 release, then when 2.43 
> rolls out and migrates the DB no error is thrown.
>
> **Refs**:
> - django-users discussion: 
> https://groups.google.com/forum/#!topic/django-users/HY_6rZZ0Kk8
> - the same problem discussed in this article, but with a slightly 
> different solution: 
> https://pankrat.github.io/2015/django-migrations-without-downtimes/#django-wishlist
>  
> (third item in the wishlist)
> - Previous django-developpers discussion: 
> https://groups.google.com/d/msg/django-developers/5ofrxeLT95E/XhppgA_KAQAJ
>
> What do you guys think?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d05e7d82-b7cd-4ced-a23c-d3c9dfa3b826%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django Async DEP

[Pkl]

Hello,

I'm a little late to the party, thanks for the big overview on this complex 
matter.

There are lots of thinks I still don't understand though, for example 
regarding "what it unlocks". What asyncio structures would allow to run 
several DB queries concurrently safely and easily, that can't be achieved 
with some threadpool-like mechanism in nowadays' cod ?

Most importantly, I'm not getting why gevent/eventlet-style solutions are 
systematically being dismissed in favor of asyncio. And the more I 
read/talk about it, the less I understand what all that hype around asyncio 
is. This is a whole new language, which forces people to trashbin half of 
the Python ecosystem, and remake it with similar code but filled with 
incompatible async/await statements. 

For sure, having so obstrusive keywords everywhere can be handy when 
writing highly concurrent code without mutexes. But who writes highly 
concurrent code in webservers ? The very principle of webservers is imho 
precisely to never interfere with other requests, never modify 
process-global structures, and at worst delegate some heavy or concurrent 
tasks to some dedicated executor.

Greenlet-style parallelism would bring long-running requests and high 
parallelism to Django without having to touch the bulk of the code code, 
just the I/O parts (DB connections...) ; and thread-local is already 
greenified by Gevent for exemple, according to docs. Why would it bring 
"much higher risk of race conditions and deadlocks without careful 
programming" ? Preemptive threads are much more dangerous than the implicit 
but deterministic context-switching which occurs when greenlets reach 
socket/disk/sleep operations, and yet race conditions seem to be the least 
of the problems of the huge majority of Django programmers.

I'd love to be wrong, but I have the feeling that with a fraction of the 
work required by django-asyncio, it would be possible to greenify the whole 
of Django, fix lots of current limitations and corner-cases of gevent (lack 
of builtin executor to offload tasks to real threads, lack of support in 
python package X/Y/Z...), and without having to recode any of the "middle" 
parts of existing modules. How can we say "third-party support for this 
style of concurrency is much weaker", whereas through monkey-patching, 
about ANY python module (except those using blocking C extensions) can be 
used in a Gevent project ?

Granted, I have little experience with Geven and Asyncio, but all 
experience feedbacks I've read so far mainly insist on minor limitations of 
greenlets, and on the fact that "people are mainly going with asyncio" (a 
self-fulfilling prophecy?). Considered the dramatic difference in workload 
between the two, I'd really love to understand what killer-features justify 
to go for the "recode everything" solution (or what greenlet limitations 
would be show-stoppers on the long term). 

Thanks in advance everyone for your feedback on this issue.

regards,
Pascal Chambon

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/605d78af-f24a-46eb-9cac-4ad75547b604%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Presenting DCP, a compatibility layer for Django (feedback welcome)

[Pkl]

I agree that reading the whole document gave some hints about the incoming 
troubles, but I guess many people (like me), on first pass, just thought 
"OK that's all I wanted to hear" and went by. Plus, it's a little like 
saying "this dogs doesn't bite", and then later "if the dogs wants to bite 
you, it barks first" - disturbing at least.   ^^

My bad, if people are guaranteed 2 x 24-month cycles before a feature gets 
removed, it's already much better. However, the same pattern as previously 
appears in docs : "each feature release will continue to have a few 
documented backwards incompatibilities where a deprecation path isn’t 
possible or not worth the cost.". I might be paranoid, but I foresee lots 
of dependency breakages in the future, if incompatibilities continue to be 
introduced at developer's will History proved even seemingly harmless 
django modifications (ex. import aliases removed) broke external code, 
sometimes forcing "commit reverts" in django code.

The idea of DCP is not to prevent ANY code removal (in particular when it 
deals with django settings, that any project can always update), but rather 
to allow very, very long-term compatibility, for trivial modifications like 
renames and signature changes ; that being said, even import proxies (white 
magic with meta importers, to keep alive modules like 
django.contrib.comments) seem to work quite well, hassle-free.

Concerning django unit-tests, here is the question : the version  of 
django, how can one prove that it really is backwards compatible with the 
branch/version  (except for a few documented breakages) ? Correct me if 
I'm wrong, but currently I see no automated way to check that, so an 
unwanted regression is always possible ; if "warnings" aspects are removed 
from tests, this becomes possible. 
I dunno which such regression happened by the past, if they did, (this 
would require deep-browsing the Git repo), but it's a point of weakness of 
the backwards compatibility commitment imho, as *any* untested commitment 
is. On a side note, this mixing of concerns prevents me, for example, from 
testing that LatestDjango+DCP is well compatible with previous versions of 
django (since warnings, at least, differ).

regards,
Pascal

Le jeudi 12 janvier 2017 17:20:56 UTC+1, Tim Graham a écrit :
>
> > "*Django promises API stability and forwards-compatibility since 
> version 1.0. In a nutshell, this means that code you develop against a 
> version of Django will continue to work with future releases*." were imho 
> untrue.
>
> Perhaps the API stability statements are too strong and need some 
> clarification, but please don't quote without reading the rest of the 
> document, e.g. "If, for some reason, an API declared stable must be removed 
> or replaced, it will be declared deprecated but will remain in the API for 
> at least two feature releases. Warnings will be issued when the deprecated 
> method is called."
>
> > If I understand correctly, with the new release process, reusable apps 
> and libraries would still be broken every two LTS releases, i.e every 16 
> months.
>
> Yes, they may break if they aren't updated, although LTS releases are 
> every 24 months.
>
> My understanding of your proposal is that Django never removes deprecated 
> features but rather moves them to a "compatibility" library. Removing 
> deprecated features helps simplify things. I believe that prohibiting code 
> to be removed from Django would be a significant overhead in complexity and 
> be a drag on the ability to evolve the framework.
>
> About testing, can you give an example of a problem that wasn't detected 
> by our current approach? I'm unsure what problem your suggestions are 
> trying to solve.
>
> On Thursday, January 12, 2017 at 10:39:54 AM UTC-5, Pkl wrote:
>>
>>
>> Hello,
>>
>> thanks for your feedback,
>>
>> Actually this DCP layer is mainly aimed at backwards compatibility of 
>> dependencies : running, on the latest version of django (with maximal 
>> security bugfixes), older reusable apps which are incompatible with 
>> each-other, or with said latest version of Django.
>>
>> When I hear "*lax semantic versioning*", it makes me think about an 
>> "almost alive parrot" or a "mostly non-lethal drink", i.e not something 
>> especially reassuring ; especially when I remember how the sentences 
>> "*Django 
>> promises API stability and forwards-compatibility since version 1.0. In a 
>> nutshell, this means that code you develop against a version of Django will 
>> continue to work with future releases*." were imho untrue. 
>>
>> If I understand correctly, with the new release p

Re: Presenting DCP, a compatibility layer for Django (feedback welcome)

[Pkl]

Hello,

thanks for your feedback,

Actually this DCP layer is mainly aimed at backwards compatibility of 
dependencies : running, on the latest version of django (with maximal 
security bugfixes), older reusable apps which are incompatible with 
each-other, or with said latest version of Django.

When I hear "*lax semantic versioning*", it makes me think about an "almost 
alive parrot" or a "mostly non-lethal drink", i.e not something especially 
reassuring ; especially when I remember how the sentences "*Django promises 
API stability and forwards-compatibility since version 1.0. In a nutshell, 
this means that code you develop against a version of Django will continue 
to work with future releases*." were imho untrue. 

If I understand correctly, with the new release process, reusable apps and 
libraries would still be broken every two LTS releases, i.e every 16 
months. For a big software ecosystem, this is still a really short 
lifespan. And since not every developers cares about deprecation warnings 
(when they are visible), it means any pypi package for django which hasn't 
been updated for 6 or 12 months would quite likely be broken. This means 
lots of forks without any added value, without evolutions nor bugfixes, 
just "so that it keeps working" .

Many major languages or frameworks have strong commitments on 
retrocompatibility, and have succeeded on this commitment for years 
(python2 being one of them) ; I remember a blogpost where a Qt developer 
explained at great lengths how they carefully crafted every function 
signature, so that it could evolve without breaking older code. If they can 
achieve this C++, how comes major python frameworks would do less ? 

My opinion is that the "walk or die" approach of Django is harmful to its 
ecosystem, and furthermore unnecessary, since a layer like DCP can ensure 
BOTH a wart-free & fast-moving codebase, and a strong retrocompatibility.

About the current tests of deprecation warnings, to me this is a wrong 
"separation of concerns". Checking that a web framework feature works, and 
checking that it properly triggers specific warnings, are aspects which 
should not be mixed with each other. 
Why ? Because it prevents advanced uses of test suites. For example, how do 
you check that Django version X.Y hasn't broken code written for previous 
releases ? Just because it passes its own test suite ? Just because 
"committers have been careful" ? I don't think it says much. What I'd do, 
is rather run the unit-tests of the previous django version (excluding some 
specifically marked "short-lived" tests), against the newest code. This 
would mean so much more.

Here is my view on why using a compatibility layer in Django (or any 
framework, actually) would be a good idea :

PROs:
- cleaner django codebase
- less work to deprecate features
- automated checks of compatibility between django version
- easily achieve long-term compatibility
- more control given to django users regarding compatibility choices

CONs:
- small additional delay at django startup ?

Am I missing something there ?

regards,
Pascal Chambon

Le mardi 10 janvier 2017 01:29:41 UTC+1, Tim Graham a écrit :
>
> Our new deprecation policy was designed to eliminate the need for this 
> type of compatibility library. If you want to support more Django versions 
> than what our guidelines recommend (which may involve supporting versions 
> that no longer receive security updates), I guess this library could be 
> useful, but I wouldn't give it an endorsement in the Django docs.
>
> Perhaps you could elaborate on "Most importantly, this way, django main 
> tests suites  would stop making asserts on the presence or absence of 
> deprecation warnings (which is a way of mixing unrelated aspects of the 
> framework)." I don't understand your complaint about the current way we're 
> testing deprecation warnings.
>
> On Saturday, January 7, 2017 at 3:17:06 PM UTC-5, Pkl wrote:
>>
>>
>> Hello,
>>
>> I've *at last* had some time to experiment with the idea I had thrown 
>> years again, of a system to workaround the recurring breakages introduced 
>> by new django releases, breakages which sometimes lead to unreconcilable 
>> requirements between django apps.
>>
>> After a sprint at PyconFR, with a colleague (R. Gomès), and some cleanups 
>> and improvements, here it is:
>>
>> https://github.com/pakal/django-compat-patcher  (also known as DCP)
>> https://pypi.python.org/pypi/django-compat-patcher
>>
>> The philosophy of the module is that, with extremly agile languages such 
>> as python, one doesn't have to choose between a fast-moving project, and a 
>> compatibility-friendly project.
>> So, in an approach reminding "aspect oriented 

Presenting DCP, a compatibility layer for Django (feedback welcome)

[Pkl]

Hello,

I've *at last* had some time to experiment with the idea I had thrown years 
again, of a system to workaround the recurring breakages introduced by new 
django releases, breakages which sometimes lead to unreconcilable 
requirements between django apps.

After a sprint at PyconFR, with a colleague (R. Gomès), and some cleanups 
and improvements, here it is:

https://github.com/pakal/django-compat-patcher  (also known as DCP)
https://pypi.python.org/pypi/django-compat-patcher

The philosophy of the module is that, with extremly agile languages such as 
python, one doesn't have to choose between a fast-moving project, and a 
compatibility-friendly project.
So, in an approach reminding "aspect oriented programming", DCP separates 
the Django codebase, which may move forward and be left mostly free of 
compatibility shims, from retrocompatibility shims ; and changing the 
"level of compatibility" of a project is just the matter of tweaking some 
Django settings.
Since, as usual, "explicit is better than implicit", logging and code 
warnings are used to ensure that project users are informed about what 
compatibility fixers are applied, and how project dependencies are rated 
deprecation-wise.

I've noticed there had been debates and evolutions for the past years, 
until the switch to a "lax semantic versioning" for Django. 
Surely it's an effort in the right direction, but I guess I'm not the only 
one worried by the "lax" adjective ; especially after the long history of 
breakages occurring for quite cosmetic reasons (eg. the removal of 
urlpatterns(), which broke most django apps available, whereas dotted-path 
URLs were rather harmless).
That's why I think DCP is still relevant, as it gives back some control 
over compatibility to end users of Django ; and it should remove part of 
the apprehension some may have known too, when installing a new Django 
version.

DCP has been successfully used to upgrade wide projects from django 1.7 to 
django 1.10, with no hassle. But more feedback would be nice, before it can 
be marked "production ready".
The project welcomes patches, especially new backward/forward compatibility 
fixers one might need.


*Apart from that *: I think many people could benefit from more 
instructions about how to handle (backwards and forwards) compatibility, so 
here are some ideas I had :


*1) Advertising compatibility systems for Django, in the official docs.*

That is to say, packages like "django-compat" (which I only discovered 
lately), dedicated to library developers who need a wide compatibility.
And packages like "django-compat-patcher" for project maintainers who have 
compatibility issues and deadlines (it's not *always* the right moment to 
fork and patch a bunch of big dependencies).
All that would come in addition to the docs now mentioning "how to support 
2 LTS versions at a time".
These measures might diminish the number of django app using handmade 
compatibility shims, or completely ignoring previous versions of django.


*2) Pushing a compatibility system like DCP into Django itself*

What would be the gain ? It would pleasantly reduce the work needed for 
Django evolutions.

At the moment, first compatibility shims must be introduced, and 
specifically tested in code ; then these compatibility shims must be 
removed, and their tests with them.

With a separate compatibility system, only half the work would be needed : 
once the code has been modified, the old version is moved to a fixer, and 
its associated unit-test is moved to the compatibility system test suite. 
The activation/deactivation of this particular fixer is then only a matter 
of django settings, not of new code commits.
Most importantly, this way, django main tests suites  would stop making 
asserts on the presence or absence of deprecation warnings (which is a way 
of mixing unrelated aspects of the framework). Django tests suites could 
thus be leveraged by compatibility systems to ensure that the whole system 
is well sane and backwards/forward compatible, whatever the series of 
django shims applied.


*Any remarks about DCP concept itself, or about suggestions for doc/core 
inclusions ?*

thanks,
regards,
Pascal

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/37750f9b-d5e4-4216-a510-546502c084c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Fate of sql* management commands

[Pkl]

Hello all,

I've noticed that all sql* commands that were in django core management 
have disappeared from the latest sources.

However, even though Django now has builtin south-style migrations now, 
these *sql** commands (especially sql-all) were priceless to debug 
differences between the models and the actual DB schema. 
Especially as long as some django apps haven't caught up with all these 
recent changes - I myself ended up with missing tables after scrupulously 
following upgrade instructions, with "manage.py migrate" saying "all is 
fine".

*What is the new way to dump the sql schema of currently installed django 
appz ?* It'd maybe be worth that I provide a doc patch to inform users 
about it.
*If there is none, is there an agreement to restore at least sqlall and 
sqlclear ?*

Also what was the reasoning behind, on the first, place, blocking access to 
these utilities in django1.7 (they seemed to work, when I forced them), and 
now removing them ? A simple warning about the new "migrations" stuffs (or 
a "--force" argument) would most probably have sufficed to prevent improper 
usage of these sql* commands, while letting django users getting the info 
they need to move forward, wouldn't it ?

Here is the related ticket : https://code.djangoproject.com/ticket/24481

thanks a lot,
regards,
Pascal

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2f4c051c-d163-4e48-8e41-9a3ab49b28d6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Please don't kill the ecosystem

[Pkl]

Hello,

Hello,

My apologies for sounding rude, and for raising these issues so late.

Yes, upgrading a project itself is a matter of a pair of evenings, the 
problem is with third-party libraries, and more again, plugins of 
third-party libraries (especially CMS-related). 
I've had to bundle a dozen of these with my app, because they didn't quite 
follow the pace of Django evolutions. Now of course, as you say, I might as 
well stabilize for some time on a LTS version, but as was said too, the 
next steps would also be much harder then. Or I could forget about modular 
apps, and go towards monolitic third-party apps instead...

I felt that crucial variables like "request.REQUEST" or "raw_post_data" 
could stay aliased for much more than 2 minor versions (but undocumented 
and with warnings, of course), for teh sake of retrocompatibility.
However if it's acknowledged, in the core team, that code maintainability 
and security would be hindered by keeping these artefacts, I can't argue 
that. 

Concerning the "rules of open source", I've yet to find a satisfying way to 
apply them regarding these "micro breaks". Imagine that project 
"myvideoplugin" is unmaintained (not handling PR) : unable to patch the 
original Repo, I'd have to fork it ; and unable to push results to the 
original pypi name, I'd have to create a separate myvideoplugin-pkl package 
on Pypi... that's quite some hassle for (often) a 10-lines patch.

Thanks Curtis for the proposal, I was thinking about a django-retrocompat 
package indeed, however I realize that doing a decent job of 
retrocompatibility would be a much harder job than initially expected, 
since breaking changes are rarely a matter of simple aliases 
(https://docs.djangoproject.com/en/1.7/releases/1.7/#backwards-incompatible-changes-in-1-7)
 
; django-retrocompat wouldn't stand the expectations of users. But a 
minimal package dealing only with "renamed fields" would be doable, I'll 
have a look at it.

regards, 
Pkl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3317f347-d82b-4edd-90fd-16da41f5dc49%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.