Re: Discussing improvements of django.setup()

[Pkl]

Hello Aymeric,

regarding django.setup() idempotence I continued discussion in the PR.

Regarding django.setup() tweakability, the generic need is simply to have a 
hook, somewhere for users to put early init code, regardless of the way 
django is launched. Code that needs to execute this early are mostly (if 
not all) monkey-patching the environnement, a little like a LD_PRELOAD 
logic python-style.

This can be:
- for (of course) the init of django-compat-patcher
- greening of the environment (e.g. as 
https://github.com/erickponce/django-db-geventpool recommends, for Gevent 
or Eventlet)
- early tweaking of some process aspects likes socket timeouts (they broke 
for me when used along with mutiprocessing, but several years ago)
- possibly setup of specific test scaffolding, for which late mock()-style 
things aren't enough (I have no precise scenario in mind for this one, even 
though I know we did suffer with a Jpype bridge to a java LDAP client, 
which required tons of care to not break, including being initialized 
rather early along with process signal handlers)

Without a hook built into django, users that need such early setup have to 
duplicate their efforts for each way Django can be launched, and they must 
call their patching code in manage.py, in the wsgi file, in their custom 
scripts, and in their specific runners likes pytest-django (this last case 
is quite uneasy to handle at the moment, due to its very early setup of 
Django).

I'm afraid I can't provide much code to help the pondering; simply, without 
builtin hook, the user has to battle to ensure the kind of lines like the 
following are always executed before django.setup(). If Django provides a 
hook, it's so much less to worry about.

from psycogreen.gevent import patch_psycopg
patch_psycopg()


There are surely lots of ways to provide such a hook, via new settings, or 
new environment variables, or modules following precise naming conventions 
that django would systematically try to import... I have no idea how 
exactly my approach is rated amongst them. 

As for alternatives to these early setup hooks, I'm alas clueless...

regards,
Pakal









On Sunday, June 30, 2019 at 10:25:15 AM UTC+2, Aymeric Augustin wrote:
>
> Hello Pakal,
>
> *Making django.setup() idempotent*
>
> In https://code.djangoproject.com/ticket/28752, you say:
>
>
> *> I've been bitten numerous times by the impredictable behaviour of 
> django when django.setup() was called numerous times.> In the old days I 
> had exceptions, now it's mainly subtle breakages of logging configuration.*
>
> This is fuzzy for a bug report. Ideally, you should say:
>
> - What you're doing (minimal reproduction instructions, within supported 
> use of Django)
> - What results you're expecting
> - What results you're getting
>
> Anyway it suggests that the problem is about logging. In this comment 
>  I 
> said that the only possible bug is that `configure_logging()` isn't 
> idempotent. Your answer 
>  confirms 
> that the issue is really about logging. Let's fix configure_logging(), then!
>
> Barring new arguments in favor of the proposed approach, which involves 
> nested locks, I'm -1. Nested locks create a possibility of deadlocks. See 
> my comment on the PR for details.
>
> *Making django.setup() tweakable*
>
> Again, the rationale is fuzzy. I'm seeing two use cases:
>
> - pytest-django
> - django-compat-patchers
>
> One more or one less monkey patch isn't going to make a difference to 
> these projects ;-)
>
> I'm ready to accept that there's a need for projects that aren't big hacks 
> but I need to see it explained clearly and specifically. I'd really like to 
> see a feature request in the form of:
>
> - I want to do X, which is a reasonable thing to do and stays within 
> supported use of Django, except I need to override django.setup()
> - Here's what the code looks like with a workaround (monkey-patching, 
> probably)
> - Here's what the code would look like with the improvement I'm asking for 
> (hopefully much better)
> - Here are the alternatives I considered and why I rejected them
> - Other use cases also include Y and Z
>
> I'm asking this because concrete use cases will put us in a better 
> position to find the best solution.
>
> Best regards,
>
> -- 
> Aymeric.
>
>
>
> On 27 Jun 2019, at 23:26, Pkl > wrote:
>
> Hello everyone,
>
> I'm bringing here, for broader review and discussion, the subject of 
> making django setup more "tweakable":
> https://code.djangoproject.com/ticket/30536
> https://github.com/django/django/pull/11435
>
> There is indeed a need for doing pre/post operations at that crucial 
> moment of the framework lifetime, especially to apply compatibility shims 
> (greening-related, API stability-related...) and setup test scaffolding, 
> stubs etc.
>
> A separate PR has been created to handle 

Re: New feature - Define a variable TESTS in settings

[Rene Fleschenberg]

Hi,

I think it is better to do what Josh mentioned in
https://code.djangoproject.com/ticket/30605#comment:2.

To elaborate on how this looks in pratice:

Write a file called ``settings_test.py`` or such, containing something
like this:

```
from .settings import *

DEFAULT_FILE_STORAGE = 'inmemorystorage.InMemoryStorage'

# More test-specific settings here
```

And run your tests with this settings module (using ``manage.py
--settings settings_test test`` or by setting the
``DJANGO_SETTINGS_MODULE`` env var).

René

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d60d44e9-06f1-d883-f041-b1e9c68f2245%40fleschenberg.net.
For more options, visit https://groups.google.com/d/optout.

Re: Fellow Reports -- June 2019

[Carlton Gibson]

Hi All, 


Calendar Week 25 -- ending 23 June.


Triaged:

https://code.djangoproject.com/ticket/30581 -- Allow constraints to be used 
for validation (in Python) (Accepted)
https://code.djangoproject.com/ticket/30577 -- feature request: custom 
rendering for readonly fields in admin (needsinfo)


Reviewed:

https://code.djangoproject.com/ticket/30427 -- Descriptors not accessible 
for inherited models.
https://code.djangoproject.com/ticket/29043 -- test --keepdb says 
Using existing test database even if its run for the first 
time
https://code.djangoproject.com/ticket/30565 -- HttpResponseBase.close not 
called when using FileResponse / WSGI file wrapper object
https://github.com/django/django/pull/11481 -- Fixed #10403 -- Declaraive 
syntax for FormSet, ModelFormSet  InlineF…
https://github.com/django/django/pull/11460 -- Refs #30485 -- Removed 
non-representative test that emitted a warning.
https://github.com/django/django/pull/11478 -- Fixed #30547 -- Docd 
how Meta.constraints are applied to model valid…
https://code.djangoproject.com/ticket/30066 -- UserManager.create_superuser 
doesnt allow for omitting email or password, unlike create_user which 
does.
https://github.com/django/django/pull/11490 -- Fixed #30572 -- Prevented 
values()/values_list() on combined queryset from mutating the list of 
columns in querysets.
https://github.com/django/django/pull/11481 -- Fixed #10403 -- Declaraive 
syntax for FormSet, ModelFormSet  InlineFormSet


Authored:

https://github.com/django/django/commit/54d0f5e62f54c29a12dd96f44bacd810cbe03ac8
-- Fixed CVE-2019-12781 -- Made HttpRequest always trust 
SECURE_PROXY_SSL_HEADER if set.



Calendar Week 26 -- ending 30 June.



VACATION ⛺️



Kind Regards,

Carlton







-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/68a228c0-c4f7-489d-9594-05373fc55d74%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fellow Reports - June 2019

[Mariusz Felisiak]

Week ending June 30, 2019.


*Triaged:*
https://code.djangoproject.com/ticket/30586 - Django 2.2 compatibility with 
PyMySQL. (duplicate)
https://code.djangoproject.com/ticket/30587 - Cloaking exceptions raised by 
get_object_or_404(). (duplicate)
https://code.djangoproject.com/ticket/30588 - ipdb breaks the autoreloader. 
(accepted)
https://code.djangoproject.com/ticket/30584 - call_command raises 
ValueError when subparser dest is passed in options. (accepted)
https://code.djangoproject.com/ticket/30589 - urlize parses email addresses 
with apostrophes incorrectly. (accepted)
https://code.djangoproject.com/ticket/30590 - Error in the documentation of 
the polling app. (invalid)
https://code.djangoproject.com/ticket/30593 - Add support for check 
constraints on MariaDB 10.2+. (created)
https://code.djangoproject.com/ticket/30591 - MySQL: 1215 - "Cannot add 
foreign key constraint" when altering type of unique field referenced by 
ForeignKey. (accepted)
https://code.djangoproject.com/ticket/30594 - Add Cache-Control: private to 
never_cache decorator. (accepted)
https://code.djangoproject.com/ticket/30595 - sqlsequencereset should 
inform that no sequences found. (accepted)
https://code.djangoproject.com/ticket/30596 - SplitArrayField.has_changed() 
is always True for numeric fields. (accepted)
https://code.djangoproject.com/ticket/30598 - Create new files when 
overriding the default admin site. (wontfix)
https://code.djangoproject.com/ticket/30599 - Cloaking PermissionErrors 
raised in ManifestFilesMixin.read_manifest(). (accepted)
https://code.djangoproject.com/ticket/30597 - Improve documentation on 
reseting an app's migrations to zero. (accepted)
https://code.djangoproject.com/ticket/30600 - Clarifiy that ValueError 
raised by converter.to_python() causes 404. (accepted)
https://code.djangoproject.com/ticket/30602 - Raise ValueError in Extract 
lookups that don't work properly with DurationField. (accepted)
https://code.djangoproject.com/ticket/26539 - Using Annotation As Update 
Parameter Generates Invalid SQL. (duplicate)
https://code.djangoproject.com/ticket/30603 - Stored file locking issue 
when using Django on Windows with the youtube_dl package. (needsinfo)
https://code.djangoproject.com/ticket/30604 - Raise exceptions in 
mail_admins()/mail_managers() when settings are not in expected formats. 
(accepted)

*Reviewed/committed:*
https://github.com/django/django/pull/11477 - Fixed #26431 -- Prevented 
django.urls.resolve() from returning missing optional parameters.
https://github.com/django/django/pull/11510 - Fixed #30477 -- Made reverse 
lookup use Field.get_db_prep_value() from the target field.
https://github.com/django/django/pull/11507 - Fixed #30588 -- Fixed crash 
of autoreloader when __main__ module doesn't have __file__ attribute.
https://github.com/django/django/pull/11514 - Fixed #30594 -- Added 
'private' Cache-Control directive to never_cache() decorator.
https://github.com/django/django/pull/11496 - Fixed #30578 -- Made 
SelectDateWidget respect a custom date format when USE_L10N is disabled.
https://github.com/django/django/pull/11518 - Fixed #28725 -- Prevented 
serializing inherited ManyToManyFields in child model.
https://github.com/django/django/pull/11129 - Fixed #28408 -- Added error 
message when updating with annotated expressions on joined fields.
https://github.com/django/django/pull/11517 - Fixed #30584 -- Fixed 
management command when using subparsers with dest parameter.
https://github.com/django/django/pull/11281 - Fixed #30400 -- Improved 
typography of user facing strings.

*Authored:*
https://github.com/django/django/pull/11520 - Fixed GeoIPTest.test04_city() 
failure with the latest GeoIP2 database.

Best regards,
Mariusz

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/dcb65e83-62e5-4b20-b48d-65b5e2da9f27%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

New feature - Define a variable TESTS in settings

[אורי]

Hi,

I think it might be useful to define a variable (such as TESTS) in
settings, which will only be True when running tests, and then Django will
know we are running tests and act accordingly, for example save files in a
different directory (such as TESTS_MEDIA_ROOT) and not in the default
MEDIA_ROOT directory, and delete them after all the tests end. And other
actions which should run only in tests or only not in tests.

If such a variable is defined, its default value should be of course False.

https://code.djangoproject.com/ticket/30605

Please see my recent comment on this ticket:
https://code.djangoproject.com/ticket/23251

I already defined different settings files in Speedy Net, and set
set django_settings.TESTS to True only in tests. Until recently it was
possible to run tests with other settings such as development, when running
tests locally, but now it's deliberately disabled (with two asserts
in speedy/core/base/test/models.py) because I don't want tests to run with
the production (or development) MEDIA_ROOT settings.

Also notice, I used django_settings.TESTS_MEDIA_ROOT instead
of django_settings.MEDIA_ROOT in tearDownClass in the above file, to
prevent deleting the production MEDIA_ROOT directory by accident if the
tests will still run in production (although it can't because of the
asserts above).

I will appreciate your feedback or comments.

אורי
u...@speedy.net

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CABD5YeFxQWQqiYxGSRm3jSUUzYvpO52Cc9B0MXtq0XfVyF6AHw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Django security releases issued: 2.2.3, 2.1.10 and 1.11.22

[Mariusz Felisiak]

Today the Django team issued 2.2.3, 2.1.10, and 1.11.22 as part of our 
security process. These releases address security issues, and we encourage 
all users to upgrade as soon as possible:

https://www.djangoproject.com/weblog/2019/jul/01/security-releases/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/343b23d8-e41d-4f99-805b-c16b18b1fc66%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Django security releases issued: 2.2.3, 2.1.10 and 1.11.22

[Mariusz Felisiak]

Today the Django team issued 2.2.3, 2.1.10, and 1.11.22 as part of our 
security process. These releases address security issues, and we 
encourage all users to upgrade as soon as possible:


https://www.djangoproject.com/weblog/2019/jul/01/security-releases/

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/787895ab-87d9-d828-78f4-c2f9af278ca4%40gmail.com.
For more options, visit https://groups.google.com/d/optout.