Re: [Django] #34849: Appending django.contrib.postgres to the INSTALLED_APPS via @modify_settings crashes.

[Django]

#34849: Appending django.contrib.postgres to the INSTALLED_APPS via
@modify_settings crashes.
-+-
 Reporter:  Mariusz Felisiak |Owner:  David
 |  Sanders
 Type:  Bug  |   Status:  closed
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Release blocker  |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Mariusz Felisiak ):

 In [changeset:"491092086929f90c048b606c58859f76238bcda2" 4910920]:
 {{{
 #!CommitTicketReference repository=""
 revision="491092086929f90c048b606c58859f76238bcda2"
 [5.0.x] Fixed #34849 -- Avoided raising RuntimeWarning about import-time
 queries when apps are reinitialized with test tools.

 Regression in fbd16438f46bc2128926958ad24331da5d1b406f.

 Backport of 4f2ae0644de34791b84b7beb11a966ce4bc48fb6 from main
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018acb307c36-186426d8-c0b0-4667-ac0d-3af64226b67f-00%40eu-central-1.amazonses.com.

Re: [Django] #34849: Appending django.contrib.postgres to the INSTALLED_APPS via @modify_settings crashes.

[Django]

#34849: Appending django.contrib.postgres to the INSTALLED_APPS via
@modify_settings crashes.
-+-
 Reporter:  Mariusz Felisiak |Owner:  David
 |  Sanders
 Type:  Bug  |   Status:  closed
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Release blocker  |   Resolution:  fixed
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak ):

 * status:  assigned => closed
 * resolution:   => fixed


Comment:

 In [changeset:"4f2ae0644de34791b84b7beb11a966ce4bc48fb6" 4f2ae06]:
 {{{
 #!CommitTicketReference repository=""
 revision="4f2ae0644de34791b84b7beb11a966ce4bc48fb6"
 Fixed #34849 -- Avoided raising RuntimeWarning about import-time queries
 when apps are reinitialized with test tools.

 Regression in fbd16438f46bc2128926958ad24331da5d1b406f.
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018acb300277-0fe5b5c7-a556-4c45-b93b-ee5935b72752-00%40eu-central-1.amazonses.com.

Re: [Django] #34849: Appending django.contrib.postgres to the INSTALLED_APPS via @modify_settings crashes.

[Django]

#34849: Appending django.contrib.postgres to the INSTALLED_APPS via
@modify_settings crashes.
-+-
 Reporter:  Mariusz Felisiak |Owner:  David
 |  Sanders
 Type:  Bug  |   Status:  assigned
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Ready for
 |  checkin
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by Mariusz Felisiak):

 * owner:  nobody => David Sanders
 * status:  new => assigned
 * stage:  Accepted => Ready for checkin


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018aca7f501e-c76bb020-727a-4209-8603-b4497ca315db-00%40eu-central-1.amazonses.com.

Re: [Django] #34849: Appending django.contrib.postgres to the INSTALLED_APPS via @modify_settings crashes.

[Django]

#34849: Appending django.contrib.postgres to the INSTALLED_APPS via
@modify_settings crashes.
-+-
 Reporter:  Mariusz Felisiak |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  1|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by David Sanders):

 * has_patch:  0 => 1


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abb92b7e1-21339e1f-ec74-43c7-bb61-a9c49e3a376a-00%40eu-central-1.amazonses.com.

Re: [Django] #34849: Appending django.contrib.postgres to the INSTALLED_APPS via @modify_settings crashes.

[Django]

#34849: Appending django.contrib.postgres to the INSTALLED_APPS via
@modify_settings crashes.
-+-
 Reporter:  Mariusz Felisiak |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by David Sanders):

 Here's a failing test that I came up with:

 {{{
 --- a/tests/postgres_tests/test_apps.py
 +++ b/tests/postgres_tests/test_apps.py
 @@ -5,7 +5,8 @@ from django.db import connection
  from django.db.backends.signals import connection_created
  from django.db.migrations.writer import MigrationWriter
  from django.test import TestCase
 -from django.test.utils import modify_settings
 +from django.test.utils import modify_settings, override_settings
 +from django.contrib.postgres.signals import get_hstore_oids

  try:
  from django.contrib.postgres.fields import (
 @@ -27,6 +28,11 @@ except ImportError:

  @unittest.skipUnless(connection.vendor == "postgresql", "PostgreSQL
 specific tests")
  class PostgresConfigTests(TestCase):
 +def test_install_app(self):
 +get_hstore_oids.cache_clear()
 +with
 override_settings(INSTALLED_APPS=["django.contrib.postgres"]):
 +pass
 +
  def test_register_type_handlers_connection(self):
  from django.contrib.postgres.signals import
 register_type_handlers
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018abb113d5c-0a297476-a7ce-4af5-bf25-1f409e02ba09-00%40eu-central-1.amazonses.com.

Re: [Django] #34849: Appending django.contrib.postgres to the INSTALLED_APPS via @modify_settings crashes.

[Django]

#34849: Appending django.contrib.postgres to the INSTALLED_APPS via
@modify_settings crashes.
-+-
 Reporter:  Mariusz Felisiak |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by David Sanders):

 @Florian to demonstrate the issue, you can make the postgres tests fail
 with `--parallel=1` if you replace the lines getting type oids with their
 underlying non-cached versions (`get_hstore_oids()` and
 `get_citext_oids()` are cached so when it comes time to reload the module
 there's no db interaction for a single process)

 Eg this diff shows what I mean (I'm using psycopg2 so I've replaced it in
 the psycopg2 block)

 {{{
 --- a/django/contrib/postgres/signals.py
 +++ b/django/contrib/postgres/signals.py
 @@ -55,7 +55,7 @@ else:
  if connection.vendor != "postgresql" or connection.alias ==
 NO_DB_ALIAS:
  return

 -oids, array_oids = get_hstore_oids(connection.alias)
 +oids, array_oids = get_type_oids(connection.alias, "hstore")
  # Don't register handlers when hstore is not available on the
 database.
  #
  # If someone tries to create an hstore field it will error there.
 This is
 }}}

 If you apply that change then `--parallel=1` will fail.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ab386-c012b934-8244-40ad-9cdf-5f9f30574694-00%40eu-central-1.amazonses.com.

Re: [Django] #34849: Appending django.contrib.postgres to the INSTALLED_APPS via @modify_settings crashes.

[Django]

#34849: Appending django.contrib.postgres to the INSTALLED_APPS via
@modify_settings crashes.
-+-
 Reporter:  Mariusz Felisiak |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Mariusz Felisiak):

 Replying to [comment:5 Florian Zimmermann]:
 > > First of all we run the entire test suite not random subsets of test
 classes.
 >
 > So it's expected that the tests behave differently when I only run a
 subset of them?

 Of course it's not, that's the issue. The way how `@modify_settings` and
 initializing `django.contrib.postgres` work are non trivial. We have here
 a long standing isolation issue, revealed by
 fbd16438f46bc2128926958ad24331da5d1b406f.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ab22d2329-384c0c8d-13d9-47bb-91f8-5fc0877bfe65-00%40eu-central-1.amazonses.com.

Re: [Django] #34849: Appending django.contrib.postgres to the INSTALLED_APPS via @modify_settings crashes.

[Django]

#34849: Appending django.contrib.postgres to the INSTALLED_APPS via
@modify_settings crashes.
-+-
 Reporter:  Mariusz Felisiak |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Florian Zimmermann):

 > First of all we run the entire test suite not random subsets of test
 classes.

 So it's expected that the tests behave differently when I only run a
 subset of them? I can't even reproduce this if I just run the failing
 tests by themselves...

 {{{
 > python runtests.py --settings test_postgres postgres_tests.test_apps
 # works

 > python runtests.py --settings test_postgres
 postgres_tests.test_aggregates
 # works

 > python runtests.py --settings test_postgres --parallel 1
 postgres_tests.test_apps postgres_tests.test_aggregates
 # works

 > python runtests.py --settings test_postgres --parallel 2
 postgres_tests.test_apps postgres_tests.test_aggregates
 # fails
 }}}

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018ab2245bcd-eaea8b0d-e69e-4e61-8baa-da44185b6a37-00%40eu-central-1.amazonses.com.

Re: [Django] #34849: Appending django.contrib.postgres to the INSTALLED_APPS via @modify_settings crashes.

[Django]

#34849: Appending django.contrib.postgres to the INSTALLED_APPS via
@modify_settings crashes.
-+-
 Reporter:  Mariusz Felisiak |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Florian Zimmermann):

 Maybe worth noting: for me this depends on the `--parallel` option with
 `--parallel 1` producing even more errors:

 `--parallel 2` no errors:
 {{{
 > python runtests.py --settings test_postgres --parallel 2 postgres_tests
 Found 590 test(s).
 
..x..xx...
 --
 Ran 590 tests in 7.974s

 OK (expected failures=3)
 }}}

 `--parallel 4` two errors:
 {{{
 > python runtests.py --settings test_postgres --parallel 4 postgres_tests
 Found 590 test(s).
 
EEx..xx...
 ==
 ERROR: test_register_serializer_for_migrations
 (postgres_tests.test_apps.PostgresConfigTests)
 --
 Traceback (most recent call last):
   File "C:\Program Files\Python310\lib\unittest\case.py", line 59, in
 testPartExecutor
 yield
   File "C:\Program Files\Python310\lib\unittest\case.py", line 591, in run
 self._callTestMethod(testMethod)
   File "C:\Program Files\Python310\lib\unittest\case.py", line 549, in
 _callTestMethod
 method()
   File "D:\django\tests\postgres_tests\test_apps.py", line 64, in
 test_register_serializer_for_migrations
 with self.modify_settings(INSTALLED_APPS={"append":
 "django.contrib.postgres"}):
   File "D:\django\django\test\utils.py", line 405, in __enter__
 return self.enable()
   File "D:\django\django\test\utils.py", line 587, in enable
 super().enable()
   File "D:\django\django\test\utils.py", line 474, in enable
 apps.set_installed_apps(self.options["INSTALLED_APPS"])
   File "D:\django\django\apps\registry.py", line 362, in
 set_installed_apps
 self.populate(installed)
   File "D:\django\django\apps\registry.py", line 124, in populate
 app_config.ready()
   File "D:\django\django\contrib\postgres\apps.py", line 71, in ready
 register_type_handlers(conn)
   File "D:\django\django\contrib\postgres\signals.py", line 40, in
 register_type_handlers
 oids, array_oids = get_hstore_oids(connection.alias)
   File "D:\django\django\contrib\postgres\signals.py", line 24, in
 get_hstore_oids
 return get_type_oids(connection_alias, "hstore")
   File "D:\django\django\contrib\postgres\signals.py", line 10, in
 get_type_oids
 cursor.execute(
   File "D:\django\django\db\backends\utils.py", line 77, in execute
 return self._execute_with_wrappers(
   File "D:\django\django\db\backends\utils.py", line 90, in
 _execute_with_wrappers
 return executor(sql, params, many, context)
   File "D:\django\django\db\backends\utils.py", line 94, in _execute
 warnings.warn(self.APPS_NOT_READY_WARNING_MSG,
 category=RuntimeWarning)
 RuntimeWarning: Accessing the database during app initialization is
 discouraged. To fix this warning, avoid executing queries in
 AppConfig.ready() or when your app modules are imported.

 

Re: [Django] #34849: Appending django.contrib.postgres to the INSTALLED_APPS via @modify_settings crashes.

[Django]

#34849: Appending django.contrib.postgres to the INSTALLED_APPS via
@modify_settings crashes.
-+-
 Reporter:  Mariusz Felisiak |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Mariusz Felisiak):

 > Shouldn't Jenkins have picked this up earlier, ...

 No, it shouldn't. First of all we run the entire test suite not random
 subsets of test classes. Secondly, this is quite tricky and you have to
 debug how types are registered for `django.contrib.postgresql` and how
 `@modify_settings` works:

 - types are registered when connection is created by `connection_created`
 handler, so it's not when app is initialized, so far so good,
 - when `@modify_settings` manipulates `INSTALLED_APPS` then we undo
 effects of `PostgresConfig.ready()`,
 - `@modify_settings` calls `PostgresConfig.ready()` again, tries to
 registered handlers for existing connections and hits the database to
 select OIDs.

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018aaf33725d-d8040a75-655e-4123-8660-23f24bbbc82e-00%40eu-central-1.amazonses.com.

Re: [Django] #34849: Appending django.contrib.postgres to the INSTALLED_APPS via @modify_settings crashes.

[Django]

#34849: Appending django.contrib.postgres to the INSTALLED_APPS via
@modify_settings crashes.
-+-
 Reporter:  Mariusz Felisiak |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-

Comment (by Florian Zimmermann):

 I'm super confused. Shouldn't Jenkins have picked this up earlier, i.e.
 when fbd16438f46bc2128926958ad24331da5d1b406f (#33143) was merged?

-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018aaf122a08-245d6260-e0f8-4e6e-9df1-e97e6ee7ca6c-00%40eu-central-1.amazonses.com.

Re: [Django] #34849: Appending django.contrib.postgres to the INSTALLED_APPS via @modify_settings crashes.

[Django]

#34849: Appending django.contrib.postgres to the INSTALLED_APPS via
@modify_settings crashes.
-+-
 Reporter:  Mariusz Felisiak |Owner:  nobody
 Type:  Bug  |   Status:  new
Component:  Database layer   |  Version:  5.0
  (models, ORM)  |
 Severity:  Release blocker  |   Resolution:
 Keywords:   | Triage Stage:  Accepted
Has patch:  0|  Needs documentation:  0
  Needs tests:  0|  Patch needs improvement:  0
Easy pickings:  0|UI/UX:  0
-+-
Changes (by David Sanders):

 * stage:  Unreviewed => Accepted


-- 
Ticket URL: 
Django 
The Web framework for perfectionists with deadlines.

-- 
You received this message because you are subscribed to the Google Groups 
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-updates+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-updates/0107018aac46b093-88c0a03f-7663-44bf-8b2f-cbfcdd0fc9f4-00%40eu-central-1.amazonses.com.