Hello follow Djangonauts.

I'm writing this post in an attempt to ignite a discussion about the 
autodetector, and how to make it expandable, such that third-party apps can 
generate their custom migrations automatically.

This has previously been discussed here:
* https://groups.google.com/d/msg/django-developers/qRNkReCZiCk/I8dIxxhoBwAJ
* https://groups.google.com/d/msg/django-developers/pbj7U7FBL6Q/eJtRV7QwDwAJ


*Motivation:*

My motivation for writing this post is two fold:

   1. First, I recently attended DjangoCon Europe 2019, where Carlton 
   Gibson had a keynote about contributing back to Django, I want to do so, 
   and thus this is my first post on the mailing list.
   2. Secondly, I believe there are various cases where being able to 
   expand the automigrator enables libraries to be a lot cleaner. For instance:
   

   - Enforcing constriants on data-models, such as it's done with 
   unique_together, and CheckConstraints. Currently these are implemented 
   directly in Django, but given an expandable autodector they could have 
   been implemented as libraries. In the long section at the end of this 
   post, I mention a similar case to this, that was my original motivation 
   for writing this post.
   - Enabling rebust auditing models via database triggers, by simply 
   adding 'audit_trigger = True' to the Meta class of models that should be 
   audited, as done by: 
      - https://github.com/carta/postgres_audit_triggers
   - Automatically enabling database extensions, when fields which require 
   these extensions are utilized. As an example, utilizing PostGIS or HStore 
   currently requires activating their corresponding extensions manually using 
   CreateExtension('extension'), which could be automated.
   - Really any case, in which one would like to automatically generate 
   migrations, and which cannot be achieved by custom fields.


*The proposal:*

Create a plugin system where plugins can register handlers / callbacks to 
be run from within the automigrator. Potentially this could even be 
implemented using the signals mechanism currently in Django, as presented 
in 'Building plugin ecosystems with Django', by Raphael Michel at DjangoCon 
Europe 2019.

I imagine the signal should be fired here:

   - https://github.com/django/django/blob/master/django
   /db/migrations/autodetector.py#L192

And that signals handlers should act as combination between the 
'create_altered_X' and 'generate_altered_X' methods, resulting either to 
calls of 'self.add_operation', or returning a list of operations to add to 
the migrationsfile.

There is an open question with regards to how dependencies and shared state 
should be handled.
- I do not have enough insight to answer this.

I am very open to other solutions, this is merely what I came up with, 
without having any throughout insight into how Django works internally.
- I am willing to implement the plugin system, and create a PR for this, if 
a consensus is found.


*My personal motivation (long section):*

I have a database design, which includes several data invariants, which I 
want to uphold. I've been able to implement most of these data invariants 
using uniqueness (on fields, unique_together, UniqueConstraint), and 
CheckConstraint.

CheckConstraint is excellent, and I'm very happy that made it to Django 
with 2.2, but for enforcing data invariants it has one major shortcoming. - 
It only operates on a single table. 

Currently to implement data invariants across multiple tables, I believe 
the suggestion is to override save methods, or to use save signals.
- However these signals are not emitted for queryset operations, such as 
bulk_inserts or update s, and as such cannot be used to maintain data 
invariants, without relying on manual procedures that are error-prone, and 
take up review time. On another note, save methods and save signals, also 
do not help if the databsae is accseed outside of the Django ORM.

Thus if you need reliable constraints that work across multiple tables, I 
don't believe that a robust solution is currently in place, thus I've been 
working on a prototype for a library to fulfill this role. The library 
enables very strong data invariant checking by allowing one to write 
database constraint triggers via Querysets.

The envisioned library is very simple, and works as follows (based upon the 
pizza toppings example):
class Topping(models.Model):
    name = models.CharField(max_length=30)

    def __str__(self):
        return self.name

class PizzaTopping(models.Model):
    class Meta:
        unique_together = ("pizza", "topping")

    pizza = models.ForeignKey('Pizza')
    topping = models.ForeignKey('Topping')

class Pizza(models.Model):
    name = models.CharField(max_length=30)
    toppings = models.ManyToManyField('Topping', through=PizzaTopping) 

    def __str__(self):
        return self.name
<http://self.name>
We envision having a data invariant on the number of toppings allowed per 
pizza, and with the library we can now enforce that, by adding the 
'constraint_triggers' option to the the Meta class of PizzaTopping:
class PizzaTopping(models.Model):
    class Meta:
        unique_together = ("pizza", "topping")
        constraint_triggers = [ConstraintTrigger(
            name='At most 5 toppings',
            query=M('PizzaTopping').objects.values(
                'pizza'
            ).annotate(
                num_toppings=Count('topping')
            ).filter(
                num_toppings__gt=5
            ),
            error='Trying to add more than 5 toppings to one pizza.',

        }]
    ...


After which one can simply run:
python manage.py makemigration

Which will generate the migration similar to:
django_constraint_triggers.operations.AddConstraintTrigger(
            model_name='pizzatopping',
            trigger_name='At most 5 toppings',
            query=django_constraint_triggers.utils.M(app_label='...', model=
'PizzaTopping', operations=[...])
            error='Trying to add more than 5 toppings to one pizza.',
        ),

That can then be applied using:
python manage.py migrate

To generate and install the trigger on the database:
Triggers:
    dct__trig__1688bfdc185c490f AFTER INSERT OR UPDATE ON app_pizzatopping 
DEFERRABLE INITIALLY DEFERRED FOR EACH ROW EXECUTE PROCEDURE 
dct__func__1688bfdc185c490f()

Where the database function is just:
IF EXISTS (
    QUERYSET_SQL

) THEN
    RAISE check_violation USING MESSAGE = '{}';
END IF;
RETURN NULL;

Upon which we are now protected against adding too many toppings to a pizza:
cheese = Topping.objects.create(name="Cheese")
ham = Topping.objects.create(name="Ham")
pepperoni = Topping.objects.create(name="Pepperoni")
mushrooms = Topping.objects.create(name="Mushrooms")
onions = Topping.objects.create(name="Onions")
# TODO: Add CheckConstraint against pineapple
pineapple = Topping.objects.create(name="Pineapple")

pizza = Pizza.objects.create(name="Django Special")
PizzaTopping.objects.create(pizza=pizza, topping=cheese)
PizzaTopping.objects.create(pizza=pizza, topping=ham)
PizzaTopping.objects.create(pizza=pizza, topping=pepperoni)
PizzaTopping.objects.create(pizza=pizza, topping=mushrooms)
PizzaTopping.objects.create(pizza=pizza, topping=onions)
# NOTE: Atleast pineapple was added last, and rejected this way.

with self.assertRaises(IntegrityError):
    PizzaTopping.objects.create(pizza=pizza, topping=pineapple)

The way the library works right now, is by monkey patching the autodetector 
to pick up the constraint_triggers entry on the meta class, and by 
expanding the OPTIONS.DEFAULT_NAMES to allow for setting this variable on 
the Meta class.
- The combination of these are then used to automatically generate entries 
in migrationfiles with custom operations.

The fact that monkey-patching is used means this library is not 
interoperable with any other libraries changing the autodetector like this, 
and that code smells.

The library althrough in it's early stage, and very dirty, can be found 
here:

   - https://github.com/magenta-aps/django_constraint_triggers

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/96703d89-be3e-44fe-9888-9aa5b3b9ceb6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to