Hi Emil, This is a very interesting idea and I find your use-case compelling. Adding support for custom triggers to Django itself would resolve your main use case this time but I also see value in a general solution that enables Django's third-party ecosystem.
I think the next step is to open a new ticket (or find an existing one). Ian On Sun, 14 Apr 2019 at 15:20, Emil Madsen <sove...@gmail.com> wrote: > Hello follow Djangonauts. > > I'm writing this post in an attempt to ignite a discussion about the > autodetector, and how to make it expandable, such that third-party apps can > generate their custom migrations automatically. > > This has previously been discussed here: > * https://groups.google.com/d/msg/django > -developers/qRNkReCZiCk/I8dIxxhoBwAJ > * https://groups.google.com/d/msg/django > -developers/pbj7U7FBL6Q/eJtRV7QwDwAJ > > > *Motivation:* > > My motivation for writing this post is two fold: > > 1. First, I recently attended DjangoCon Europe 2019, where Carlton > Gibson had a keynote about contributing back to Django, I want to do > so, and thus this is my first post on the mailing list. > 2. Secondly, I believe there are various cases where being able to > expand the automigrator enables libraries to be a lot cleaner. For > instance: > > > - Enforcing constriants on data-models, such as it's done with > unique_together, and CheckConstraints. Currently these are implemented > directly in Django, but given an expandable autodector they could have > been implemented as libraries. In the long section at the end of this > post, I mention a similar case to this, that was my original > motivation for writing this post. > - Enabling rebust auditing models via database triggers, by simply > adding 'audit_trigger = True' to the Meta class of models that should be > audited, as done by: > - https://github.com/carta/postgres_audit_triggers > - Automatically enabling database extensions, when fields which > require these extensions are utilized. As an example, utilizing PostGIS or > HStore currently requires activating their corresponding extensions > manually using CreateExtension('extension'), which could be automated. > - Really any case, in which one would like to automatically generate > migrations, and which cannot be achieved by custom fields. > > > *The proposal:* > > Create a plugin system where plugins can register handlers / callbacks to > be run from within the automigrator. Potentially this could even be > implemented using the signals mechanism currently in Django, as presented > in 'Building plugin ecosystems with Django', by Raphael Michel at > DjangoCon Europe 2019. > > I imagine the signal should be fired here: > > - https://github.com/django/django/blob/master/django > /db/migrations/autodetector.py#L192 > > And that signals handlers should act as combination between the > 'create_altered_X' and 'generate_altered_X' methods, resulting either to > calls of 'self.add_operation', or returning a list of operations to add > to the migrationsfile. > > There is an open question with regards to how dependencies and shared > state should be handled. > - I do not have enough insight to answer this. > > I am very open to other solutions, this is merely what I came up with, > without having any throughout insight into how Django works internally. > - I am willing to implement the plugin system, and create a PR for this, > if a consensus is found. > > > *My personal motivation (long section):* > > I have a database design, which includes several data invariants, which I > want to uphold. I've been able to implement most of these data invariants > using uniqueness (on fields, unique_together, UniqueConstraint), and > CheckConstraint. > > CheckConstraint is excellent, and I'm very happy that made it to Django > with 2.2, but for enforcing data invariants it has one major shortcoming. - > It only operates on a single table. > > Currently to implement data invariants across multiple tables, I believe > the suggestion is to override save methods, or to use save signals. > - However these signals are not emitted for queryset operations, such as > bulk_inserts or update s, and as such cannot be used to maintain data > invariants, without relying on manual procedures that are error-prone, and > take up review time. On another note, save methods and save signals, also > do not help if the databsae is accseed outside of the Django ORM. > > Thus if you need reliable constraints that work across multiple tables, I > don't believe that a robust solution is currently in place, thus I've been > working on a prototype for a library to fulfill this role. The library > enables very strong data invariant checking by allowing one to write > database constraint triggers via Querysets. > > The envisioned library is very simple, and works as follows (based upon > the pizza toppings example): > class Topping(models.Model): > name = models.CharField(max_length=30) > > def __str__(self): > return self.name > > class PizzaTopping(models.Model): > class Meta: > unique_together = ("pizza", "topping") > > pizza = models.ForeignKey('Pizza') > topping = models.ForeignKey('Topping') > > class Pizza(models.Model): > name = models.CharField(max_length=30) > toppings = models.ManyToManyField('Topping', through=PizzaTopping) > > def __str__(self): > return self.name > <http://self.name> > We envision having a data invariant on the number of toppings allowed per > pizza, and with the library we can now enforce that, by adding the > 'constraint_triggers' option to the the Meta class of PizzaTopping: > class PizzaTopping(models.Model): > class Meta: > unique_together = ("pizza", "topping") > constraint_triggers = [ConstraintTrigger( > name='At most 5 toppings', > query=M('PizzaTopping').objects.values( > 'pizza' > ).annotate( > num_toppings=Count('topping') > ).filter( > num_toppings__gt=5 > ), > error='Trying to add more than 5 toppings to one pizza.', > > }] > ... > > > After which one can simply run: > python manage.py makemigration > > Which will generate the migration similar to: > django_constraint_triggers.operations.AddConstraintTrigger( > model_name='pizzatopping', > trigger_name='At most 5 toppings', > query=django_constraint_triggers.utils.M(app_label='...', > model='PizzaTopping', operations=[...]) > error='Trying to add more than 5 toppings to one pizza.', > ), > > That can then be applied using: > python manage.py migrate > > To generate and install the trigger on the database: > Triggers: > dct__trig__1688bfdc185c490f AFTER INSERT OR UPDATE ON app_pizzatopping > DEFERRABLE INITIALLY DEFERRED FOR EACH ROW EXECUTE PROCEDURE > dct__func__1688bfdc185c490f() > > Where the database function is just: > IF EXISTS ( > QUERYSET_SQL > > ) THEN > RAISE check_violation USING MESSAGE = '{}'; > END IF; > RETURN NULL; > > Upon which we are now protected against adding too many toppings to a > pizza: > cheese = Topping.objects.create(name="Cheese") > ham = Topping.objects.create(name="Ham") > pepperoni = Topping.objects.create(name="Pepperoni") > mushrooms = Topping.objects.create(name="Mushrooms") > onions = Topping.objects.create(name="Onions") > # TODO: Add CheckConstraint against pineapple > pineapple = Topping.objects.create(name="Pineapple") > > pizza = Pizza.objects.create(name="Django Special") > PizzaTopping.objects.create(pizza=pizza, topping=cheese) > PizzaTopping.objects.create(pizza=pizza, topping=ham) > PizzaTopping.objects.create(pizza=pizza, topping=pepperoni) > PizzaTopping.objects.create(pizza=pizza, topping=mushrooms) > PizzaTopping.objects.create(pizza=pizza, topping=onions) > # NOTE: Atleast pineapple was added last, and rejected this way. > > with self.assertRaises(IntegrityError): > PizzaTopping.objects.create(pizza=pizza, topping=pineapple) > > The way the library works right now, is by monkey patching the > autodetector to pick up the constraint_triggers entry on the meta class, > and by expanding the OPTIONS.DEFAULT_NAMES to allow for setting this > variable on the Meta class. > - The combination of these are then used to automatically generate entries > in migrationfiles with custom operations. > > The fact that monkey-patching is used means this library is not > interoperable with any other libraries changing the autodetector like this, > and that code smells. > > The library althrough in it's early stage, and very dirty, can be found > here: > > - https://github.com/magenta-aps/django_constraint_triggers > > -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at https://groups.google.com/group/django-developers. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/96703d89-be3e-44fe-9888-9aa5b3b9ceb6%40googlegroups.com > <https://groups.google.com/d/msgid/django-developers/96703d89-be3e-44fe-9888-9aa5b3b9ceb6%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFv-zfKv%2B6Dtq6hqeVVz_sHUNTTH5OyOPZhORXEdw94CmbTMHg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
