Hi Christian, we are doing such a thing for quite a while in our open source project pretix[1]. I'm not sure if its something that Django needs to do better, since requirements for this tend to deviate a lot and there's already a solid basis in the Python packaging toolchain to start from: To discover apps/plugins, we rely on setuptools' entry point feature[2], which allows us to easily load all compatible apps installed in the local Python requirement.
We then disable/enable plugins on a per-client level by using a custom subclass from Signal[4] and automatically wrapping all views with a decorator that makes them conditional[6] through some URLConf inclusion tricks[5]. >From a plugin author perspective, this makes for a pretty clean view[7] and we are pretty satisfied with the approach. You can also watch me explaining it here: https://www.youtube.com/watch?v=5NxRdzLTFik Cheers Raphael [1] https://github.com/pretix/pretix [2] https://packaging.python.org/specifications/entry-points/ [3] https://github.com/pretix/pretix/blob/master/src/pretix/settings.py#L264 [4] https://github.com/pretix/pretix/blob/master/src/pretix/base/signals.py#L21 [5] https://github.com/pretix/pretix/blob/master/src/pretix/multidomain/maindomain_urlconf.py#L23 [6] https://github.com/pretix/pretix/blob/master/src/pretix/multidomain/plugin_handler.py [7] https://docs.pretix.eu/en/latest/development/api/index.html Am Sun, 29 Jul 2018 13:26:04 +0200 schrieb Christian González <christian.gonza...@nerdocs.at>: > Hello DjangoDevs, > > I'm new here to this group, and to be honest, just a "fake" developer. > Doing that is not my main job. So please be patient with my maybe > BadIdea(tm). > > Another warning: much text. Hint: there is a TL;DR at the end. > > I stumbled upon a "missing feature" in Django, which I call "dynamic" > app loading, a while ago, since I try to create a Django based > application which can dynamically add plugins to itself. > > I first tried to google the internet, and found many Stackexchange Q&A > where this topic is handled, but either in an insufficient way, not > applicable to Django 2.0, or else. > > Best ones: > https://stackoverflow.com/questions/24027901/dynamically-loading-django-apps-at-runtime > https://stackoverflow.com/questions/7933596/django-dynamic-model-fields > > And my own question with no answer so far: > https://stackoverflow.com/questions/51234829/dynamic-django-apps > > So I began implementing my own way of handling this. > > Let me first tell a "user story", so you can imagine what I mean. > > My application should more or less be a framework that provides a > loosely-coupled bunch of modules working together, with a dependency > tree and versioning. There is a "core module", and others that depend > on it (e.g. "notifications" etc.). Third party apps should be > possible, and something like an "app store" should be created to > dynamically download apps from within the program, and add that > functionality to the main application. > > So, my first approach was creating zip files with a predefined > structure (models.py, schema.py, views.py, client stuff etc.), and > tried to load this code during runtime. I soon realized that I had to > re-implement most of the stuff Django does anyway, and doing > migrations isn't an easy task when done barefoot. > > I then changed my mind, and found the best way of having "dynamic > plugins" is using "Django apps" as plugins. > But: Django apps are not pluggable. They have to be inserted > hardcodedly into INSTALLED_APPS to have a predictable order of > loading. Yes, I've read > https://groups.google.com/forum/#!searchin/django-developers/app-loading%7Csort:date/django-developers/_iggZzrYtJQ/FWFPgCflSnkJ > - and I "kind of" understand the Django setup() process (see later). > > I started to fiddle with INSTALLED_APPS, as recommended in > Stackexchange etc., and dynamically searched a "plugins" directory to > add some plugins into the list of other apps, just by extending > INSTALLED_APPS. Django sees no difference, has no cache problems and > happily loads all my plugins. > > BUT: this is no way dynamic. First thing I recognized is: You can't > simply call DB requests anywhere near the settings.py loading time. > Because there is no DB at that moment, let alone models. I then > stuffed the code into AppConfig.ready() of the core app, and was a > step further, even if it's not recommended to call models there: > I need to use models there: I want to check if a plugin app on disk is > "deactivated", and NOT load it in that case. Aaaargh. Back to the > start. > > * In settings.py, you can tell Django to dynamically load plugins, > using disk IO code there. BadIdea(tm). > * in AppConfig.ready(), you can use models, even if discouraged, but > it's too late to find "plugin apps" now and add it to INSTALLED_APPS. > * in a middleware, you can use Models (somehow), but same problem. > It's too late to add models. > > I at least managed to add this plugins' URLs to my main urls by > providing a plugin hook in the main urls.py which is called in all the > plugins: > > main urls.py: > > PluginManager.load_plugin_submodule('urls') > > for patternplugin in IURLPattern: > urlpatterns += patternplugin.get_patterns() > > Where IURLPattern is a "Interface" class that can be used in plugins: > > @implements(IURLpattern) > class FooPluginURLs: > def get_patterns(self(): > return [path('foo/', FooAPIView.as_view(), name='test')] > > So the main URLs add all dynamically added urls. But, like I said: no > way dynamic, as it's fully deterministic at Django start. > > What I wanted is: Danymically download such an app, stuff it into the > Django system and - bling - it works, with models, URLs, and > everything, after a "django_reload" magic. > > Ok, next: I pimped my PluginManager, created a middleware that starts > at Django start and loads all plugins. So, no INSTALLED_APPS tweaks, > done in middleware, after all models are available. > > The plugin manager now reimplements the django.apps.populate() method > and does the same things again, bypassing checks of already loaded > apps. This works somehow(tm). But there are many problems remaining, > and I think it is worth rethinking the whole Django app loading > process to make it more dynamic: > > > TL;DR: > https://code.djangoproject.com/ticket/29554 > Could Django be changed to not load apps hardcodedly at startup, but > load them "dynamically", e.g. reusing the existing "apps.populate()" > method? > > I think one of the best approaches would be: let the user call > apps.populate() too, like setup() does it with INSTALLED_APPS. But let > the user dynamically add lines to that variable. or even better: Let > the user load one app at a time, like by extending the "Apps" class: > > django.apps.load_app("my.own.app.apps.MyAppConfig") > > Like Aymeric states in the issue: there has to be discussed what > Django needs to do about dependencies. ATM app loading is fully > deterministic (order defined by INSTALLED_APPS). > But I think this is not necessary for dynamically added apps. Django > CANNOT decide that, as it would imply a sophisticated dependency and > versioning schema of apps. Which IMHO Django should NOT provide, to be > as flexible as possible. But it SHOULD provide the ability to create > such a framework, which it does not at the moment. > The framework I try to implement should be responsible for loading the > apps in the right order (because of here: dependency tree, versions, > etc. - but this could be done completely different with another Django > application!) > > So, somtehing like an "apps.load_app(dotted_appname, reload:bool = > False)" method should only be responsible for: > > [X] loading AppConfigs (check if already loaded, reload?) > -> code can be reused, as populate() ATM just creates a possibly > missing AppConfig and creates an internal list of apps > [X] loading Models (check if already loaded, reload?) > -> code can be reused as well here. > [ ] invalidate cache (models, AppConfigs etc) > > and consequently, there must be a apps.unload_app(dotted_appname) > method, to remove that again. > > Aymeric writes in the issue: > > I'd like to see a thorough discussion of the pros and cons before > making this decision as well as an analysis of which caches need to be > invalidated and how this could happen. > > I'm the wrong one to give input here. No knowledge of Django > internals. > > Please tell me what you think of that feature. > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at https://groups.google.com/group/django-developers. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20180730085320.5489b78f%40kvothe. For more options, visit https://groups.google.com/d/optout.
pgp92lJRXzNJN.pgp
Description: Digitale Signatur von OpenPGP
