Re: Database "execute hooks" for instrumentation

[Adam Johnson]

django-perf-rec  would love this,
it currently monkey patches connection.ops.last_executed_query to listen to
all the queries going on.

On 7 April 2017 at 16:10, Shai Berger  wrote:

> On Friday 07 April 2017 17:47:51 Carl Meyer wrote:
> > Hi Shai,
> >
> > On 04/07/2017 06:02 AM, Shai Berger wrote:
> > > This is an idea that came up during the djangocon-europe conference:
> Add
> > > the ability to install general instrumentation hooks around the
> database
> > > "execute" and "executemany" calls.
> > >
> > > Such hooks would allow all sorts of interesting features. For one, they
> > > could replace the current special-case allowing assertNumQueries &
> > > friends to record queries out of debug mode (it's an ugly hack,
> really),
> > > but they could also support my imagined use case -- a context-manager
> > > which could prevent database access during execution of some code (I'm
> > > thinking mostly of using it around "render()" calls and serialization,
> > > to make sure all database access is being done in the view).
> >
> > Another use-case is for preventing database access during tests unless
> > specifically requested by the test (e.g. pytest-django does this,
> > currently via monkeypatching).
> >
>
> Yep. This feels right.
>
> > > My idea for implementation is to keep a thread-local stack of context-
> > > managers, and have them wrap each call of "execute". We could actually
> > > even use one such context-manager instead of the existing
> > > CursorDebugWrapper.
> >
> > Why a new thread-local instead of explicitly per-connection and stored
> > on the connection?
> >
>
> Sorry for implying that it would be a new thread-local, I just hadn't
> thought
> it through when I wrote this. Of course it goes on the (already
> thread-local)
> connection.
>
> Shai.
>



-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM00XnYSh0rNQ91-ALG2toCoT3RCgyBqpsFvq%2BpSccx5PA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: DJANGO_SETTINGS_FILE

[Adam Johnson]

I implemented this based upon the code snippet I showed in my first reply. 
Anyone who wishes to use DJANGO_SETTINGS_FILE can now pip install 
django-settings-file and follow its setup instructions: 
https://pypi.python.org/pypi/django-settings-file

I've added some caveats I thought of during development in the project 
README. I'll copy those here that are relevant to the idea in itself, not 
just my implementation:

   - If the Python file defined by DJANGO_SETTINGS_FILE tries to do any 
   imports, the directory the file is in will not be on PYTHONPATH, so the 
   author of the settings file might get some surprises.
   - Additionally, you might experience other problems from loading a file 
   whilst it’s not on PYTHONPATH.
   - If you want your settings file to extend another one, it will probably 
   find it easiest to import the base one from a location on PYTHONPATH, 
   otherwise it too will have to do use the same import ‘hacks’ to load the 
   default settings.
   - File paths are not portable between operating systems, so you may need 
   logic to support both Unix and Windows at once.
   - 
   - File paths are not portable between .py and .pyc files. In most cases 
   this means a .pyc file will not be used for settings since it can't be 
   guaranteed to be there, slightly slowing down import time.
   
I think Django core should still stick with DJANGO_SETTINGS_MODULE. 
Python's import machinery does a lot for us.

On Monday, April 10, 2017 at 6:12:02 AM UTC+1, Luke Plant wrote:
>
>
>
> On 09/04/17 16:16, Josh Smeaton wrote:
>
>
> So I think there are a few questions to go over.
>
> 1. What are the more successful strategies that work in the wild? (files 
> in /etc/, PYTHONPATH, env vars, local_settings.py)
> 2. Are any of the above clearly superior?
> 3. Is this a serious problem that people are having?
> 4. If yes, does it need to be solved in core?
>
>
> To put a word in on number 2 regarding environment variables, we should 
> take into account articles like:
>
>  
> https://diogomonica.com/2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/
>
>
> The argument for solving in core would be if there are security issues we 
> can genuinely fix by putting something in core rather than allowing custom 
> mechanisms, and if that mechanism will really work in most cases. We also 
> need to consider shared hosting where /etc/ is not an option.
>
> Luke
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e569dfb9-5809-423b-9f27-1208af9b1f93%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Feature Request] Let manage.py ignore migration dependencies

[Shai Berger]

Hi,

On Thursday 13 April 2017 18:18:57 Brock Hallenbeck wrote:
> 
> Due to these duplicate tables, my process of setting up an app database is
> as follows:
> 
> 1.Create database on server / point django at it / set up router
> 2. migrate  --database=app_db
> 3. Delete all the duplicate auth_* and django_* tables
> 4. Run FDW script importing auth_* and django_* tables from auth_db
> 
> If I could do something like python manage.py migrate 
> --database=app_db --ignore-dep I could do something like:
> 
> 1. Create database on server / point django at it / set up router
> 2. Run FDW script importing auth_* and django_* tables from auth_db
> 3. migrate  --database=app_db --ignore-dep
> 
> All thoughts and criticisms welcome.

Instinctively, the process I would expect is:

1. Create database on server / point django at it / set up router
2. Run FDW script importing auth_* and django_* tables from auth_db
3. migrate --database=app_db --fake auth  # etc
4. migrate  --database=app_db

Why does that not solve your problem?

[Feature Request] Let manage.py ignore migration dependencies

[Brock Hallenbeck]

We are a municipality using Django to make a CRUD tool. Our use cases are 
usually some department in the town wants to do CRUD stuff so they come to 
us. Our dream is to make an app, plug in some models and be mostly done. 
However it is very common for us to need to use different databases. Due to 
the lack of cross database foreign key relationships this means we will 
have to make a copy of our project every time we need to use a separate 
database because in our CRUD project, every object that is created has a 
'_created_by' field that foreign keys to User. 

After some digging I have found that using Postgres' foreign data wrappers, 
I can designate and 'auth' database, and have other databases access it's 
tables as their own. Using database routers I write to the auth database 
and read from the app database. This results in all our apps using all 
different databases can consult the auth database instead of local auth_* 
tables which would be out of sync with any other app we have.

As far as I can tell this is working. However setting this up was a bit of 
a hassle. After creating the auth database by getting the auth_* and 
django_* tables set up, the next step was setting up my first app database.

In our case, we have a base model that every single model we write 
subclasses, this model simply adds some utility fields that we like, namely 
'created_by' and '_last_updated_by'. These are fields that foreign key into 
Django's 'User' model. The result is that for every single application we 
deploy, when we migrate, due to dependencies, we get the app's tables, and 
all the auth_*, django_* tables. 

Due to these duplicate tables, my process of setting up an app database is 
as follows:

1.Create database on server / point django at it / set up router
2. migrate  --database=app_db
3. Delete all the duplicate auth_* and django_* tables
4. Run FDW script importing auth_* and django_* tables from auth_db

If I could do something like python manage.py migrate  
--database=app_db --ignore-dep I could do something like:

1. Create database on server / point django at it / set up router
2. Run FDW script importing auth_* and django_* tables from auth_db
3. migrate  --database=app_db --ignore-dep

I could probably streamline steps 1 and 2 somehow. 

I understand that without the dependencies the sql output might be invalid 
as the tables don't exist. I am essentially asking for the ability to deal 
with the repercussions of not having the correct tables present in the 
database myself. 

All thoughts and criticisms welcome.




-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/bbc940d6-5e9a-4d59-9a5b-0537ec22416f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.