Re: Template-based widget rendering

[Carl Meyer]

Hi Tim,

On 11/16/2016 06:23 AM, Tim Graham wrote:
> I took a stab at backwards-compatibility with respect to allowing the
> admin to work without adding 'django.forms' to INSTALLED_APPS. It seems
> to work for at least the simple TEMPLATES = 'BACKEND':
> 'django.template.backends.django.DjangoTemplates', ...} case, although
> I'm not proud of the code and I'm nervous it's a bit fragile. What do
> you think of the approach?

I have similar concerns about fragility, as well as the behavior being
complex and hard to keep a clear mental model of.

I went back and re-read this thread and took a fresh approach to the
problem, trying to adhere to "keep the easy things easy, make the hard
things possible" and keep the behavior as explicit as possible. Here's
what I came up with, let me know what you think:

We provide three built-in template renderer classes:

1. The default FORM_RENDERER is DjangoTemplateRenderer. It does not use
settings.TEMPLATES/get_template(), but constructs its own DTL template
engine. This engine has APP_DIRS=True, and automatically inserts the
built-in form templates directory _after_ the app dirs. (This will
require a subclass of the normal DTL Engine class that overrides the
template_dirs property, because DIRS takes priority over APP_DIRS, so
the default Engine has no way to specify a directory that has lower
priority than APP_DIRS. But I don't think needing this subclass is a
problem, and I do think it's important that out of the box you can
override the default form templates, as well as adding your own; this
latter of course is needed for the admin.)

2. We also provide JinjaTemplateRenderer, which is just like
DjangoTemplateRenderer, except it uses Jinja. (I.e. we get rid of the
"automatically use Jinja2 if it's installed" behavior of the
StandaloneTemplateRenderer in the current PR, and instead split into two
different renderers for DTL vs Jinja. Magically changing behavior just
because something is installed is bad. The choice to use Jinja should be
explicitly configured, not automatic when you install it.)

3. Lastly, we provide ProjectTemplateRenderer (name up for
bikeshedding), for those who want form template rendering to use their
overall project templating settings, as defined in TEMPLATES. This
renderer just uses the stock get_template() function. If you choose to
switch to this renderer, you will need to either put 'django.forms' in
INSTALLED_APPS (and have at least one engine with APP_DIRS=True), or add
the forms directory in DIRS of one of our engines, or supply all the
form templates you need yourself, or whatever. You've chosen to take
full control, so it's your responsibility to make the form templates you
need are available.

I think this meets all the requirements:

The default renderer is backwards-compatible with no settings changes,
and does not require Jinja2, but it still allows overrides/additions of
widget templates. It's also standalone-compatible, in that it is
self-contained and has no dependency on TEMPLATES config.

Switching to Jinja2 is just a six-character change to your FORM_RENDERER
setting.

If you want complete control of how your widget templates are sourced,
switch to ProjectTemplateRenderer.

And of course you can always write your own renderer class if you need
even more control than that.

Anything I missed?

Carl

 I thought I'd try something to get the
> conversation going again. I've been reacting to some tickets based on
> the assumption that this will get into 1.11 so I'd like to make it happen.

That would be great.

Carl

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7b4d51f5-6f48-acb5-a3bf-802cf7a8ea1f%40oddbird.net.
For more options, visit https://groups.google.com/d/optout.


signature.asc
Description: OpenPGP digital signature

Re: two-phase commit / distributed transaction

[Mateusz Mikołajczyk]

If anybody is interested, I created a proof of concept code for PostgreSQL 
which extends existing Atomic context:

https://github.com/toudi/django-tpc-proof-of-concept

the actual implementation can be found inside tpc/atomic_tpc.py and there 
are two commands available:

./manage.py prepare
./manage.py commit

if atomic() would have been extendible (i.e. if one could replace 
connection.commit() with something else ) then this whole lot of 
monkeypatching wouldn't be necesary.

What do you guys think? Also, has nobody ever written stuff like this? Or 
am searching the wrong way?

cheers,
toudi

W dniu piątek, 2 grudnia 2016 22:02:44 UTC+1 użytkownik Aymeric Augustin 
napisał:
>
> Hello,
>
> To be honest I’m pessimistic about the feasibility of emulating 
> transactional behavior — pretty much the most complicated and low level 
> thing databases do — in the application. I don’t think that would be 
> considered suitable for Django.
>
> Usually Django handles such cases with a database feature flag and make 
> the methods no-ops on databases that don’t support the corresponding 
> features. For instance that’s how Django ignores transactions on MySQL + 
> MyISAM.
>
> Best regards,
>
> -- 
> Aymeric.
>
> On 2 Dec 2016, at 20:43, Mateusz Mikołajczyk  > wrote:
>
> Well, I suppose that it would either lead to very obfuscated 
> implementation code, or very weird syntax (client code). As for your first 
> argument ( promise that the transaction is unlikely to fail ):
>
> from django.db import distributed:
>
> with distributed('foo') as foo:
> MyModel.get_or_create(field=123)
>
> then, before calling the emulated behavior, the db would have to:
>
> * do all the operations (like it would normally do with regular commit - 
> thus checking every constrainst and so on)
> * then do a rollback (so that it doesn't store the actual values in the db)
> * then serialize them in separate journal (the additional model I 
> mentioned - an analogy to an actual separate journal of PostgreSQL)
>
> Utterly ugly / hacky solution if you ask me, but please keep in mind that 
> this would be only emulation of the actual algorithm for the databases 
> which don't support this standart
>
> As for the relations, I have thought a lot about it and the only 
> pseudocode I could think of was utterly ugly as well:
>
> with distributed('foo') as foo:
> foo.add(MyModel.objects.get_or_create, {'field': 123}, 
> namespace='mymodel')
> foo.add(MyOtherModel.objects.create, {'my_model_id': 
> ('from-namespace', 'mymodel')})
>
> Theoretically, both of these syntaxes could co-exist. If you wouldn't have 
> any relations, you could use the cleaner syntax.
>
> So I'd say it would technically be possible but would lead to very, very, 
> very ugly code (at least in the second scenario with relations). And I 
> realize that this is not an option in the Django world.
>
> I understand that because of all the above it is unlikely to create a nice 
> interface which would work in database-agnostic way, therefore Django would 
> have to throw IntegrityError if somebody would be trying to do distributed 
> transaction on non-supported database? But if that's the case then this 
> code doesn't really belong in the django core, does it? Which means that 
> I'm probably left with the monkey-patching thing :( Or .. ? I have to 
> prepare this functionality either way - because I need it ;)
>
> Thank you for all the answers !
>
> W dniu piątek, 2 grudnia 2016 14:32:51 UTC+1 użytkownik Patryk Zawadzki 
> napisał:
>>
>> W dniu piątek, 2 grudnia 2016 12:05:11 UTC+1 użytkownik Mateusz 
>> Mikołajczyk napisał:
>>>
>>> What would you say about checking which CRUD operations were executed 
>>> within atomic() call (in order to serialize them and save into a special 
>>> model for databases which don't support this functionality) ? Is it 
>>> realistic? 
>>>
>>
>> It would likely break the promise that distributed two-step transactions 
>> give you: that once all statements are prepared the transaction is unlikely 
>> to fail during commit. In this case the commit would mean "start over and 
>> try to repeat my steps" at which point any of the recorded statements is 
>> likely to fail constraint checks. (Even more so if your code used 
>> get_or_create().)
>>
>> Also how would relations work? You begin a transaction, create a Foo 
>> instance and the returned PK is 5. You assign it to child models. At this 
>> point the transaction is saved and rolled back. During replay the insert 
>> returns PK = 7, at this point there's no way to detect that some of the 
>> stored fives should now be treated as sevens while some should remain fives.
>>
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com .
> To post to 

trouble with pull request builders

[Tim Graham]

This week the Jenkins pull request builders haven't been working correctly, 
probably due to an update in Jenkins itself or somewhere else. After 
migrating the Jenkins master from Ubuntu 12.04 to 16.04, I started 
configuring a different pull request builder plugin that Simon Charette 
recommended. It has some nice features that the current (broken) one 
doesn't have like automatic aborting of a running build if the PR is 
updated. Meanwhile, pull request checks aren't enabled. Sorry for the 
inconvenience. I hope things will be back to normal no later than early 
next week.

new plugin: 
https://wiki.jenkins-ci.org/display/JENKINS/GitHub+Integration+Plugin
old plugin: 
https://wiki.jenkins-ci.org/display/JENKINS/GitHub+pull+request+builder+plugin

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2c521328-8443-4f83-8e8b-0f38692a5518%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Attaching custom migartion operations to model definition.

[Mārtiņš Šulcs]

Hello,

I'd like to mark my models in some declerative way(meta option, base class) 
and make autodetector insert custom migration operations after CreateModel, 
DeleteModel.
For this particular use case i have generation of triggers in mind but i 
can think of other uses cases when this could be useful.
Is this possible in non-intrusive/ugly way.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9e1a6529-e85b-4cb1-9d5b-6e373aba969d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: two-phase commit / distributed transaction

[Aymeric Augustin]

Hello,

To be honest I’m pessimistic about the feasibility of emulating transactional 
behavior — pretty much the most complicated and low level thing databases do — 
in the application. I don’t think that would be considered suitable for Django.

Usually Django handles such cases with a database feature flag and make the 
methods no-ops on databases that don’t support the corresponding features. For 
instance that’s how Django ignores transactions on MySQL + MyISAM.

Best regards,

-- 
Aymeric.

> On 2 Dec 2016, at 20:43, Mateusz Mikołajczyk  
> wrote:
> 
> Well, I suppose that it would either lead to very obfuscated implementation 
> code, or very weird syntax (client code). As for your first argument ( 
> promise that the transaction is unlikely to fail ):
> 
> from django.db import distributed:
> 
> with distributed('foo') as foo:
> MyModel.get_or_create(field=123)
> 
> then, before calling the emulated behavior, the db would have to:
> 
> * do all the operations (like it would normally do with regular commit - thus 
> checking every constrainst and so on)
> * then do a rollback (so that it doesn't store the actual values in the db)
> * then serialize them in separate journal (the additional model I mentioned - 
> an analogy to an actual separate journal of PostgreSQL)
> 
> Utterly ugly / hacky solution if you ask me, but please keep in mind that 
> this would be only emulation of the actual algorithm for the databases which 
> don't support this standart
> 
> As for the relations, I have thought a lot about it and the only pseudocode I 
> could think of was utterly ugly as well:
> 
> with distributed('foo') as foo:
> foo.add(MyModel.objects.get_or_create, {'field': 123}, 
> namespace='mymodel')
> foo.add(MyOtherModel.objects.create, {'my_model_id': ('from-namespace', 
> 'mymodel')})
> 
> Theoretically, both of these syntaxes could co-exist. If you wouldn't have 
> any relations, you could use the cleaner syntax.
> 
> So I'd say it would technically be possible but would lead to very, very, 
> very ugly code (at least in the second scenario with relations). And I 
> realize that this is not an option in the Django world.
> 
> I understand that because of all the above it is unlikely to create a nice 
> interface which would work in database-agnostic way, therefore Django would 
> have to throw IntegrityError if somebody would be trying to do distributed 
> transaction on non-supported database? But if that's the case then this code 
> doesn't really belong in the django core, does it? Which means that I'm 
> probably left with the monkey-patching thing :( Or .. ? I have to prepare 
> this functionality either way - because I need it ;)
> 
> Thank you for all the answers !
> 
> W dniu piątek, 2 grudnia 2016 14:32:51 UTC+1 użytkownik Patryk Zawadzki 
> napisał:
> W dniu piątek, 2 grudnia 2016 12:05:11 UTC+1 użytkownik Mateusz Mikołajczyk 
> napisał:
> What would you say about checking which CRUD operations were executed within 
> atomic() call (in order to serialize them and save into a special model for 
> databases which don't support this functionality) ? Is it realistic? 
> 
> It would likely break the promise that distributed two-step transactions give 
> you: that once all statements are prepared the transaction is unlikely to 
> fail during commit. In this case the commit would mean "start over and try to 
> repeat my steps" at which point any of the recorded statements is likely to 
> fail constraint checks. (Even more so if your code used get_or_create().)
> 
> Also how would relations work? You begin a transaction, create a Foo instance 
> and the returned PK is 5. You assign it to child models. At this point the 
> transaction is saved and rolled back. During replay the insert returns PK = 
> 7, at this point there's no way to detect that some of the stored fives 
> should now be treated as sevens while some should remain fives.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to django-developers@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-developers 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/8d4ca04f-cdcd-4d8b-a060-fc5598b3baf8%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you 

Re: two-phase commit / distributed transaction

[Mateusz Mikołajczyk]

Well, I suppose that it would either lead to very obfuscated implementation 
code, or very weird syntax (client code). As for your first argument ( 
promise that the transaction is unlikely to fail ):

from django.db import distributed:

with distributed('foo') as foo:
MyModel.get_or_create(field=123)

then, before calling the emulated behavior, the db would have to:

* do all the operations (like it would normally do with regular commit - 
thus checking every constrainst and so on)
* then do a rollback (so that it doesn't store the actual values in the db)
* then serialize them in separate journal (the additional model I mentioned 
- an analogy to an actual separate journal of PostgreSQL)

Utterly ugly / hacky solution if you ask me, but please keep in mind that 
this would be only emulation of the actual algorithm for the databases 
which don't support this standart

As for the relations, I have thought a lot about it and the only pseudocode 
I could think of was utterly ugly as well:

with distributed('foo') as foo:
foo.add(MyModel.objects.get_or_create, {'field': 123}, 
namespace='mymodel')
foo.add(MyOtherModel.objects.create, {'my_model_id': ('from-namespace', 
'mymodel')})

Theoretically, both of these syntaxes could co-exist. If you wouldn't have 
any relations, you could use the cleaner syntax.

So I'd say it would technically be possible but would lead to very, very, 
very ugly code (at least in the second scenario with relations). And I 
realize that this is not an option in the Django world.

I understand that because of all the above it is unlikely to create a nice 
interface which would work in database-agnostic way, therefore Django would 
have to throw IntegrityError if somebody would be trying to do distributed 
transaction on non-supported database? But if that's the case then this 
code doesn't really belong in the django core, does it? Which means that 
I'm probably left with the monkey-patching thing :( Or .. ? I have to 
prepare this functionality either way - because I need it ;)

Thank you for all the answers !

W dniu piątek, 2 grudnia 2016 14:32:51 UTC+1 użytkownik Patryk Zawadzki 
napisał:
>
> W dniu piątek, 2 grudnia 2016 12:05:11 UTC+1 użytkownik Mateusz 
> Mikołajczyk napisał:
>>
>> What would you say about checking which CRUD operations were executed 
>> within atomic() call (in order to serialize them and save into a special 
>> model for databases which don't support this functionality) ? Is it 
>> realistic? 
>>
>
> It would likely break the promise that distributed two-step transactions 
> give you: that once all statements are prepared the transaction is unlikely 
> to fail during commit. In this case the commit would mean "start over and 
> try to repeat my steps" at which point any of the recorded statements is 
> likely to fail constraint checks. (Even more so if your code used 
> get_or_create().)
>
> Also how would relations work? You begin a transaction, create a Foo 
> instance and the returned PK is 5. You assign it to child models. At this 
> point the transaction is saved and rolled back. During replay the insert 
> returns PK = 7, at this point there's no way to detect that some of the 
> stored fives should now be treated as sevens while some should remain fives.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8d4ca04f-cdcd-4d8b-a060-fc5598b3baf8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: two-phase commit / distributed transaction

[Patryk Zawadzki]

W dniu piątek, 2 grudnia 2016 12:05:11 UTC+1 użytkownik Mateusz Mikołajczyk 
napisał:
>
> What would you say about checking which CRUD operations were executed 
> within atomic() call (in order to serialize them and save into a special 
> model for databases which don't support this functionality) ? Is it 
> realistic? 
>

It would likely break the promise that distributed two-step transactions 
give you: that once all statements are prepared the transaction is unlikely 
to fail during commit. In this case the commit would mean "start over and 
try to repeat my steps" at which point any of the recorded statements is 
likely to fail constraint checks. (Even more so if your code used 
get_or_create().)

Also how would relations work? You begin a transaction, create a Foo 
instance and the returned PK is 5. You assign it to child models. At this 
point the transaction is saved and rolled back. During replay the insert 
returns PK = 7, at this point there's no way to detect that some of the 
stored fives should now be treated as sevens while some should remain fives.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/098bbb3a-1ba0-42e5-8b75-441474981855%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: two-phase commit / distributed transaction

[Mateusz Mikołajczyk]

What would you say about checking which CRUD operations were executed 
within atomic() call (in order to serialize them and save into a special 
model for databases which don't support this functionality) ? Is it 
realistic? 

What I mean by that is that when you do:

from django.db import transaction

with transaction.atomic():
MyModel.objects.create(field=123)

then the generated SQL is something like

BEGIN;
INSERT INTO mymodel values (123);
COMMIT;

However, if the database doesn't support the TPC functionality, the SQL 
would have to be slightly different, say:

BEGIN;
INSERT INTO prepared_transactions (txn_id, model, operation, params) values 
('foo', 'MyModel', 'create', '{field:123}');
COMMIT;

But on the other hand, if the database does support that, it could be 
'normal', i.e.:

BEGIN;
INSERT INTO mymodel values ( ... )
BEGIN TRANSACTION 'foo';
(no COMMIT)

If it is not possible to trace the CRUD operations, would it be easier to 
introduce a slightly different syntax, say ...

from django.db import prepare_distributed:

with prepare_distributed('foo') as prepare:
prepare.add_operation(MyModel.objects.create, {'field': 123})

After all, it's not like the developer doesn't know whether he's doing a 
distributed transaction or not.. 
 
As for making the atomic() more complex, I don't think that it would be 
significantly hard. The distributed transaction isn't really *that* 
different - it's just calling PREPARE TRANSACTION 'foo' (without calling 
COMMIT). I thought that the Atomic class could simply have some kind of 
inner method hooks. The default class could then implement those:

class Atomic(ContextDecorator):
def _commit_wrapper(self, connection):
return connection.commit()

but the Two phase could do it differently:

class TwoPhaseAtomic(Atomic):
def _commit_wrapper(self, connection):
return 
connection.prepare_distributed(self.distributed_transaction_id);

Of course, the prepare_distributed call would create models in the special 
table if the database wouldn't support the functionality and call regular 
commit() at the end, or call appropriate command otherwise - so this seems 
like the easiest thing to do. The problem that I haven't figured out yet 
would be to trace the instances being saved / created / etc ..

W dniu czwartek, 1 grudnia 2016 14:54:27 UTC+1 użytkownik Florian Apolloner 
napisał:
>
>
>
> On Thursday, December 1, 2016 at 2:04:53 PM UTC+1, Aymeric Augustin wrote:
>>
>> The person who will write and test that code has all my sympathy :-) 
>>
>
> I'll second that, I have no idea how Aymeric managed to keep his sanity 
> while rewriting the transaction code :D 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/0a95bc57-d1d2-491f-9ce6-8d7956be2954%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.