Re: Proposal for a transaction.on_before_commit

[Aymeric Augustin]

Hello Raphael,

Oh - a use case for django-transaction-signals 
 ;-) I'm bringing up 
this elaborate joke because you're essentially asking for a "pre-commit" signal 
here and the README contains a good list of things that can go wrong with 
transaction signals. (Please ignore how this package demonstrates a way to do 
it as third-party code *cough* *cough* *cough*)

> I figured a perfect way to do this would be using `save()` or
> `post_save` to add the changed model instance to some kind of
> thread-local list, and then using `transaction.on_commit` to "schedule"
> the aggregation and create the log entries when all changes have been
> made. However, this obviously is not a good enough because `on_commit`
> runs *after* the `COMMIT` statement and thus we're not guaranteed that
> all log entries are persisted to the database.


In my opinion "saving the log entries may fail after a successful transaction" 
isn't the main design problem here. The bigger problem is "log may contain 
entries for writes that don't actually happen, because some savepoints were 
rolled back, typically due to atomic blocks exiting with an exception". And 
then you get dragged into the whole complexity that the README of 
django-transaction-signals outlines and that we're trying to avoid in Django.

(If you don't have any savepoint rollbacks, then your code sounds sufficiently 
constrained to implement logging of changes explicitly at the application layer 
rather than at the framework layer.)

If you run with ATOMIC_REQUESTS, I would suggest to replace it by a custom 
middleware that wraps get_response(request) in an atomic block. Then you know 
that this is the outermost traction and you can do whatever needed before 
exiting the atomic block. You also need the same in all management commands, 
which could be a problem if you depend on third-party management commands.

Failing that, in order to log changes with transactional correctness enforced 
by the ACID guarantees of PostgreSQL, I'd recommend doing it at that database 
level with triggers — which always execute in the same transaction. I realize 
it may be difficult to perform the kind of aggregation you have in mind.

As a last resort, I'd try a custom database backend to track accurately 
transactions and savepoints and maintain an up-to-date record of changes that 
will happen when the transaction is committed.

Hope this helps!

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/E028A2F2-9E00-4599-B1F0-E1406F5810A6%40polytechnique.org.

Re: Proposal for a transaction.on_before_commit

[Florian Apolloner]

Because the scoping of the transactions makes it hard I imagine. A view 
could start more than one transaction for instance, and one of those might 
succeed and the other might get rolled back. Thinking more about this, a 
system like this might get hard to implement. Even if thread locals are 
used, one might have to clear the list in cases of rollbacks or at least 
partially clear it for savepoint rollbacks. And all that doesn't even 
consider autocommit yet :) 

On Sunday, October 10, 2021 at 5:39:18 PM UTC+2 Shai Berger wrote:

> Just to clarify the use-case: 
>
> Why is a before-commit signal preferable to a vanilla Python 
> context-manager around the code? Or, if it is in the context of 
> requests, a middleware? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/549f5570-8cf3-4a1a-b381-e72a5d15d66en%40googlegroups.com.

Re: Proposal for a transaction.on_before_commit

[Shai Berger]

Just to clarify the use-case:

Why is a before-commit signal preferable to a vanilla Python
context-manager around the code? Or, if it is in the context of
requests, a middleware?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20211010183855.02017a38.shai%40platonix.com.

Re: Proposal for a transaction.on_before_commit

[Markus Holtermann]

Hi Raphael,

This is funny. Just last week I looked into exactly the same thing for audit 
logging as well. Except that I'm writing multiple audit log events and want to 
batch them at the end of the transaction in a single bulk_create operation 
instead of a dozen save() calls.

I haven't gotten anywhere for now. But was considering the thread local 
approach as well. But then, thread locals and Django are not the nicest 
combination.

Maybe we can come up with an approach together?

Cheers

Markus

On Sun, Oct 10, 2021, at 3:55 PM, Raphael Michel wrote:
> Hi everyone,
>
> I've spent some days thinking about a use case we have internally where
> we want to create some database records automatically based on some
> other records, think e.g. of an audit log where whenever an instance of
> model X is changed, we want to create an instance of model Y as a log
> record. Of course, there are *lots* of ways and libraries that allow you
> to do this with Django, mostly somehow circling around overriding
> `save()` or using the `post_save` signal.
>
> However, we want to do this in an *aggregated* way, i.e. even if you
> add hundreds of model instances in the same transaction, we only want
> *one* new line in our audit log to be created. This is impossible with
> `save()` or `post_save` alone since we can't know which change is the
> "last" that should trigger the audit log to be created.
>
> I figured a perfect way to do this would be using `save()` or
> `post_save` to add the changed model instance to some kind of
> thread-local list, and then using `transaction.on_commit` to "schedule"
> the aggregation and create the log entries when all changes have been
> made. However, this obviously is not a good enough because `on_commit`
> runs *after* the `COMMIT` statement and thus we're not guaranteed that
> all log entries are persisted to the database.
>
> So I was wondering if there was potential for a
> `transaction.on_before_commit` that works just like
> `transaction.on_commit` except that it is executed before the `COMMIT`
> statement. I imagine there are lots of other possible uses as well and
> without looking into it much deeper, it seems easy enough to implement.
>
> Does anyone have opinions on whether or not this would be a good
> feature to add to Django? Unfortunately it doesn't seem possible to do
> this as third-party code (except if we were shipping entire database
> backends) so it would need to be an acceptable change to Django to be a
> viable option.
>
> Thanks
> Raphael
>
> -- 
> You received this message because you are subscribed to the Google 
> Groups "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send 
> an email to django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/20211010155522.6489b86d%40amara.localdomain.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6cd73b6f-d445-4706-b42a-276ae99213d1%40beta.fastmail.com.

Re: Why can't range function be added in templates?

[Dhruva Shaw]

well my bad, tbh i wasn't shouting, just that caps was mistakenly on. If 
ever I was shouting then I would be using some words which wouldn't be 
appealing, :thinking: is bruh noo.. a slang or just an expression came out 
of shock! , i believe anyways `bruh no` is not a slang, *anyways nvm*
On Saturday, October 9, 2021 at 4:15:11 PM UTC+5:30 Adam Johnson wrote:

> Please don't use shouty capitals or angry slang like "Bruh no" on this 
> mailing list. This mailing list is for calm professional discussion.
>
> On Sat, 9 Oct 2021 at 11:35, Dhruva Shaw  wrote:
>
>> SINCE THE NO OF SELECT BOXES THAT I HAVE GENERATE COMES DIRECTLY FROM THE 
>> DATABSE, THUS RANGE FUNCTION IS NEEDED!
>>
>> On Saturday, October 9, 2021 at 2:40:29 PM UTC+5:30 f.apo...@gmail.com 
>> wrote:
>>
>>> On Friday, October 8, 2021 at 6:14:19 PM UTC+2 dhruv...@gmail.com wrote:
>>>
 Ik range function can be added but I also mentioned that there are some 
 cases range function can't be direct generated from the view like this  
 https://github.com/Tanzanite-lpu/tgl-2.0.0/blob/ddce5ef87a4b5248b8cb8cd5fa3df4adb3f00b31/main/templates/groups.html#L36

>>>
>>> Why was "|range" added there? You are not using {{ j }} there at all but 
>>> {{ forloop.counter }}. There doesn't seem to be any reason to use "|range" 
>>> there. 
>>>
>> -- 
>>
> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/3e733a40-b9a0-4bc7-ba87-943bc5aee635n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c06c5991-7e5a-46b9-81b8-c358676d80dbn%40googlegroups.com.

Proposal for a transaction.on_before_commit

[Raphael Michel]

Hi everyone,

I've spent some days thinking about a use case we have internally where
we want to create some database records automatically based on some
other records, think e.g. of an audit log where whenever an instance of
model X is changed, we want to create an instance of model Y as a log
record. Of course, there are *lots* of ways and libraries that allow you
to do this with Django, mostly somehow circling around overriding
`save()` or using the `post_save` signal.

However, we want to do this in an *aggregated* way, i.e. even if you
add hundreds of model instances in the same transaction, we only want
*one* new line in our audit log to be created. This is impossible with
`save()` or `post_save` alone since we can't know which change is the
"last" that should trigger the audit log to be created.

I figured a perfect way to do this would be using `save()` or
`post_save` to add the changed model instance to some kind of
thread-local list, and then using `transaction.on_commit` to "schedule"
the aggregation and create the log entries when all changes have been
made. However, this obviously is not a good enough because `on_commit`
runs *after* the `COMMIT` statement and thus we're not guaranteed that
all log entries are persisted to the database.

So I was wondering if there was potential for a
`transaction.on_before_commit` that works just like
`transaction.on_commit` except that it is executed before the `COMMIT`
statement. I imagine there are lots of other possible uses as well and
without looking into it much deeper, it seems easy enough to implement.

Does anyone have opinions on whether or not this would be a good
feature to add to Django? Unfortunately it doesn't seem possible to do
this as third-party code (except if we were shipping entire database
backends) so it would need to be an acceptable change to Django to be a
viable option.

Thanks
Raphael

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20211010155522.6489b86d%40amara.localdomain.


pgpl51uPmIu7I.pgp
Description: Digitale Signatur von OpenPGP

Re: Error in Models Documentation

[Ken Whitesell]

This is not an error.

The identified example does _not_ say that it will "return" those 
entries. It states it will _exclude_ entries matching either of those 
two conditions.


Likewise, the SQL statement referenced is a negation ("and not") of the 
condition.


This portion of the documentation is accurate and correct with what it's 
saying.


Ken


On 10/10/2021 8:15 AM, Pranav Mittal wrote:

Hello everyone.

https://docs.djangoproject.com/en/3.2/ref/models/querysets/#django.db.models.query.QuerySet.exclude

The second example mentions that the query will return entries /whose 
/pub_date/ is later than 2005-1-3 /*_O_**_R _*/whose headline is “Hello”./

/
/
/But the equivalent SQL statement mentions /_AND_.


Untitled.png
--
You received this message because you are subscribed to the Google 
Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9028b784-9bd8-4207-91e0-d46e9907fb44n%40googlegroups.com 
.


--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2d528a89-2612-554e-d772-fce9ebfaf1ed%40comcast.net.

Error in Models Documentation

[Pranav Mittal]

Hello everyone.

https://docs.djangoproject.com/en/3.2/ref/models/querysets/#django.db.models.query.QuerySet.exclude

The second example mentions that the query will return entries *whose *
pub_date* is later than 2005-1-3 **O**R **whose headline is “Hello”.*

*But the equivalent SQL statement mentions **AND*.


[image: Untitled.png]

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9028b784-9bd8-4207-91e0-d46e9907fb44n%40googlegroups.com.

Re: Django ORM query syntax enhancement

[Asif Saif Uddin]

Hi Adam,

I agree with some of your points. however djngo orm query syntax is the 
main pain point for anyone new to django orm.  The reason those packages 
are not widely used is because most people dont know about them and not 
about this DEP. And most new users mainly end up learning built in query 
syntax. not many know much about these Deps and orm syntactic sugars.

If some movement were seen in this front and advocacy then probably there 
will be more attention to syntactic sugar packages. The orm sugar package 
is still works in production with django orm as well. So it would be great 
to see the Dep could be improved based on the both packages good parts.

I think I should try to  write some blogs as well using those packages.

Best,
Asif
On Wednesday, October 6, 2021 at 3:55:58 PM UTC+6 Adam Johnson wrote:

> I would not be for merging anything into Django at this time.
>
> There are several libraries providing "enhanced" query syntax: 
> django-orm-sugar, django-model-values, and django-natural-query. None of 
> them seems to be particularly popular (<100 github stars, minimal PyPI 
> downloads), and only django-model-values is maintained. Moreover they all 
> achieve their goals without any changes to Django, so there's not much of a 
> compelling reason to merge anything to core.
>
> To consider adding "another way to do things" in the ORM we'd want more 
> evidence that users want it. I'd suggest further contributions and advocacy 
> for django-model-values or similar. Its documentation could definitely be 
> improved, and blog posts could promote its advantages.
>
> On Wed, 6 Oct 2021 at 10:13, Asif Saif Uddin  wrote:
>
>> Hey all,
>>
>> can we have some consensus on this?
>>
>> Asif
>>
>> On Saturday, April 1, 2017 at 12:55:27 PM UTC+6 Alexey Zankevich wrote:
>>
>>> Hey all,
>>>
>>> Please check a draft DEP related to external query language support by 
>>> Django ORM https://github.com/django/deps/pull/40. 
>>>
>>> Regards,
>>> Alexey
>>>
>>>
>>> On Wednesday, March 22, 2017 at 10:07:51 AM UTC+3, Asif Saifuddin wrote:

 Hi Aric,

 I checked your package. it's nice too. thanks for the work.

 Asif

 On Monday, October 17, 2016 at 4:38:26 AM UTC+6, Aric Coady wrote:
>
> +1.  I implemented much the same thing for part of django-model-values 
> , but went with F 
> expressions 
>  as the 
> basis instead.  Primarily because F expressions already support operator 
> overloading and are a natural intermediate object;  from there one can 
> create Q, OrderBy, and Func objects.
>
> In []: from model_values import F
>
> In []: F.user.created
> Out[]: FExpr(user__created)
>
> In []: F.user.created >= 0
> Out[]: 
>
> In []: F.user.created.min()
> Out[]: Min(FExpr(user__created))
>
> In []: -F.user.created
> Out[]: OrderBy(FExpr(user__created), descending=True)
>
> In []: F.text.iexact('...')
> Out[]: 
>
>
>
>
> On Thursday, October 6, 2016 at 10:04:56 AM UTC-7, Alexey Zankevich 
> wrote:
>>
>> Hey all,
>>
>> Just want to announce recent changes in Django ORM Sugar library, 
>> which might be useful in future Django query syntax enhancement (if ever 
>> happens).
>> 1. Now it supports indexes and slices:
>>
>> >>> Q.data.owner.other_pets[0].name='Fishy'
>> Q(data__owner__other_pets__0__name='Fishy')
>> >>> Q.tags[0] == 'thoughts'
>> Q(tags__0='thoughts')
>> >>> Q.tags[0:2].contains(['thoughts']) 
>> Q(tags__0_2__contains=['thoughts'])
>>
>>
>> 2. Previously, it was only possible to call defined method (like it 
>> is done for *is_not_null()* function) or registered with decorator. 
>> Now if you try to call unknown function of sugar Q, it will internally 
>> pass 
>> function name and append it as *__functionname *to lookup field:
>>
>> >>> Q.user.username.contains('Rodriguez')
>> Q(user__username__contains='Rodriguez')
>>
>>
>> There is no such function "contains" anymore in sugar Q, however, it 
>> just adds *__contains* to lookup field and passes parameter to it.
>>
>> 3. It will pass a tuple to lookup if multiple arguments passed:
>>
>> >>> Q.user.create_datetime.range(d1, d2)
>> Q(user__create_datetime__range=(d1, d2))
>>
>>
>> I think the library is at the final state now and isn't going to get 
>> new substantial features, but responses are highly appreciated.
>>
>> Regards,
>> Alexey
>>
>>
>> On Sunday, August 16, 2015 at 4:18:26 PM UTC+3, Alexey Zankevich 
>> wrote:
>>>
>>> Hi all,
>>>
>>> This topic is related to the current ORM query syntax with 
>>> underscores.
>>> There are lots of arguing related to it, anyway it has pros