Re: Django ORM query syntax enhancement

[Alexey Zankevich]

Hey all,

Please check a draft DEP related to external query language support by 
Django ORM https://github.com/django/deps/pull/40. 

Regards,
Alexey


On Wednesday, March 22, 2017 at 10:07:51 AM UTC+3, Asif Saifuddin wrote:
>
> Hi Aric,
>
> I checked your package. it's nice too. thanks for the work.
>
> Asif
>
> On Monday, October 17, 2016 at 4:38:26 AM UTC+6, Aric Coady wrote:
>>
>> +1.  I implemented much the same thing for part of django-model-values 
>> , but went with F 
>> expressions 
>>  as the 
>> basis instead.  Primarily because F expressions already support operator 
>> overloading and are a natural intermediate object;  from there one can 
>> create Q, OrderBy, and Func objects.
>>
>> In []: from model_values import F
>>
>> In []: F.user.created
>> Out[]: FExpr(user__created)
>>
>> In []: F.user.created >= 0
>> Out[]: 
>>
>> In []: F.user.created.min()
>> Out[]: Min(FExpr(user__created))
>>
>> In []: -F.user.created
>> Out[]: OrderBy(FExpr(user__created), descending=True)
>>
>> In []: F.text.iexact('...')
>> Out[]: 
>>
>>
>>
>>
>> On Thursday, October 6, 2016 at 10:04:56 AM UTC-7, Alexey Zankevich wrote:
>>>
>>> Hey all,
>>>
>>> Just want to announce recent changes in Django ORM Sugar library, which 
>>> might be useful in future Django query syntax enhancement (if ever happens).
>>> 1. Now it supports indexes and slices:
>>>
>>> >>> Q.data.owner.other_pets[0].name='Fishy'
>>> Q(data__owner__other_pets__0__name='Fishy')
>>> >>> Q.tags[0] == 'thoughts'
>>> Q(tags__0='thoughts')
>>> >>> Q.tags[0:2].contains(['thoughts']) 
>>> Q(tags__0_2__contains=['thoughts'])
>>>
>>>
>>> 2. Previously, it was only possible to call defined method (like it is 
>>> done for *is_not_null()* function) or registered with decorator. Now if 
>>> you try to call unknown function of sugar Q, it will internally pass 
>>> function name and append it as *__functionname *to lookup field:
>>>
>>> >>> Q.user.username.contains('Rodriguez')
>>> Q(user__username__contains='Rodriguez')
>>>
>>>
>>> There is no such function "contains" anymore in sugar Q, however, it 
>>> just adds *__contains* to lookup field and passes parameter to it.
>>>
>>> 3. It will pass a tuple to lookup if multiple arguments passed:
>>>
>>> >>> Q.user.create_datetime.range(d1, d2)
>>> Q(user__create_datetime__range=(d1, d2))
>>>
>>>
>>> I think the library is at the final state now and isn't going to get new 
>>> substantial features, but responses are highly appreciated.
>>>
>>> Regards,
>>> Alexey
>>>
>>>
>>> On Sunday, August 16, 2015 at 4:18:26 PM UTC+3, Alexey Zankevich wrote:

 Hi all,

 This topic is related to the current ORM query syntax with underscores.
 There are lots of arguing related to it, anyway it has pros and cons.

 Let's take a concrete example of querying a model:

 >>> 
 GameSession.objects.filter(user__profile__last_login_date__gte=yesterday)


 Pros:

 1. The syntax is easy to understand
 2. Can be extended with custom transforms and lookups

 However, there are several cons:

 1. Long strings is hard to read, especially if we have fields with 
 underscores.
 It's really easy to make a mistake by missing one:

 >>> 
 GameSession.objects.filter(user_profile__last_login_date__gte=yesterday)

 Not easy to catch missing underscore between user and profile, is it? 
 Even
 though, it's not easy to say whether it should be "user_profile" 
 attribute or
 user.profile foreign key.

 2. Query strings can't be reused, thus the approach violates DRY 
 principle.
 For example, we need to order results by last_login_date:

 >>> 
 GameSession.objects.filter(user__profile__last_login_date__gte=yesterday) \
 .order_by('user__profile__last_login_date')

 We can't keep user__profile_login_date as a variable as in the first 
 part of the
 expression we use a keyword argument, meanwhile in the second part - 
 just a 
 string. And thus we just have to type query path twice.

 3. Lookup names not natural to Python language and require to be 
 remembered or
 looked up in documentation. For example, "__gte" or "__lte" lookups 
 tend to be
 confused with "ge" and "le" due to similarity to methods "__ge__" and 
 "__le__".

 4. Lookup keywords limited to a single argument only, very inconvenient 
 when
 necessary to filter objects by range.

 I was thinking a lot trying to solve those issues, keeping in mind 
 Django
 approaches. Finally I came up with solution to extend Q objects with dot
 expression syntax:

 >>> GameSession.objecs.filter(Q.user.profile.last_login_date >= 
 yesterday)

 Q is a factory instance for old-style Q objects. Accessing attribute by 
 dot
 returns a chi

Re: A proposal for a new auto-reloader in Django

[qingnian213]

Hi Aymeric,

  Thanks for this detailed and informative response! I will try to figure 
out the difficulties of integrating Watchman during the weekend. For the 
pure-Python solution, I might implement a standalone autoreloader based on 
Carl's work and replace the current one. Does this look good to you?

David Ma

On Thursday, March 30, 2017 at 6:39:42 AM UTC-7, Aymeric Augustin wrote:
>
> Hello, 
>
> > On 29 Mar 2017, at 01:05, qingn...@gmail.com  wrote: 
> > 
> > the best library to replace Watchdog is Watchman, a library that 
> supports both of these three platforms (for Windows it's still in the alpha 
> stage.) 
>
> Django currently doesn't use Watchdog. Watchdog is a cross-platform Python 
> library that can be used for implementing an autoreloader. It's a plausible 
> alternative to Watchman, likely easier to integrate, but Watchman has some 
> fancy features that I'd love to have available -- like waiting for git to 
> finish its operations before notifying a change. 
>
> > It runs as a standalone process and has a python wrapper library that is 
> incompatible with Python 3. Therefore, one of the goals is to figure out 
> how to interact with this library. 
>
> In order to be selected for this project, you have to convince us that you 
> can solve this problem. At a minimum you must prove that you understand 
> where the difficulties lie and suggest some possible approaches to tackle 
> them. 
>
> > So my goals would be: 
> > 
> > • Replace the old auto-reloader with the Watchman version 
>
> Since installing watchman isn't trivial, you need to keep a pure-Python 
> alternative so that beginners get an acceptable experience out of the box. 
> This alternative may be the current "poll every second approach" or could 
> be backed by watchdog, assuming it's pip-installable without complicated 
> dependencies (e.g. a C compiler and development headers for a bunch of 
> libraries) on all operating systems. 
>
> > • Make sure that the new auto-reloader can work with Python3 and 
> on other platforms 
>
> Did you mean "work with Python 2" ? In any case, this isn't necessary, as 
> you'd be working off the master branch which no longer supports Python 2. 
>
> > • Make it compatible with other Python libraries 
> > • Allow Django users to choose the files that should be watched 
> > • Integrate the Watchman package and its Python wrapper into 
> Django 
>
> Watchman should be installed separately by developers who wish to use it. 
> Django should use it when it's available (and there should be a flag to 
> disable that if for any reason someone has watchman installed systemwide 
> but doesn't want to use it). 
>
> I hope this helps! 
>
> -- 
> Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b8f5740c-a154-4b9b-90f1-9a79db712f0c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Adding signals to bulk update/create operations

[Todor Velichkov]

@Tim, sorry about that, I did a search before I posted, but it looks like 
it slipped away somehow.

@Anssi, thank you for your response.
I really haven't think about threat safety, but maybe its because in our 
specific usage it's close to impossible to happen.

What do you think about this:
1) Put everything into a transaction
2) Before update, count the queryset.
3) Fire the pre_update with the queryset, and the count.
4) Do the actual update -> get the updated rows
5) Assert the updated rows is equal to the queryset.count().
6) Revert the transaction if there is a difference.

No pk fetching, this is left to the be implemented by the listener if he 
needs it.

On Friday, March 31, 2017 at 8:01:45 AM UTC+3, Anssi Kääriäinen wrote:
>
> The problem with passing the queryset is that it's possible that some 
> object is added to or removed from the queryset between the pre_update and 
> actual update execution. To avoid this the execution should go somewhere 
> along the lines of:
>1) if there is pre_update or post_update do stages 2-5, if not, update 
> as with current code
>2) fetch primary keys for models to be updated to a set, with 
> .select_for_update() applied
>3) fire pre_update, give the primary key set as argument
>4) do the update against a queryset with .filter(pk__in=pk_set)
>5) fire post_update with primary key set as argument
>
> This way the pre and post update signals will execute against a fixed set 
> of instances. The bad part is that this can significantly slow down the 
> .update() call, but to get actually safe signals, I don't see a way around 
> that.
>
>  - Anssi
>
> On Friday, March 31, 2017 at 3:51:34 AM UTC+3, Tim Graham wrote:
>>
>> There's an accepted ticket about adding pre_update and post_update 
>> signals: https://code.djangoproject.com/ticket/21461. From a quick 
>> glance, I think this is what you're proposing.
>>
>> On Thursday, March 30, 2017 at 4:28:00 PM UTC-4, Todor Velichkov wrote:
>>>
>>> Consider the following piece of code:
>>>
>>> @receiver(pre_save, sender=MyModel)
>>> def my_handler(sender, **kwargs):
>>> instance = kwargs['instance']
>>> if instance.verified:
>>> do_something(instance)
>>> else:
>>> do_else(instance)
>>>
>>>
>>>
>>> Its good, because it keeps `MyModel` decoupled from `do_something`
>>>
>>> But there is one flaw. If we do:
>>>
>>> MyModel.objects.filter(verified=False).update(verified=True)
>>>
>>> we are screwed, `do_something` is not executed, and our models get 
>>> out-of-sync.
>>>
>>> If we try to get smart and manually fire the pre_save signal for each 
>>> instance, we are gonna have a hard time.
>>> Its gonna be slow.
>>> And its gonna be memory inefficient.
>>>
>>> We already experienced it in our app.
>>>
>>> So our new approach is like this:
>>>
>>> pre_bulk_update = Signal(providing_args=["queryset", "update_kwargs"
>>> ])
>>> post_bulk_update = Signal(providing_args=["update_kwargs",])
>>>
>>> pre_bulk_create = Signal(providing_args=["objs", "batch_size"])
>>> post_bulk_create = Signal(providing_args=["objs", "batch_size"])
>>>
>>>
>>> class MyModelQuerySet(models.QuerySet):
>>> def update(self, **kwargs):
>>> pre_bulk_update.send(sender=self.model, queryset=self, 
>>> update_kwargs=kwargs)
>>> res = super(MyModelQuerySet, self).update(**kwargs)
>>> # The queryset will be altered after the update call
>>> # so no reason to send it.
>>> post_bulk_update.send(sender=self.model, update_kwargs=
>>> kwargs)
>>> return res
>>>
>>> def bulk_create(self, objs, batch_size=None):
>>> pre_bulk_create.send(sender=self.model, objs=objs, 
>>> batch_size=batch_size)
>>> res = super(MyModelQuerySet, self).bulk_create(objs, 
>>> batch_size)
>>> post_bulk_create.send(sender=self.model, objs=objs, 
>>> batch_size=batch_size)
>>> return res
>>>
>>>
>>> class MyModel(models.Model):
>>> #...
>>> objects = MyModelQuerySet.as_manager()
>>>
>>>
>>>
>>> This gives us a nice interface to handle all kind of changes regarding 
>>> `MyModel`
>>> Our example usage looks like this:
>>>
>>>
>>> @receiver(pre_save, sender=MyModel)
>>> def my_handler(sender, **kwargs):
>>> instance = kwargs['instance']
>>> if instance.verified:
>>> do_something(instance)
>>> else:
>>> do_else(instance)
>>>
>>>
>>> @receiver(pre_bulk_update, sender=MyModel)
>>> def my_bulk_update_handler(sender, **kwargs):
>>> update_kwargs = kwargs['update_kwargs']
>>> if 'verified' not in update_kwargs:
>>> # no change im interested in
>>> # no need to take any action
>>> return
>>>
>>> queryset = kwargs['queryset']
>>> pks_to_be_updated = queryset.values_list('pk', flat=True)
>>> if update_kwargs['verified']:
>>> do_something_bulk_update_implementation(pks_to_be_updated)
>>> else:
>>>

Re: A proposal for a new auto-reloader in Django

[Melvyn Sopacua]

On Thursday 30 March 2017 22:01:00 qingnian...@gmail.com wrote:
> Hi Carl,
>   Thanks for mentioning this awesome project! I saw it in one of the
> discussions but did not take a close look. I'll definitely check this
> out and try to integrate wsgiwatcher/watcher.py into Django.
> 
> On Thursday, March 30, 2017 at 7:47:13 AM UTC-7, Carl Meyer wrote:
> > Anyone working on this project should at least be aware of
> > https://github.com/Pylons/hupper
> >  > pper&sa=D&sntz=1&usg=AFQjCNGVwtqvdo53UFfK80kaQ1qxL7ST8Q> (based 
on
> > work David Glick and I
> > originally did in https://github.com/carljm/wsgiwatcher
> >  > giwatcher&sa=D&sntz=1&usg=AFQjCNGqLfsmxsp37Lng7_d_DaVzja9c6Q>),
> > which aims to
> > be a framework-agnostic solution to this problem for any Python web
> > project. Docs at
> > http://docs.pylonsproject.org/projects/hupper/en/latest/
> > 
> > Carl
I was going to suggest the same, because you already get watchdog support for 
free 
and can probably build on that integration to pick up watchman. Also, it's used 
in 
devpi[1], so it gets some scruteny from there (and devpi is awesome to 
distribute 
internal or augmented Django apps amongst projects).
-- 
Melvyn Sopacua


[1] http://doc.devpi.net/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1871865.SgAvvE3WaE%40devstation.
For more options, visit https://groups.google.com/d/optout.