Re: Proposal to format Django using black

2019-04-17 Thread Tom Forbes
I would be +1 for Black. I think it makes a lot of sense for a project like 
Django, and it does seem that a non trivial amount of both contributor and 
reviewer time is spent on formatting fixes.

The choice of double quotes by default used to annoy me, but after using black 
for a while I think that the benefits outweigh the downsides.

One thing we have not considered here is that after running black on Django a 
huge portion of our outstanding merge requests will have conflicts, some of 
which might be tricky to rebase. I’m not sure there is much we can do about 
that though.

Tom

> On 17 Apr 2019, at 09:26, Tobias Kunze  wrote:
> 
> Hi Dan,
> 
>> On 19-04-16 20:33:29, Dan Davis wrote:
>> +1 isort
>> -1 black
>> 
>> I think that codestyle checkers are better, because you teach yourself
>> proper style for python.
> 
> I appreciate this argument, but: As Django community our primary concern
> in this discussion has to be the impact black would have on the Django
> code base and the Django development process – educating our
> contributors cannot be a primary concern, compared to making
> contributions as easy as possible.
> 
> Tobias
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/20190417082648.miicsigxjwoevmkl%40cordelia.localdomain.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/AFFEB466-A328-4DA6-8845-F2DFB55C1ED1%40tomforb.es.
For more options, visit https://groups.google.com/d/optout.


Re: Why does ModelForm do validation and not Model

2019-04-16 Thread Tom Forbes
The idea is that you generally always have to do extensive validation when
accepting user input through a form. These validations could require
additional database queries or other somewhat expensive lookups (especially
with validate unique).

However if you are loading data from a trusted source, e.g:

for row in your_csv_file:
instance = Model(**row)
instance.save()

Then there is no need to call that potentially slow full_clean(). There is
not much value in slowing down all .save()’s needlessly - the developer
should know when it’s appropriate to run validations and can run
full_clean() when needed.




On 16 April 2019 at 21:42:24, Will Gordon (wpg4...@gmail.com) wrote:

So the validation is cheaper when performed by ModelForm, as opposed to the
Model?
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/8d929b8e-b0e9-4a88-b796-26f00266f729%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMUPSptux9yeSVtcHLi2HR80YfzgXmhMqnWQ_eEMYqqdA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Add to QuerySet method `one` that acts as `first` but without ordering

2019-04-13 Thread Tom Forbes
I don’t think we should add a method like this for a few reasons. Firstly 
without an order by in SQL the order of rows is undefined, in practice myql 
orders rows but Postgres returns a different order per transaction. This will 
be confusing to users who don’t understand this and come to implicitly rely on 
first() being stable.

Secondly if you are filtering on an indexed column the overhead will be next to 
none. Is this not the case?

And lastly, changing this would be major backwards incompatible change.

Tom

> On 13 Apr 2019, at 21:58, alTus  wrote:
> 
> I've posted the source code if `first`. You can see there that if qs is not 
> ordered then ordering by pk is added.
> It's the main point of this issue btw.
> 
> суббота, 13 апреля 2019 г., 22:53:52 UTC+3 пользователь Florian Apolloner 
> написал:
>> 
>> qs.order_by().first() should do what you want and is not worth adding .one()
>> 
>> Cheers,
>> Florian
>> 
>>> On Saturday, April 13, 2019 at 5:48:29 PM UTC+2, alTus wrote:
>>> Hello.
>>> 
>>> Please consider if that proposal can be useful not only for me :)
>>> 
>>> `QuerySet.first` is quite useful shortcut but its drawback is that it 
>>> involves ordering so some DB queries become expensive.
>>> But quite often (and very often while debugging) there's a need just to get 
>>> one object that satisfy some filters.
>>> 
>>> Current `first` implementation to compare with:
>>> 
>>> def first(self):
>>> """Return the first object of a query or None if no match is 
>>> found."""
>>> for obj in (self if self.ordered else self.order_by('pk'))[:1]:
>>> return obj
>>> 
>>> Proposal (as an example of implementation):
>>> 
>>> def one(self):
>>> """Return one random object of a query or None if no match is 
>>> found."""
>>> for obj in self.order_by()[:1]:
>>> return obj
>>> 
>>> That would be short, simple and wouldn't require any imports in client code.
>>> 
>>> Thank you.
>>> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/fc6e71e5-a9b5-4cb5-9283-de1130ff0b2e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/B83882F9-E932-4258-93D7-100A3D4A933D%40tomforb.es.
For more options, visit https://groups.google.com/d/optout.


Re: Request to reconsider ticket #27910: using an Enum class in model Field choices

2019-04-13 Thread Tom Forbes
I really like this implementation, it seems really clean and simple.

I would suggest that it might be worth seeing it’s simple to make the
display text optional, I think in the general case capitalising the key
name would be acceptable. Users can always override it if required or if
they need translation support.

Even if we decide to make the display names required in all cases, it’s a
great improvement over the current way of defining choices IMO.

Tom




On 13 April 2019 at 10:33:39, Shai Berger (s...@platonix.com) wrote:

Back to this issue for the DjangoCon EU sprints...

James' objection about the inclusion testing needed for validation
seems moot, because validation can be done by conversion to the enum
type (as noted by Tom).

Raphael's warning about the woes of translation are already handled by
the suggested implementation -- because it isn't the enum value that
we make translatable, but rather an attribute on it.

I should point that the suggested implementation uses IntEnum and
StrEnum. The Python documentation recommends against using these,
except in the case that one needs to account for compatibility with an
existing protocol -- which, I submit to you, is exactly our case (the
"protocol" being the types available for database fields).

As a reminder, the relevant links are:

https://code.djangoproject.com/ticket/27910
https://github.com/django/django/compare/master...shaib:ticket_27910?expand=1

Thanks,
Shai.

-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/20190413113323.1342ed7d.shai%40platonix.com.

For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMMh6nH8VdAozeVnLbQXanqB5b864Ro1g%2BGm2N6hWM%3DgQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Official Django Docker Container Deprecated

2019-02-28 Thread Tom Forbes
I think the point we are trying to make is that it’s fundamentally not a
good thing to try and distribute a one-size fits all docker image for a
specific framework.

For reference here is one you can use yourself:

FROM python:3
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD [ "gunicorn", "my.app" ]

If someone is unable to make an equivalent Dockerfile then they will be
really confused when they realise that they need to customise it, because
few projects are as simple as that.

You should also likely not embed Apache inside your app container - it’s
kind of missing the general idea of Docker.

To re-iterate: The Django project had no hand in creating the ‘official’
image. The Docker project retired the original Django image for reasons
that are clearly explained here , and
those reasons still hold today.

Tom




On 28 February 2019 at 12:56:33, Alexander Lyabah (a.lya...@checkio.org)
wrote:

I can make a version for production use (in a week or two), for your
critics.

For example, based on Appache wsgi.

PS: maybe it is also worth to make a docker image for testing changes in
Django source?

On Wednesday, February 27, 2019 at 4:31:17 PM UTC+2, Jamesie Pic wrote:
>
> > most people currently lean towards a microservice architecture and
> therefore towards flask.
> "according to the 2018 JetBrains Developer Survey" and some people.
> Why start a project with flask in 2019 instead of Quart which or
> Starlette is another question that I suppose is out of the scope of
> this mailing list.
>
> Anyway, the point of Docker is to build your own image that supports
> both development and production given different runtime parameters.
> The agile practice with docker is to build your immutable image in CI,
> test it, deploy it to staging, have on-click deployment to production.
>
> The security and best practice documentation from docker are indeed a
> lot to grasp, and beginners will most of the time start making
> insecure (running as root) and inefficient (fat) images. Therefore for
> their security Django might want to document making a docker file,
> perhaps based on the alpine image that's the most lightweight.
>
> --
> ∞
>
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/f07ad32e-e74f-4cd3-945a-ed92692c2209%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJP5k%3Dsn8VkdwseuVqnMC6X40y349urNMRFZfD7FEypvbg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Official Django Docker Container Deprecated

2019-02-26 Thread Tom Forbes
There never was an official Django image, it was an "official docker" 
Django image that they maintained. The page image page explains why it was 
deprecated: https://hub.docker.com/_/django

For most usages of this image, it was already not bringing in django from 
> this image, but actually from your project's requirements.txt, so the 
> only "value" being added here was the pre-installing of mysql-client, 
> postgresql-client, and sqlite3 for various uses of the django framework.
>

I fully agree with this, it made little sense as an image at all.

On Tuesday, 26 February 2019 07:39:34 UTC, Alexander Lyabah wrote:
>
> Hi guys,
>
> I found out that official django container is deprecated. Why you don't 
> want to support it?
>
> When you search "django" in Python subreddit 
> https://monosnap.com/file/R3uAqdrcrtxjCyKgHhhe7B8lO3up5q
>
> you see "Flask has overtaken Django according to the 2018 JetBrains 
> Developer Survey" 
> https://www.reddit.com/r/Python/comments/ao5dml/flask_has_overtaken_django_according_to_the_2018/
>
> and the main point in this discussion that it is hard to start from 0 to 
> real project.
>
> That why I was surprised that you don't have a simple docker container 
> with configured django with apache or/and django with nginx . It can save 
> so much time for people who just want to launch a simple pet project with 3 
> visitors per week.
>
> Thank you, I love Django and wish you be better and bigger, have a great 
> week.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/db603927-bed9-4734-8e52-a0500c9d69e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Django 2.2 and the watchman reloader

2019-02-25 Thread Tom Forbes
I have a small PR here to remove the “watchman unavailable” message, whilst
keeping the one that specifies which reloader we are using:
https://github.com/django/django/pull/11025.




On 21 February 2019 at 20:52:29, Claude Paroz (cla...@2xlibre.net) wrote:

Le jeudi 21 février 2019 21:43:43 UTC+1, Tom Forbes a écrit :
>
> Hey Claude,
> Thanks for your feedback on the feature, I fully agree with you. I think
> we should remove that warning message about the missing package. I will
> make a PR to do that.
>

I'm not completely sure it's a good idea to entirely remove the message.
Maybe just telling the used reloader would be fine.


> Regarding creating another reloader: it should not be that difficult to do
> at all since we have all the other pieces in place. In theory it's just
> implementing a class with single generator method.
>
> If people agree I would like to use the 'watchdog' package for this rather
> than the pyinotify library as it would be quicker to implement, a lot nicer
> to work with and is easier to test.
>

++1, watchdog is better maintained, looks like pyinotify is dead. Tell me
if you need help, even if you seems a lot more knowledgeable than me on the
subject.

Thanks.

Claude


> On Thu, 21 Feb 2019, 19:56 Claude Paroz,  wrote:
>
>> Le jeudi 21 février 2019 19:48:31 UTC+1, Dan Davis a écrit :
>>>
>>> Claude,
>>>
>>> I've tested a Django based application on 2.2b1 without watchman on
>>> Windows, it does tell you about watchman, but it doesn't fail to run.
>>> Apparently, it falls back to the old way of reloading.   Is this not the
>>> behavior on Debian/Ubuntu?
>>>
>>
>> Yes it is. I would say this is still a slight regression in two ways:
>>
>> - no messages told you the reload method was not optimal before. So now
>> people will try to "fix" their system, more than before.
>> - for Debian-based systems, you could improve the reloading performance
>> by installing system or pip packages in 30 seconds. Now you have to spend
>> 30 minutes to search how watchman can be installed and to compile the
>> package (+ you have to care yourself for any security issue).
>>
>> Claude
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-develop...@googlegroups.com.
>> To post to this group, send email to django-d...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit https://groups.google.com/d/
>> msgid/django-developers/b059553a-e25e-4d90-beed-
>> bf7e0f797305%40googlegroups.com
>> <https://groups.google.com/d/msgid/django-developers/b059553a-e25e-4d90-beed-bf7e0f797305%40googlegroups.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/6de70f64-9bbb-4017-bd10-c85e94ae08d0%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/6de70f64-9bbb-4017-bd10-c85e94ae08d0%40googlegroups.com?utm_medium=email_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOOELOoSEGGx9LrMryrYFhqm%2Bzd_PPZ-ztarjiZyR%2Bvnw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Use CDN for djangoproject.com

2019-02-24 Thread Tom Forbes
Awesome work! For my location (Lisbon, Portugal) it takes about 130ms to
retrieve the HTML for a docs page (
https://django-docs.global.ssl.fastly.net/en/2.1/intro/reusable-apps/ to be
specific). The same page on docs.djangoproject.com responds in 800–900ms.




On 24 February 2019 at 14:35:55, Tobias McNulty (tob...@caktusgroup.com)
wrote:

Hi Tom,

Thanks for your message. I think we'll end up with Fastly since it would be
free, but I'm waiting to see their sponsorship contract. CloudFront would
work too but I don't know of any such open source sponsorship options with
AWS.

I will say wildcard purging looks a bit simpler in CloudFront, but your
idea purging the whole cache only for non-dev builds could work (provided
we have a lower cache timeout or a single wildcard purge condition set up
for the dev builds, I guess).

Feel free to test and post any feedback about Fastly prior to the potential
transition here: https://django-docs.global.ssl.fastly.net/en/2.1/ (this is
set up on a free dev account, so no custom SSL)

For the sake of comparison I'm working on getting a distribution set up for
CloudFront too, but it won't be so simple to test (without a DNS or server
configuration change) since I don't think CloudFront supports passing a
custom Host header to the origin like Fastly does (i.e., you'll probably
need to edit /etc/hosts).

Cheers,
Tobias


On Sat, Feb 23, 2019, 7:15 PM Tom Forbes  wrote:

> Sorry, I did not completely grok your message. I would be in favour of
> just invalidating the whole cache if needed, it seems the simplest
> solution. Invalidating most of the cache on every non-dev deploy would also
> be OK I think.
>
> On Sun, 24 Feb 2019, 00:10 Tom Forbes,  wrote:
>
>> Which CDN are we going to use? Fastly has awesome sub 100ms global
>> invalidation which we can trigger on every deploy, and cloudflare has
>> something similar.
>>
>> On Sun, 24 Feb 2019, 00:00 Tobias McNulty, 
>> wrote:
>>
>>> Hi all,
>>>
>>> An implementation question has come up regarding cache lifetime (see this
>>> PR <https://github.com/django/djangoproject.com/pull/870>). Right now,
>>> the whole site (including docs) has the site-wide Django cache enabled
>>> <https://github.com/django/djangoproject.com/blob/master/djangoproject/settings/prod.py#L43-L45>,
>>> with a timeout of 5 minutes
>>> <https://github.com/django/djangoproject.com/blob/master/djangoproject/settings/common.py#L27>.
>>> A couple docs views (search_suggestions
>>> <https://github.com/django/djangoproject.com/blob/master/docs/views.py#L248-L250>
>>> and search_description
>>> <https://github.com/django/djangoproject.com/blob/master/docs/views.py#L272-L274>)
>>> views have longer timeouts set (to 1 hour and 1 week, respectively).
>>>
>>> Once released, the vast majority of Django docs won't change much,
>>> except for the release notes section and any (likely minor) related updates
>>> to the docs themselves. To get the most benefit out of a CDN, it would
>>> obviously be desirable to set the timeout to something greater than 5
>>> minutes.
>>>
>>> At the same time, there are moments when a quick update to the docs *is*
>>> desired, and waiting an hour or more for any cached pages to expire may
>>> cause significant confusion, for example, in conjunction with a security
>>> release for which stubbed (non-final) release notes may have already been
>>> pushed out and cached.
>>>
>>> I see two main options at this point (which could even be combined):
>>>
>>> 1) Invalidate the whole cache (or at least some key release notes URLs)
>>> any time there's a docs build that has changes. It would be pretty easy to
>>> piggyback off of the existing business logic for avoiding a rebuild
>>> <https://github.com/django/djangoproject.com/blob/master/docs/management/commands/update_docs.py#L86-L93>
>>> if the git checkout hasn't changed (in the update_docs management command).
>>> 2) Pick subsections of the docs (e.g., for anything matching
>>> '///releases/*' and perhaps the development docs) that would
>>> keep a shorter cache timeout of 5-10 minutes. All URLs not specifically
>>> requiring this special treatment would get a longer timeout, perhaps
>>> somewhere between 1 and 24 hours.
>>>
>>> So, some questions for the list:
>>>
>>> * Are there sections of the docs besides '///releases/'
>>> and '//dev/' that might update frequently and merit some combination
>>> of invalidation and/or a shorter cache time? And what's a good cache
>>> timeout for such pages?
>

Re: Use CDN for djangoproject.com

2019-02-23 Thread Tom Forbes
Sorry, I did not completely grok your message. I would be in favour of just
invalidating the whole cache if needed, it seems the simplest solution.
Invalidating most of the cache on every non-dev deploy would also be OK I
think.

On Sun, 24 Feb 2019, 00:10 Tom Forbes,  wrote:

> Which CDN are we going to use? Fastly has awesome sub 100ms global
> invalidation which we can trigger on every deploy, and cloudflare has
> something similar.
>
> On Sun, 24 Feb 2019, 00:00 Tobias McNulty,  wrote:
>
>> Hi all,
>>
>> An implementation question has come up regarding cache lifetime (see this
>> PR <https://github.com/django/djangoproject.com/pull/870>). Right now,
>> the whole site (including docs) has the site-wide Django cache enabled
>> <https://github.com/django/djangoproject.com/blob/master/djangoproject/settings/prod.py#L43-L45>,
>> with a timeout of 5 minutes
>> <https://github.com/django/djangoproject.com/blob/master/djangoproject/settings/common.py#L27>.
>> A couple docs views (search_suggestions
>> <https://github.com/django/djangoproject.com/blob/master/docs/views.py#L248-L250>
>> and search_description
>> <https://github.com/django/djangoproject.com/blob/master/docs/views.py#L272-L274>)
>> views have longer timeouts set (to 1 hour and 1 week, respectively).
>>
>> Once released, the vast majority of Django docs won't change much, except
>> for the release notes section and any (likely minor) related updates to the
>> docs themselves. To get the most benefit out of a CDN, it would obviously
>> be desirable to set the timeout to something greater than 5 minutes.
>>
>> At the same time, there are moments when a quick update to the docs *is*
>> desired, and waiting an hour or more for any cached pages to expire may
>> cause significant confusion, for example, in conjunction with a security
>> release for which stubbed (non-final) release notes may have already been
>> pushed out and cached.
>>
>> I see two main options at this point (which could even be combined):
>>
>> 1) Invalidate the whole cache (or at least some key release notes URLs)
>> any time there's a docs build that has changes. It would be pretty easy to
>> piggyback off of the existing business logic for avoiding a rebuild
>> <https://github.com/django/djangoproject.com/blob/master/docs/management/commands/update_docs.py#L86-L93>
>> if the git checkout hasn't changed (in the update_docs management command).
>> 2) Pick subsections of the docs (e.g., for anything matching
>> '///releases/*' and perhaps the development docs) that would
>> keep a shorter cache timeout of 5-10 minutes. All URLs not specifically
>> requiring this special treatment would get a longer timeout, perhaps
>> somewhere between 1 and 24 hours.
>>
>> So, some questions for the list:
>>
>> * Are there sections of the docs besides '///releases/'
>> and '//dev/' that might update frequently and merit some combination
>> of invalidation and/or a shorter cache time? And what's a good cache
>> timeout for such pages?
>> * How long are we comfortable waiting for *other* (not frequently
>> updated) pages to timeout, in the event they do change?
>>
>> Tobias
>>
>> On Fri, Feb 15, 2019 at 7:13 AM Tobias McNulty 
>> wrote:
>>
>>> Thanks for sharing the results.
>>>
>>> I did manage to get a domain set up with working SSL, in case you want
>>> to use it: https://django-docs.global.ssl.fastly.net/en/2.1/
>>>
>>> Tobias
>>>
>>> On Thu, Feb 14, 2019, 11:49 PM Cheng C >>
>>>> Thanks for the test site, Tobias.
>>>>
>>>> Tested from Melbourne, Australia:
>>>>
>>>> https://docs.djangoproject.com/en/2.1/
>>>> Average Ping: 268ms
>>>>  Browser: 22 requests, 238KB transferred, Finish: 2.72s,
>>>> DOMContentLoaded: 1.37s, Load: 1.68s
>>>>
>>>> https://docs.djangoproject.com.global.prod.fastly.net/en/2.1/
>>>> Average Ping: 28ms
>>>>  Browser: 22 requests, 240KB transferred, Finish: 947ms,
>>>> DOMContentLoaded: 627ms, Load: 910ms
>>>>
>>>> Tested on Chrome with "Disable cache" checked, and no render issue was
>>>> found.
>>>>
>>>> On Friday, February 15, 2019 at 2:09:09 PM UTC+11, Tobias McNulty wrote:
>>>>>
>>>>> Adam, is there another provider you would recommend instead, that does
>>>>> not require changing DNS providers? FWIW, python.org does in fact use
>>>>> Fastly:
>

Re: Use CDN for djangoproject.com

2019-02-23 Thread Tom Forbes
es:
>>>> https://docs.djangoproject.com.global.prod.fastly.net/en/2.1/ (Not for
>>>> permanent use, obviously; you'll get a cert warning, and some pages will
>>>> redirect you back to https://docs.djangoproject.com.)
>>>>
>>>> To keep this thread from getting too noisy, you can find me (tobias1)
>>>> in #django-dev on FreeNode.
>>>>
>>>> Cheers,
>>>> Tobias
>>>>
>>>> On Thu, Feb 14, 2019 at 8:26 AM Adam Johnson  wrote:
>>>>
>>>>> I have not had great experience with Fastly in the past and would
>>>>> avoid them. They run an old fork of Varnish which is not fun to configure.
>>>>>
>>>>> On Thu, 14 Feb 2019 at 11:16, Josh Smeaton 
>>>>> wrote:
>>>>>
>>>>>> Cloudflare have many SSL options, including fully encrypted and
>>>>>> authenticated comms all the way through (terminate and reconnect).
>>>>>> Typically done by having a “hidden” origin domain that also hosts a
>>>>>> certificate. I’m unsure if it’s possible to have both origin and front
>>>>>> hosting the same name so that DNS alone can decide to hit cdn or origin.
>>>>>>
>>>>>> Anyway, it seems weird to me to dismiss a CDN offhand “because
>>>>>> security”. Especially considering the size of the providers and the
>>>>>> expertise their teams have.
>>>>>>
>>>>>> Cloudflare (fastly, cloudfront, whatever) aren’t some “random TLS”
>>>>>> providers. I would probably go as far to say that putting a CDN in front 
>>>>>> of
>>>>>> both the docs and the release packages would likely be a net improvement 
>>>>>> in
>>>>>> security for users.
>>>>>>
>>>>>> On Thu, 14 Feb 2019 at 21:58, Tom Forbes  wrote:
>>>>>>
>>>>>>> That makes sense, but in this case we are only talking about
>>>>>>> potentially yielding control of the docs subdomain which is not used to
>>>>>>> serve sensitive build artefacts?
>>>>>>>
>>>>>>> Another option is fastly.com, who support other large open source
>>>>>>> projects for free. They essentially give you geographically distributed
>>>>>>> HAProxy instances and you have a lot more control over them. I believe
>>>>>>> several large Linux distributions use them to serve cached apt packages.
>>>>>>>
>>>>>>> Regarding TLS termination, unfortunately any CDN we use will likely
>>>>>>> need to do this for the whole domain to get any benefit. The Django docs
>>>>>>> are text/html heavy with very few, if any, images. So the real speed
>>>>>>> benefit will have to come from serving that, which requires TLS 
>>>>>>> termination
>>>>>>> (and therefore interception) at their end.
>>>>>>>
>>>>>>> On Thu, 14 Feb 2019, 06:32 Markus Holtermann, <
>>>>>>> in...@markusholtermann.eu> wrote:
>>>>>>>
>>>>>>>> Hi all
>>>>>>>>
>>>>>>>> to elaborate on what Tobias said: we deliberately have the
>>>>>>>> infrastructure spread across multiple service providers: DNS registry,
>>>>>>>> nameservers, hosting, TLS certificate authority, … None of them have 
>>>>>>>> access
>>>>>>>> to everything. The reason is that we offer the download of the release
>>>>>>>> artifacts from the djangoproject.com website. And we would like to
>>>>>>>> ensure that the TLS termination happens by us and not some random 
>>>>>>>> service
>>>>>>>> provider. After all, Django is used by enterprises that do have some
>>>>>>>> restrictions on where you're allowed to download software from.
>>>>>>>>
>>>>>>>> By handing over DNS to some CDN provider, we loose the ability to
>>>>>>>> ensure that happens.
>>>>>>>>
>>>>>>>> That said, if there's a CDN that works as a reverse proxy and
>>>>>>>> doesn't require us to hand over control of DNS, I guess we could be
>>>>>>>> interested in m

Re: Django 2.2 and the watchman reloader

2019-02-21 Thread Tom Forbes
Hey Claude,
Thanks for your feedback on the feature, I fully agree with you. I think we
should remove that warning message about the missing package. I will make a
PR to do that.

Regarding creating another reloader: it should not be that difficult to do
at all since we have all the other pieces in place. In theory it's just
implementing a class with single generator method.

If people agree I would like to use the 'watchdog' package for this rather
than the pyinotify library as it would be quicker to implement, a lot nicer
to work with and is easier to test.

On Thu, 21 Feb 2019, 19:56 Claude Paroz,  wrote:

> Le jeudi 21 février 2019 19:48:31 UTC+1, Dan Davis a écrit :
>>
>> Claude,
>>
>> I've tested a Django based application on 2.2b1 without watchman on
>> Windows, it does tell you about watchman, but it doesn't fail to run.
>> Apparently, it falls back to the old way of reloading.   Is this not the
>> behavior on Debian/Ubuntu?
>>
>
> Yes it is. I would say this is still a slight regression in two ways:
>
> - no messages told you the reload method was not optimal before. So now
> people will try to "fix" their system, more than before.
> - for Debian-based systems, you could improve the reloading performance by
> installing system or pip packages in 30 seconds. Now you have to spend 30
> minutes to search how watchman can be installed and to compile the package
> (+ you have to care yourself for any security issue).
>
> Claude
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/b059553a-e25e-4d90-beed-bf7e0f797305%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMK_MCbKByWkVKC7QY8_As30%2BnKA4rR1c9UNqWLhUX60g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: BitBounce Spam Replies From the Mailing List

2019-02-17 Thread Tom Forbes
I figured I’d email their CEO (stewart.den...@bitbounce.com) and ask if he
can look into this, because it’s kind of ridiculous. I think I should have
known beforehand what kind of automated reply I got…

I’ve also marked them as spam and so don’t receive them anymore but I can
imagine it’s pretty annoying for anyone that has not.




On 17 February 2019 at 23:50:22, Kye Russell (m...@kye.id.au) wrote:

https://twitter.com/stewart__dennis/status/1081973497025331201?s=21

I’m sure that falsely replying to mailing list emails helps with these
numbers.

I’m just going ahead and marking them as spam, because that’s what they
are. It’s negligence at this point.

On Mon, 18 Feb 2019 at 4:35 am, Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> Since Twitter is the only place where BitBounce responded, I tried again:
> https://twitter.com/aymericaugustin/status/1097231848973967362
>
> I'm skeptical about their willingness to fight spam: they're using it as
> their primary marketing channel. The more we're talking about them, the
> happier they are...
>
> --
> Aymeric.
>
>
>
> On 29 Jan 2019, at 11:30, Patryk Zawadzki  wrote:
>
> Sorry for resurrecting but this is still very much a problem. Same person,
> same autoresponder.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/41722075-8e89-4777-8872-ee4660f14b26%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/4C847606-C965-4C61-8851-63DA676381CF%40polytechnique.org
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CANK-ykmivQ4MOfPnWOEHKN_Cd2CT0S39_9doWrDVh3ZfsxDEQA%40mail.gmail.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJPHJoQkOnzu61-SzHqK%3Dd7Di6ZhnjcR0u5Kq5uAuVpJjg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Google Summer of Code 2019

2019-02-14 Thread Tom Forbes
Out of interest, do the projects for GSOC have to fall within the Django 
repository themselves? Could they be an associated project under the Django 
umbrella?

I think something that helps with deployment in general could be interesting, 
but it definitely does not live inside Django core. However some form of 3rd 
party package exploring this is another matter entirely.

On 14 February 2019 at 17:56:04, Carlton Gibson (carlton.gib...@gmail.com) 
wrote:

Hmmm. I think that would be out for scope for Django. More suited to tools such 
as Ansible (and similar). I don't think adding a one step deployment story 
would really be feasible within a GSoC project. (As you say, it's very 
complex.) 

On Thursday, 14 February 2019 18:46:52 UTC+1, Shashank shet wrote:
Thank you Carlton. I had in mind an idea for providing out-of-the-box webserver 
deployment capability. In the projects I've done, one of the most time 
consuming tasks for us was deployment on a server, mainly due to a lack of 
experience. That's why I figured that an automated system for deployment would 
be helpful. It would work similar to the runserver command, taking essential 
configurations from a designated file and not having to actually make changes 
to files in different parts of the filesystem, or following a long list of 
instructions manually. Any suggestion would be very helpful.


On Thursday, February 14, 2019 at 6:32:17 PM UTC+5:30, Carlton Gibson wrote:
Hi Shashank. 

"Sir"? That's my father, surely.  (Not "sir" ) 

We've applied as an Org. Haven't heard yet whether we'll be accepted. 

If so, proposals would be welcome. We'd then need to think about mentors. Good 
proposals will draw out people there. 

If we can get all that lined up then, yes, in principle we're accepting 
applications. 

Kind Regards,

Carlton


On Thursday, 14 February 2019 13:46:57 UTC+1, Shashank shet wrote:
Hello Carlton sir,
I have worked on Django as a part of a project and wished to ask if the 
organization is still accepting proposals for GSoC 2019.

On Monday, February 4, 2019 at 8:25:42 PM UTC+5:30, Carlton Gibson wrote:
And thank you Tim, yes, exactly what I need. 

To all: 

I will put in the org application today. We'll then see about the proposals. 

The main thing is that we need to know who you are, and have confidence in you 
in order to commit to the project with you. 
For people to mentor you is a big commitment of time and energy. You need to 
demonstrate that will be well invested. 

The way to do that is to get involved and show us who you are over the next few 
months. 
Help reproduce bugs, review patches, create patches etc. 
It doesn't take much before you're visible. (Really!) 

The best way (also) to come up with project ideas is to see where there are 
issues already. 
(Much better than us providing a list.) So again, be involved. 

If you start now, there's still lots of time, so I'm hoping. 

Kind Regards,

Carlton





On Monday, 4 February 2019 15:43:39 UTC+1, Tim Graham wrote:
It's a private wiki that only Django team members like Carlton can access. It 
contains the information for Django to apply for GSoC.

On Monday, February 4, 2019 at 9:07:40 AM UTC-5, Giuseppe De Marco wrote:
Hi Tim,

It returns 404 to me
https://github.com/django/django-team-wiki/wiki/Google-Summer-of-Code-Application-Info

Il giorno lun 4 feb 2019 alle ore 13:05 Tim Graham  ha 
scritto:
All answers are at 
https://github.com/django/django-team-wiki/wiki/Google-Summer-of-Code-Application-Info
--
You received this message because you are subscribed to the Google Groups 
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/87bb632d-360c-486d-92e1-431ddcb05bb2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/etPan.5c65ac70.3059ef5e.2b4%40tomforb.es.
For more options, visit https://groups.google.com/d/optout.


Re: Use CDN for djangoproject.com

2019-02-14 Thread Tom Forbes
That makes sense, but in this case we are only talking about potentially
yielding control of the docs subdomain which is not used to serve sensitive
build artefacts?

Another option is fastly.com, who support other large open source projects
for free. They essentially give you geographically distributed HAProxy
instances and you have a lot more control over them. I believe several
large Linux distributions use them to serve cached apt packages.

Regarding TLS termination, unfortunately any CDN we use will likely need to
do this for the whole domain to get any benefit. The Django docs are
text/html heavy with very few, if any, images. So the real speed benefit
will have to come from serving that, which requires TLS termination (and
therefore interception) at their end.

On Thu, 14 Feb 2019, 06:32 Markus Holtermann, 
wrote:

> Hi all
>
> to elaborate on what Tobias said: we deliberately have the infrastructure
> spread across multiple service providers: DNS registry, nameservers,
> hosting, TLS certificate authority, … None of them have access to
> everything. The reason is that we offer the download of the release
> artifacts from the djangoproject.com website. And we would like to ensure
> that the TLS termination happens by us and not some random service
> provider. After all, Django is used by enterprises that do have some
> restrictions on where you're allowed to download software from.
>
> By handing over DNS to some CDN provider, we loose the ability to ensure
> that happens.
>
> That said, if there's a CDN that works as a reverse proxy and doesn't
> require us to hand over control of DNS, I guess we could be interested in
> moving the docs behind that.
>
> /Markus
>
> On Thu, Feb 14, 2019, at 2:22 AM, Tobias McNulty wrote:
> > For me it's the trust factor (allowing someone else to decrypt and
> > re-encrypt all our data). This may be less of an issue for the docs
> > site, *if* we don't have to assign DNS authority for the whole domain
> > to the CDN provider.
> >
> > Tobias
> >
> >
> > On Wed, Feb 13, 2019, 7:47 PM Kye Russell  > > I’ve been hearing that there are other CDN providers that offer a very
> comparable service for a fraction of the cost of CloudFront.
> > >
> > > Anyways, at this stage let’s not get bogged down on provider
> decisions. I’m curious if anyone has any general objections to a CDN of any
> kind.
> > >
> > > It shouldn’t be that big a deal to automatically invalidate when the
> docs are updated. But I’m sure there’s something I’m missing.
> > >
> > > On Thu, 14 Feb 2019 at 8:36 am, Cristiano Coelho <
> cristianocc...@gmail.com> wrote:
> > >> Consider AWS's cloudfront then :)
> > >>
> > >> El martes, 12 de febrero de 2019, 2:34:09 (UTC-5), Florian Apolloner
> escribió:
> > >>> Especially cloudflare is a service we do not want to use. as for the
> docs only, does the mirror on rtd work better for you? They are probably
> behind a CDN.
> > >>>
> > >>> Cheers,
> > >>> Florian
> > >>>
> > >>> On Tuesday, February 12, 2019 at 6:43:41 AM UTC+1, Cheng C wrote:
> >  Hi,
> > 
> >  Is it possible to utilize a CDN service for djangoproject.com, or
> at least on docs.djangoproject.com? The site is actually quite fast for
> me but I think there is still room for improvement. Cloudflare sponsored
> dozens of open source projects <
> https://developers.cloudflare.com/sponsorships/>, probably they can
> provide free service for django as well.
> > 
> >  Tested from Melbourne, Australia:
> > 
> >  https://www.djangoproject.com/
> >   Average Ping: 245ms
> >   Browser: 21 requests, 211KB transferred, Finish: 2.52s,
> DOMContentLoaded: 1.16s, Load: 1.48s
> > 
> >  https://git-scm.com/
> >   Average Ping: 5ms
> >   Browser: 42 requests, 351KB transferred, Finish: 717ms,
> DOMContentLoaded: 564ms, Load: 699ms
> > 
> >  Tested on Chrome with "Disable cache" checked (but not the first
> time visit, so DNS query time might not be included).
> > 
> >  Best regards and thanks for all your great work.
> > >>
> >
> >
> > >>  --
> > >>  You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> > >>  To unsubscribe from this group and stop receiving emails from it,
> send an email to django-developers+unsubscr...@googlegroups.com.
> > >>  To post to this group, send email to
> django-developers@googlegroups.com.
> > >>  Visit this group at
> https://groups.google.com/group/django-developers.
> > >>  To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/548db807-647f-4d0b-99c2-f9f229f7175e%40googlegroups.com
> <
> https://groups.google.com/d/msgid/django-developers/548db807-647f-4d0b-99c2-f9f229f7175e%40googlegroups.com?utm_medium=email_source=footer
> >.
> > >>  For more options, visit https://groups.google.com/d/optout.
> > >>
> > >
> >
> >
> > >  --
> > >  You received this message because you are subscribed to the 

Re: Use CDN for djangoproject.com

2019-02-13 Thread Tom Forbes
I would highly recommend cloudflare. It's free, can be set up in 15 minutes
and works really, really well.

On Thu, 14 Feb 2019, 00:36 Cristiano Coelho, 
wrote:

> Consider AWS's cloudfront then :)
>
> El martes, 12 de febrero de 2019, 2:34:09 (UTC-5), Florian Apolloner
> escribió:
>>
>> Especially cloudflare is a service we do not want to use. as for the docs
>> only, does the mirror on rtd work better for you? They are probably behind
>> a CDN.
>>
>> Cheers,
>> Florian
>>
>> On Tuesday, February 12, 2019 at 6:43:41 AM UTC+1, Cheng C wrote:
>>>
>>> Hi,
>>>
>>> Is it possible to utilize a CDN service for djangoproject.com, or at
>>> least on docs.djangoproject.com? The site is actually quite fast for me
>>> but I think there is still room for improvement. Cloudflare sponsored
>>> dozens of open source projects
>>> , probably they can
>>> provide free service for django as well.
>>>
>>> Tested from Melbourne, Australia:
>>>
>>> https://www.djangoproject.com/
>>>  Average Ping: 245ms
>>>  Browser: 21 requests, 211KB transferred, Finish: 2.52s,
>>> DOMContentLoaded: 1.16s, Load: 1.48s
>>>
>>> https://git-scm.com/
>>>  Average Ping: 5ms
>>>  Browser: 42 requests, 351KB transferred, Finish: 717ms,
>>> DOMContentLoaded: 564ms, Load: 699ms
>>>
>>> Tested on Chrome with "Disable cache" checked (but not the first time
>>> visit, so DNS query time might not be included).
>>>
>>> Best regards and thanks for all your great work.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/548db807-647f-4d0b-99c2-f9f229f7175e%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJNvAvUxZbsi5TGkHzGw3TuXjstMtac-woB9x_1iNZvdhQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: django.utils.dateparse

2019-02-04 Thread Tom Forbes
For me, I get:

In [4]: %timeit  datetime_heuristic_parser('2019-02-03T17:27:58.645194')
18.9 µs ± 431 ns per loop (mean ± std. dev. of 7 runs, 10 loops each)

And for Django:

In [3]: %timeit parse_datetime('2019-02-03T17:27:58.645194')
6.97 µs ± 408 ns per loop (mean ± std. dev. of 7 runs, 10 loops each)

I assume there is something wrong with the way you benchmarked the code.
Python is not *that* slow, but 0.0241 per loop is way, way too fast.



On 4 February 2019 at 09:22:03, Giuseppe De Marco (
giuseppe.dema...@unical.it) wrote:

Hello everyone, first of all I am grateful for your time and your attention.

@Tom Forbes
The first time I runned it I thought the same thing! Please use
https://github.com/peppelinux/Django-snippets/blob/master/datetime_heuristic_parser.py
and not the previous pasted one. I'm quite sure that all the tests passes
well, because of their output. As we can see I deal with a tuple that
contains ('format', 'compiled_regexp', 'values dictionary'), this obviously
just for test purpose.

Parsing succesfull on "04/12/2018":
[('%d/%m/%Y', '(?P\\d{1,2})/(?P\\d{1,2})/(?P\\d{4})$',
{'year': 2018, 'month': 12, 'day': 4})]

Parsing succesfull on "04/12/2018 3:2:1":
[('%d/%m/%Y %H:%M:%S',
'(?P\\d{1,2})/(?P\\d{1,2})/(?P\\d{4})
(?P\\d{1,2}):(?P\\d{1,2}):(?P\\d{1,2})$', {'second':
1, 'year': 2018, 'minute': 2, 'month': 12, 'hour': 3, 'day': 4})]

Parsing succesfull on "2018-03-4 09:7:4":
[('%Y-%m-%d %H:%M:%S',
'(?P\\d{4})-(?P\\d{1,2})-(?P\\d{1,2})
(?P\\d{1,2}):(?P\\d{1,2}):(?P\\d{1,2})$', {'second':
4, 'year': 2018, 'minute': 7, 'month': 3, 'hour': 9, 'day': 4})]

Parsing succesfull on "2018-03-04T09:7:4.645194":
[('%Y-%m-%dT%H:%M:%S.%f',
'(?P\\d{4})-(?P\\d{1,2})-(?P\\d{1,2})T(?P\\d{1,2}):(?P\\d{1,2}):(?P\\d{1,2}).(?P\\d{6})$',
{'second': 4, 'year': 2018, 'minute': 7, 'month': 3, 'microsecond': 645194,
'hour': 9, 'day': 4})]

Parsing succesfull on "20180304121940.948000Z":
[('%Y%m%d%H%M%S.%fZ',
'(?P\\d{4})(?P\\d{1,2})(?P\\d{1,2})(?P\\d{1,2})(?P\\d{1,2})(?P\\d{1,2}).(?P\\d{6})Z$',
{'second': 40, 'year': 2018, 'minute': 19, 'month': 3, 'microsecond':
948000, 'hour': 12, 'day': 4})]

Yesterday I coded it on a tablet, this morning from my laptop I got this
performance:
python -m timeit -s "from datetime_heuristic_parser import
datetime_heuristic_parser;
datetime_heuristic_parser('2019-02-03T17:27:58.645194')"
1 loops, best of 3: 0.00891 usec per loop

I also added a simple raise Exception in the case it should return an
error, thank you for your suggestion.

@Augustin
Regarding your questions:

- would this be useful?
I think yes, for the following reasons:
1. We have an authentic regexp compiler based on DATE_FORMATS and
DATETIME_FORMATS
3. We don't have to write datetime regexp anymore, this code will compile a
regexp from a format, indipendently of its delimiter char (if -, / or
whatever)
4. We get generalized function that returns datetime objects, no try/except
and datetime.strptime, It's faster then other implementations!
5. It's settings.py focused, all we have to worry is a correct settings.py
configuration. In other words We just have to collect all the possibile
date/datetime formats that could be used in the project, even if they are
used in forms or in model.fields
6. We don't need anymore to hardcode datetime regexp pattern in our code,
the regexp compiler will work on top of date formats strings!

- would a Form not be a better choice?
Sure, I'm tring to generalize a method that could be a stop application for
all the date and datetime approaches. It could be used for forms, in
DATETIME_INPUT_FORMATS and DATE_INPUT_FORMAT. These could generate form
specialized regexp compilations if this approach will be implemented.

The main goal is to give a tool that will work well and in every
conditions,and be funny too!


Il giorno lun 4 feb 2019 alle ore 01:30 Tom Forbes  ha
scritto:

> I’m pretty sure 0.0241 usec per loop is either a typo or a mistake during
> benchmarking. I’ve got no comment what you’re proposing but correct and
> valid benchmarks are important, so I would double check that.
>
>
> On 3 February 2019 at 23:37:14, Giuseppe De Marco (
> giuseppe.dema...@unical.it) wrote:
>
> Regarding the previous example,
> better to read it here (my fault: I mistaken the format
> '%Y-%m-%dT%H:%M:%S.%f'):
>
> https://github.com/peppelinux/Django-snippets/blob/master/datetime_heuristic_parser.py
>
> and also, it should came also with tzinfo regexp and other functions as
> well, like parse_date time_duration... it's only an example to share our
> experiences.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.c

Re: django.utils.dateparse

2019-02-03 Thread Tom Forbes
I’m pretty sure 0.0241 usec per loop is either a typo or a mistake during
benchmarking. I’ve got no comment what you’re proposing but correct and
valid benchmarks are important, so I would double check that.



On 3 February 2019 at 23:37:14, Giuseppe De Marco (
giuseppe.dema...@unical.it) wrote:

Regarding the previous example,
better to read it here (my fault: I mistaken the format
'%Y-%m-%dT%H:%M:%S.%f'):
https://github.com/peppelinux/Django-snippets/blob/master/datetime_heuristic_parser.py

and also, it should came also with tzinfo regexp and other functions as
well, like parse_date time_duration... it's only an example to share our
experiences.

--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CABms%2BYpcU3gbv7MqV1FkwU5mt9xhTRBoPeG6bPoYCnLvTJLGAw%40mail.gmail.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJPciEjaiXO0oO-UTDCTnbgcSP3Us3D-gOdP4jvMZPGu_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: revisiting the Python version support policy

2019-01-25 Thread Tom Forbes
This message really resonated with me, especially after helping a few
beginners get started with Python and watching them struggle with exactly
this kind of thing.

I'd be +1 on following Python. Looking through the diff there is not a huge
amount of things to remove and IMO none of them are really holding us back
or all that serious. We've fixed some issues with mangling cached property
names, some workarounds for ModuleNotFoundError/ImportError and an issue
with sqlite3 on 3.5.

On 24 January 2019 at 20:33:42, Carlton Gibson (carlton.gib...@gmail.com)
wrote:

To be honest, I'm surprised there's even one person who comes within a 1000
miles of this list who's using Python 3.5. :)

My reason for thinking we should follow Python's supported versions is
users, and particularly beginning users, who have got they-don't-know
version and find a tutorial just what... no sorry need... `pip3 install
Django` to work, and give them the version of Django that corresponds to
what they see when they visit docs.djangoproject.com.

I don't agree this is theoretical at all.

It's not just Debian. (Which doesn't fit my mental model here really...)

It's all those few-years-old computers out there.

It's for example Raspbian, which as of this month is still shipping Python
3.5.

So my boy, who's 10, says,

- What would you use?
- Well I'd use Django (obviously)
- Can I use that?
- Yeah...

If we do drop Python 3.5 I have to say, "Well, no. But you can use this old
one." That's not as cool.
But there will be people who are more seriously affected.

> Who is saying, "I want to use the latest version of Django, but I want to
use a really old version of Python."

No one is saying this. The notion of versions doesn't come into it. We're
well beyond the barrier-to-entry before we get there.
I (just my opinion on this) think we mistake our audience if we forget
this.
(For this reason I don't think the deployment issue is the relevant one.
It's about people learning to programme, not professionals.)

We can't support everything forever, and I'm as keen as anyone to push
forward, but following Python is (for me) the thing we should do.
I think Django's position in the Python eco-system requires it.

Of course if we don't, things are easier for us, yes.

Again, just my opinion.
C.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/19e099cf-6087-4efd-9138-d338f12bbf2c%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJPdaWYJ9LCPZsG9dE66ka%2B8M_8Y4_%2BzMwFNz4er1pXi8w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: bulk_update: add **exclude** arg

2019-01-18 Thread Tom Forbes
Hey Mohammad,
Thanks for your suggestion. I considered this while implementing bulk
update but I decided against it.

In my opinion you should be very explicit about the fields you wish to
update, due to the way that bulk update works it can generate very large
and expensive queries if a lot of columns are updated, so it's better to be
upfront about what you want to update rather than potentially having s
situation where adding a large text field suddenly causes performance
issues (or even database errors) anywhere you use bulk update with that
model.

Tom

On Fri, 18 Jan 2019, 19:49 Mohammad Etemaddar  Dear Django developers,
> Thank you very much for great work and team-work on our beloved Django.
>
> I suddenly saw the new great feature *bulk_update *for models and so
> updated to the dev branch.
> As other similar situations like forms, we have exclude argument for *lazy
> programmers.*
>
> *Thank you,*
> *Mohammad Etemaddar*
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/9cbf3c99-70b5-49d4-a141-d317652ff66a%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOfKK8rgKmcV%2BWDsOn3DLptNBzTJKU8pUfCLivKZ--krw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Request to reconsider ticket #27910: using an Enum class in model Field choices

2019-01-12 Thread Tom Forbes
While I agree that Enum’s are a bit clunky (and IMO removing in is a poor
choice), is this going to really take be that hard to work with or that
difficult to validate?

Enums are types that raise ValueError, like any other, so it’s not that
confusing and fits in with Django’s existing validation code. Once we have
coerced the input into an enum member then validation against choices is
simple no?




On 11 January 2019 at 13:30:40, James Bennett (ubernost...@gmail.com) wrote:

Shai, your rebuttal still misses an important use case, which is
containment.

To continue with your example of a 'SchoolYear(str, Enum)', the following
will both be False:

'FR' in SchoolYear
'FR' in SchoolYear.__members__

The first of those is also becoming illegal soon -- attempting an 'in'
comparison on an Enum, using an operand that isn't an instance of Enum,
will become a TypeError in Python 3.8.

Instead you have to do something like:

'FR' in [choice.value for choice in SchoolYear]

to get a containment check. And you need a containment check to perform
validation.

There are other ways to do it, but they're all clunky, and this is going to
be code that people have to write (in some form) over and over again,
unless we build our own choice abstraction that hides this by wrapping an
Enum and implementing a __contains__ that does what people want. And you'd
basically have to do it in a wrapper, because Enum does metaclass stuff
that interferes with a typical "just subclass and override" approach.

So I still don't think this is going to work for the model-choice use case
without a lot of fiddling (and I'm still not a fan of the enum module in
general, largely because it seems to have gone out of its way to make this
use case difficult to implement).
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CAL13Cg_KEtgmNzRo%3DC8cnCiNsukr5YOCTv3MF8AybV%3D%3DiUXP0g%40mail.gmail.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOSdesTHqak%3Dstj%2Be65YBYOPg2PiZD7iMpoAH2kSpa%2Bgw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Request to reconsider ticket #27910: using an Enum class in model Field choices

2018-12-31 Thread Tom Forbes
Unfortunately I made this for a python 2 app before enums where in the
standard library. I think it has some useful ideas though especially around
categories, which I find I needed a lot. I'd be interested to know if
anyone else has written any workflow-driven apps in Django that require
groups of choice values (i.e "if value in Choices.SOME_GROUP)?

Its a shame about the way xgetext worked, but I guess it's not too bad to
duplicate this if you require it. I'm not too familiar with it and assumed
it worked by importing modules and recording calls to the function.

In any case, we could make the display text optional if you do not require
translations and generate them from the keys?

The way Django handles choices has always irked me as it's really not DRY,
you need to create sort-of enums using class or global variables and
tuples. It's repetitive and boring, and honestly enums made this much more
usable.

We currently have a -1 and a +1, does anyone else have any input on this?

Tom

On Mon, 31 Dec 2018, 22:15 Shai Berger  Hi Tom,
>
> On Mon, 31 Dec 2018 18:26:14 +
> Tom Forbes  wrote:
>
> > I was describing my django-choice-object library[1] and forgot to
> > link it!
> >
>
> Thanks, I took a look -- the library looks nice, but notably, Choice
> objects are not Python enums, although they share some of their traits.
>
> > I would be +1 on re-opening it, I've used enums for this before and I
> > find it much more DRY than the more 'standard' Django way.
>
> Thanks.
>
> > Perhaps we could reduce some boilerplate here by somehow automatically
> adding
> > the key names to gettext?
> >
>
> This, I'm afraid, won't fly. The way xgettext (the engine used by
> makemessages) generates the translation files is by scanning the text
> of source files, looking for specific function calls, and picking their
> arguments with a simple text transformation. So, for the key name to
> somehow be identified, it must be a parameter in a function call (not
> the target of assignment), and the form to be translated must be the
> form that's in the source -- no changing capitalization, or
> underscores-to-spaces, or anything of the sort. So, the best we could
> do to minimize repetitions would create Choice classes looking like
>
> class SchoolYear(Choices):
> choice('Freshman', 'FR')
> choice('Junior', 'JR')
>
> where the choice() calls add members to the class, and the name of the
> member is generated by some transformation over the label. That,
> IMO, would be unacceptably unpythonic.
>
> Have fun,
> Shai.
>
> > 1. https://github.com/orf/django-choice-object
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/20190101001534.29c8c673.shai%40platonix.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJPJcqi7bELk_1QFOp3ZMZV1njYd9nWSc_bumymkzta1BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Request to reconsider ticket #27910: using an Enum class in model Field choices

2018-12-31 Thread Tom Forbes
Hey Shai,
I have not had a chance to look at the implementation, but I was describing
my django-choice-object library[1] and forgot to link it!

I would be +1 on re-opening it, I've used enums for this before and I find
it much more DRY than the more 'standard' Django way. Perhaps we could
reduce some boilerplate here by somehow automatically adding the key names
to gettext?

1. https://github.com/orf/django-choice-object

On Mon, 31 Dec 2018, 18:19 Shai Berger  Hi all,
>
> Lately I've run into ticket 27910[1], which asks for Python Enums to be
> usable for generating choices in Django model (and form) fields. This
> ticket was closed Wontfix because the original suggestion did not offer
> any way to handle translated labels. However, after the ticket was
> closed, Tom Forbes brought up a suggestion for a suitable API;
> regretfully, AFAICT this suggestion was never discussed here, and (at
> least on the ticket) Tom hasn't shared his implementation.
>
> Inspired by his description, I came up with an implementation[2] that is
> simple -- less than 20 lines of executable code -- and I think has a
> lot of nice properties. I think this can make choices-related code
> significantly more elegant.
>
> May we please reopen the ticket?
>
> Thanks,
> Shai.
>
> [1] https://code.djangoproject.com/ticket/27910
> [2]
> https://github.com/django/django/compare/master...shaib:ticket_27910?expand=1
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/20181231201915.6709633b.shai%40platonix.com
> .
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMie_bAtpFHti2Qk59P0kU2LF4ENLtfvFd%2BiZGuQHDNVg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: A faster paginator for django

2018-12-23 Thread Tom Forbes
I would be strongly against misusing EXPLAIN like that inside Django, and I
feel keyset/cursor pagination is best if your going down this road.

I've got no concrete evidence for this opinion, but I feel like navigating
to an exact page is very rarely used and only then as a proxy for range
filtering on a column (i.e "page 5 contains columns within the range I
want").

On Sun, 23 Dec 2018, 12:41 Tim Allen  On Friday, December 21, 2018 at 10:18:04 AM UTC-5, Cristiano Coelho wrote:
>>
>> Let's not forget how the various *count *calls starts to kill your
>> database when you get over 1 million rows (postgres at least).
>>
>> So far the only options I have found with postgres are:
>> - Estimate count for non filtered queries: SELECT reltuples::BIGINT FROM
>> pg_class WHERE relname = '%s';
>> - If queries are filtered, replace it with a subquery that will first
>> limit the results to a reasonable number (such as 1k), this is silly, and
>> won't allow you to go through all results but at least the count call won't
>> kill your database. This is only useful if the filtered query returns over
>> one million rows as well.
>>
>
> I had this problem with very large tables of data being presented as
> endpoints through DRF. For filtered queries estimate, we parse the EXPLAIN
> syntax to get an approximate rowcount, which handles filtered queries. It
> is by no means perfect, but allows us to have next / previous pagination
> while using the EXPLAIN approximate count for an estimated total. We then
> keep showing NEXT until there's a page with no results. More frequent
> vacuums of PostgreSQL are another option for improving accurance of the
> estimate count you mention. We subclassed DRF's limit/offset pagination and
> overrode the count() method to accomplish this. It might be an option to
> consider for Django pagination as well, especially with EXPLAIN now being
> available through the ORM in 2.1:
> https://docs.djangoproject.com/en/2.1/ref/models/querysets/#explain
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/fb3fd1b6-75ed-454d-85c3-ba0672e5368f%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMXrK_YVgCV714Gc4updRf3PNVyy31U15DyGhdLdPKQFw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Widening participation (Thoughts from DjangoCon)

2018-12-12 Thread Tom Forbes
Regarding Gitlab: I love gitlab and for organizations it's one of if not
the best tools in its space. But it falls down for projects like Django,
and I don't think moving migrating the code from GitHub is a good idea.

The labels would need automating, which would require a GitHub bot of some
kind. The workflow could be as simple as "new tickets are assigned the
awaiting review tag" and "only members of the Django org can modify tags"?


On Tue, 11 Dec 2018, 19:38 Josh Smeaton  For what it's worth, I agree. I think we should consider using GitHub
> issues. I don't think there's anything in Trac, from a user perspective,
> that we couldn't really do with Issues. The main issue, I think, would be
> allowing non-committers (organisational members) to triage tickets and
> change the ticket status.
>
> But I don't have the time to plan or champion a change like that through
> these days, so while I'd like to see it happen, I'm not the person to push
> for it to happen.
>
> Jamesie: please see "Why not Gitlab?" [0]. While it might be better from a
> technical standpoint, it would not be better than GH Issues from an
> onboarding perspective.
>
> [0]https://www.python.org/dev/peps/pep-0581/#why-not-gitlab
>
> On Tuesday, 11 December 2018 23:20:21 UTC+11, Tom Forbes wrote:
>>
>> > The only reason Github issues would be a consideration is if the group
>> thought the onboarding experience (being where users already are with a
>> tool they're already familiar with) would have more value than sticking
>> with with the status quo which is strictly better from a feature
>> perspective than GH Issues
>>
>> I really think it's worth considering - I'm not convinced Trac is
>> strictly better from a feature perspective. Things like the dashboards are
>> interesting, sure, but what does it do that Github issues don't? And by
>> this I don't mean features that exist but we don't use, I mean our general
>> ticket workflow that 99% of contributors are exposed to. There are far
>> larger projects than Django currently using Github issues successfully and
>> they have a far more complex workflow including CLA's, high-granularity
>> tags, automatic reviewer selection, automatic closing, etc etc.
>>
>> The cost of a migration would not be cheap, but is it worth doing a
>> formal evaluation and consider it? The "why Github section" (
>> https://www.python.org/dev/peps/pep-0581/#why-github) in that PEP lists
>> some good reasons that apply to us as well. The key one for me is lowering
>> the barrier to entry which I think we should be aiming towards. If we are
>> looking to expand the development group and get more people onboard we may
>> need to shake things up, as the status quo is problematic in that area.
>>
>> On 11 December 2018 at 12:04:32, Josh Smeaton (josh.s...@gmail.com)
>> wrote:
>>
>> I don't think something like Jira would even be a consideration.
>>
>> The only reason Github issues would be a consideration is if the group
>> thought the onboarding experience (being where users already are with a
>> tool they're already familiar with) would have more value than sticking
>> with with the status quo which is strictly better from a feature
>> perspective than GH Issues. Incrementally improving Trac can improve the
>> value prop for not changing, but I don't think anyone that actually uses
>> Trac would say it's worse from a feature perspective. Even if that value
>> judgement did land on GH issues side, there would still be a large
>> operational undertaking to make that transition.
>>
>> Again, the PEP for doing this for CPython would be very similar for
>> Django, so please refer to https://www.python.org/dev/peps/pep-0581/ for
>> some examples of why and what operational concerns there can be (migration,
>> permissions, etc).
>>
>> On Tuesday, 11 December 2018 07:26:58 UTC+11, Ira Abbott wrote:
>>>
>>> Apologies for the double post - my last one was not clear.  "just that"
>>> means that the payment is intended to indicate that it is worth a JIRA
>>> license to me to not use JIRA.
>>>
>>> This group does great things.  I am sure that the group can come up with
>>> some interesting ways to scale that will ultimately benefit the framework
>>> as a whole.
>>>
>>> On Monday, December 10, 2018 at 3:21:53 PM UTC-5, Ira Abbott wrote:
>>>>
>>>> FWIW: Please consider my contribution of $84 (one bulk JIRA license for
>>>> one year) to be just that.
>>>>
>>>> On Monday, December 10, 2018 at 2:14:07 PM UTC-5, Zac

Re: Widening participation (Thoughts from DjangoCon)

2018-12-11 Thread Tom Forbes
> The only reason Github issues would be a consideration is if the group
thought the onboarding experience (being where users already are with a
tool they're already familiar with) would have more value than sticking
with with the status quo which is strictly better from a feature
perspective than GH Issues

I really think it's worth considering - I'm not convinced Trac is strictly
better from a feature perspective. Things like the dashboards are
interesting, sure, but what does it do that Github issues don't? And by
this I don't mean features that exist but we don't use, I mean our general
ticket workflow that 99% of contributors are exposed to. There are far
larger projects than Django currently using Github issues successfully and
they have a far more complex workflow including CLA's, high-granularity
tags, automatic reviewer selection, automatic closing, etc etc.

The cost of a migration would not be cheap, but is it worth doing a formal
evaluation and consider it? The "why Github section" (
https://www.python.org/dev/peps/pep-0581/#why-github) in that PEP lists
some good reasons that apply to us as well. The key one for me is lowering
the barrier to entry which I think we should be aiming towards. If we are
looking to expand the development group and get more people onboard we may
need to shake things up, as the status quo is problematic in that area.

On 11 December 2018 at 12:04:32, Josh Smeaton (josh.smea...@gmail.com)
wrote:

I don't think something like Jira would even be a consideration.

The only reason Github issues would be a consideration is if the group
thought the onboarding experience (being where users already are with a
tool they're already familiar with) would have more value than sticking
with with the status quo which is strictly better from a feature
perspective than GH Issues. Incrementally improving Trac can improve the
value prop for not changing, but I don't think anyone that actually uses
Trac would say it's worse from a feature perspective. Even if that value
judgement did land on GH issues side, there would still be a large
operational undertaking to make that transition.

Again, the PEP for doing this for CPython would be very similar for Django,
so please refer to https://www.python.org/dev/peps/pep-0581/ for some
examples of why and what operational concerns there can be (migration,
permissions, etc).

On Tuesday, 11 December 2018 07:26:58 UTC+11, Ira Abbott wrote:
>
> Apologies for the double post - my last one was not clear.  "just that"
> means that the payment is intended to indicate that it is worth a JIRA
> license to me to not use JIRA.
>
> This group does great things.  I am sure that the group can come up with
> some interesting ways to scale that will ultimately benefit the framework
> as a whole.
>
> On Monday, December 10, 2018 at 3:21:53 PM UTC-5, Ira Abbott wrote:
>>
>> FWIW: Please consider my contribution of $84 (one bulk JIRA license for
>> one year) to be just that.
>>
>> On Monday, December 10, 2018 at 2:14:07 PM UTC-5, Zachary Garwood wrote:
>>>
>>> I'd pay money to NOT use Jira.
>>>
>>> On Mon, Dec 10, 2018, 11:09 AM Dan Davis >>
 Tom, you are right about the UX issues, but full-text and inverted
 indexing would help with the responsiveness as well.  Technically, I was
 talking about djangoproject's TracSearch
 .  That's closer to good UX as
 well, because you can get there by progressive enhancement rather than
 taking stuff away.

 You are also right that switching to another system such as github
 issues could be a better fix than customizing Trac.   But I find it
 difficult to believe that Github issues grows up to cover as much workflow
 management as Trac issues or JIRA.   Maybe we need money to get JIRA.


 --
 You received this message because you are subscribed to the Google
 Groups "Django developers (Contributions to Django itself)" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to django-develop...@googlegroups.com.
 To post to this group, send email to django-d...@googlegroups.com.
 Visit this group at https://groups.google.com/group/django-developers.
 To view this discussion on the web visit https://groups.google.com/d/
 msgid/django-developers/CAFzonYZ%3DwRCX3HpfYCj0G2i3UrcQmVCDj%
 3Da%2BOb7zeqtvQm-pxg%40mail.gmail.com
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>> --
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to 

Re: Explore integrating django-docker-box in some way?

2018-12-04 Thread Tom Forbes
To have this completely working at sprints without having everyone building
their own local images we would need to have the Jenkins server use docker
in some capacity. This would also require an official django account on
Docker hub.

The pattern I’m using right now is that on every build we pull the
django-ci:latest image (from my personal account). Docker uses this image
as a cache automatically (preventing rebuilds). On any successful master
build we push the new image to docker-hub, so subsequent builds can utilise
it.

Then anyone wanting to speed up their bootstrapping can do docker-compose
pull and automatically have the latest image available for running right
away. We can make this smaller, for sure, but we can also suggest people
download this beforehand (i.e at their hotel).

I don’t know how feasible this is but it’s also very easy to run a caching
docker mirror (docker run -p 5000:5000 registry). Organizers could run this
at large events and configuring docker to use a local mirror on the network
is a one-line change for atendees.




On 4 December 2018 at 23:52:42, Josh Smeaton (josh.smea...@gmail.com) wrote:

Size of the image could definitely be a concern, especially at sprints
where wifi speeds aren't always optimal. The django-box image is
significantly larger though so it'd still be a net win. There are also
optimisations that can be made to the image for reducing size over time, so
I'd fully expect it to come down. I've spent a little bit of time trying to
optimise a $work$ python docker file, I'll provide what I've got as an
issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step
forward. I'd also like to hear from the infrastructure team what their
thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this
list of names is from memory).

On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote:
>
> Thank you for the reply Josh. I didn’t anticipate any suggestions for
> including this inside core but off the back of your suggestion I’ve made a
> ticket here: https://code.djangoproject.com/ticket/30010.
>
> I don’t think it should be complex at all to include this inside Django -
> it’s four or five new files at most. Hopefully this should improve the
> experience at sprints, however the current Dockerfile weighs in at 650+mb
> so the problem may switch from ‘it is hard to set up an environment’ to ‘it
> is hard to download one’!
>
>
>
> On 5 November 2018 at 23:02:30, Josh Smeaton (josh.s...@gmail.com) wrote:
>
> I'm sorry I haven't had the time to review or contribute yet, but I think
> it'll be a very useful project - especially for new contributors that might
> have a little docker experience. The current vagrant solution is heavy,
> does not work properly on windows and some linuxes, and isn't that easy to
> maintain or deploy. I'd be in favour of adding the docker files directly to
> django/django to minimise setup burden (DJANGO_PATH), and improving the
> contributing docs to show users how to test using docker.
>
> One of the hardest things I found at sprints was getting development
> environments setup to effectively contribute - even using the docker-box
> project which I understand quite well. Anything we can do to improve that
> situation will be very beneficial.
>
> I have fewer opinions about the official CI story, hopefully some of the
> infrastructure team can comment more on that. I think that replacing the
> ansible roles with a docker setup can have some definite improvements and
> open up CI tasks to a larger pool of people (anyone that can edit docker
> files), but it'd come with maintaining the host that runs docker (cleaning
> up images, dealing with disk space issues, etc).
>
>
> On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:
>>
>> Hello all,
>>
>> I’ve been working on a docker-compose based alternative to django-box
>> (imaginatively named django-docker-box) over the last month and it
>> finally appears to be mostly complete.
>>
>> For reference the tool is just a Dockerfile and a docker-compose
>> definition that is able to run a complete test matrix of every supported
>> Python and DB version. It’s as simple as docker-compose run sqlite. You
>> can see a full test run (excluding oracle) here:
>> https://travis-ci.com/orf/django-docker-box/builds/90167436
>>
>> Florian suggested I create a thread here to gather feedback and discuss
>> any potential future directions for the project, so here goes:
>>
>> Firstly I’d like to know if there is any support for moving this under
>> the Django project itself, maybe even as a replacement for django-box? I
>> think the setup is pretty quick compared 

Re: Explore integrating django-docker-box in some way?

2018-12-04 Thread Tom Forbes
Thank you for the reply Josh. I didn’t anticipate any suggestions for
including this inside core but off the back of your suggestion I’ve made a
ticket here: https://code.djangoproject.com/ticket/30010.

I don’t think it should be complex at all to include this inside Django -
it’s four or five new files at most. Hopefully this should improve the
experience at sprints, however the current Dockerfile weighs in at 650+mb
so the problem may switch from ‘it is hard to set up an environment’ to ‘it
is hard to download one’!




On 5 November 2018 at 23:02:30, Josh Smeaton (josh.smea...@gmail.com) wrote:

I'm sorry I haven't had the time to review or contribute yet, but I think
it'll be a very useful project - especially for new contributors that might
have a little docker experience. The current vagrant solution is heavy,
does not work properly on windows and some linuxes, and isn't that easy to
maintain or deploy. I'd be in favour of adding the docker files directly to
django/django to minimise setup burden (DJANGO_PATH), and improving the
contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development
environments setup to effectively contribute - even using the docker-box
project which I understand quite well. Anything we can do to improve that
situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the
infrastructure team can comment more on that. I think that replacing the
ansible roles with a docker setup can have some definite improvements and
open up CI tasks to a larger pool of people (anyone that can edit docker
files), but it'd come with maintaining the host that runs docker (cleaning
up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:
>
> Hello all,
>
> I’ve been working on a docker-compose based alternative to django-box
> (imaginatively named django-docker-box) over the last month and it
> finally appears to be mostly complete.
>
> For reference the tool is just a Dockerfile and a docker-compose
> definition that is able to run a complete test matrix of every supported
> Python and DB version. It’s as simple as docker-compose run sqlite. You
> can see a full test run (excluding oracle) here:
> https://travis-ci.com/orf/django-docker-box/builds/90167436
>
> Florian suggested I create a thread here to gather feedback and discuss
> any potential future directions for the project, so here goes:
>
> Firstly I’d like to know if there is any support for moving this under the
> Django project itself, maybe even as a replacement for django-box? I think
> the setup is pretty quick compared to django-box and is more flexible in
> terms of database version support as well as working with Oracle. I’d also
> really like some help improving Oracle support if anyone has the time!
>
> Secondly is there any support for integrating this with our current
> Jenkins setup? I think it would be pretty neat to have parity between what
> runs on the CI and what we can run locally and have any improvements shared
> between both. Perhaps a full matrix run (which right now is 66 different
> environments) is out of the question but a smaller subset could be good?
>
> Thirdly, and this is a bit wild, but what about using this to reduce the
> burden of running Jenkins by running the tests on a managed CI service like
> Travis CI? We would likely still need Jenkins due to issues with Oracle and
> running tests on Windows (unless https://github.com/django/
> django/pull/10259 works with Docker!), but we could offload some of the
> environments onto a third party service. Travis gives large OS projects
> like Django increased concurrency limits on their accounts so we could end
> up with pretty speedy test runs. Also with docker-compose switching between
> CI services (including Jenkins) would be very simple.
>
> The repo is here: https://github.com/orf/django-docker-box.
>
> Any feedback on these points or the project itself would be greatly
> appreciated,
>
> Tom
>
>
> --
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com?utm_medium=email_source=footer>
.
For more options, visit https://groups.goo

Re: Moving database backends out of the core

2018-11-26 Thread Tom Forbes
I see where you are coming from, but I'm not sure it will have the intended
effect. One of the great things about Django is that for the most part
database features work everywhere. If we split them out there will be more
innovation, sure, but we may loose the 'database transparency' we currently
have leading to inconsistencies.

It may also be harder to coordinate changes across databases that require
changes to core.

On your point about non-standard backends, maybe we should focus on making
it easier to add third party backends and standardize some of the
internals? We could treat the backend as external (i.e no special hacks for
them) but keep them in the Django package. If we make the interfaces a bit
more abstract it would be a lot easier for these backends to exist without
hacks perhaps.

As with everything, it's a trade off. I'm not sure there is a clear answer
or better solution for all cases than we currently have.

On Mon, 26 Nov 2018, 08:57 Johannes Hoppe  I want to address a completely different point, and that *innovation*. I
> believe that 3rd party backends could lead to more innovation in the Django
> ORM.
> Currently if you want to introduce a new feature for your database, you
> are faced with a lot of complications added by databases you might not be
> familiar with. Furthermore you might be requested to makes those features
> available for databases you haven't used before. This drastically increases
> the bar for contributing innovative new features. As an example, I wanted
> to add database defaults for Postgres and multiple insert return values. I
> finished the postgres feature in 2 sprints, but it took me another half
> year to implement the same for Oracle. Mainly because I never used Oracle
> before.
>
> Beyond that I see that the sheer effort to remove the backends from core
> could lead to better design. We currently assume that all databases speak
> some flavor of Std SQL, which isn't even true for the currently supported
> databases and certainly not for the verify famous mongo-db backend.
>
> In conclusion, more separation will lead to more diversity. But that is a
> good thing, something to embrace and can lead to great results. I would
> even go as far as to kick Postgres out too, contrary to what Florian
> suggested. I believe Postgres could benefit from a separate package.
>
> I think Django has the best ORMs and us being able to make changes and
> innovate, can ensure that this is still true a decade from now.
>
> On Friday, March 15, 2013 at 3:29:29 PM UTC+1, VernonCole wrote:
>>
>> My organization just hit a use case where we need MS-SQL support.
>>
>>  I am jumping on board, so there are at least two of us who can do
>> maintenance.
>>
>> I must say that I would prefer quasi-supported status (akin to admin
>> and geodjango) rather than actually being in the core.  I think it would
>> be a better fit for most situations.  We will always be a small minority
>> of
>> django users.  I would just like some assurance that pull requests needed
>> to provide good hook support for external database backends got prompt
>> attention from the core developers.
>> --
>> Vernon Cole
>>
>> On Thursday, March 7, 2013 6:46:18 PM UTC+1, Jacob Kaplan-Moss wrote:
>>>
>>> On Wed, Mar 6, 2013 at 3:22 AM, Marc Tamlyn  wrote:
>>> > I don't know why Oracle is included and MSSQL is not [...]
>>>
>>> It's essentially because a group of people made a concerted effort to
>>> get the Oracle backend up to snuff (around the 1.0 timeline, I think?)
>>> and then committed to maintaining it. Lately the original people who'd
>>> made that commitment have dropped off a bit, but there's always been
>>> enough attention to keep it up to date.
>>>
>>> If someone -- preferably a group -- stepped up and committed to
>>> keeping a MSSQL backend up-to-date in core, I'd be +1 on merging it
>>> and giving those people commit access to keep it up to date.
>>>
>>> [This is me speaking personally, not as a BDFL. It'd have to be a
>>> discussion, not just my fiat.]
>>>
>>> Jacob
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/978f91f8-709a-48dc-b2e2-b940e19a9f7f%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe 

Re: Removing Oracle from Django core in 3.0

2018-11-26 Thread Tom Forbes
Indeed, the initial setup of the database takes a horrendous amount of time
(like 30 minutes to init an empty database!).

On Mon, 26 Nov 2018, 14:12 Johannes Hoppe  To quote the documentation:
> https://github.com/orf/django-docker-box/blob/85780dcc81d62a4c0c1142b45eb69e825d97b074/README.md#oracl
> <https://github.com/orf/django-docker-box/blob/85780dcc81d62a4c0c1142b45eb69e825d97b074/README.md#oracle>e
>
>
> "As usual Oracle is a bit more complex to set up.” ;)
>
>
> --
> Johannes Hoppe
>
> www.johanneshoppe.com
>
> Want to chat? Let's get a coffee!
> https://calendly.com/codingjoe/coffee
>
> Lennéstr. 19
> 14469 Potsdam
>
> USt-IdNr.: DE284754038
> On 26. Nov 2018, 14:55 +0100, charettes , wrote:
>
> I haven't tried it out for Oracle yet but Tom Forbes' django-docker-box
> seems to make it a not-too-painful process[0]
>
> Simon
>
> [0] https://github.com/orf/django-docker-box#oracle
>
> Le lundi 26 novembre 2018 04:05:41 UTC-5, Johannes Hoppe a écrit :
>>
>>
>> On Monday, November 26, 2018 at 9:49:46 AM UTC+1, Florian Apolloner
>> wrote:
>>>
>>> Hi,
>>>
>>> I personally agree with Mariusz here. Oracle might have it's own quirks,
>>> but the same could be said for any database. Taking my experience with the
>>> ORM into account I do not think that Oracle requires much more work (if at
>>> all) than any other database. I think in the end it does not matter whether
>>> one works around the limitations/features of Oracle or MySQL. All in all I
>>> think having Oracle is a good thing because databases like MSSQL and
>>> Informix/DB2 quite often behave similar to Oracle and just narrowing the
>>> core scope to MySQL/Pg/Sqlite might lead to a kind of tunnel vision.
>>>
>>> Granted, Oracle might be a bit harder to install; but with the developer
>>> VMs and docker containers that argument doesn't really hold either imo.
>>>
>> There is not official container and the one you can build from their
>> repo, didn't work for me. Furthermore you need to register with oracle and
>> give them your Phone number, just to download the python library bindings.
>> So it is somewhat harder than others ;)
>>
>>>
>>> Cheers,
>>> Florian
>>>
>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/dg8BUVHKOo4/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/ee9fa331-1606-48fb-9362-e86cd9f3c9ae%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/ee9fa331-1606-48fb-9362-e86cd9f3c9ae%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/d638aa8b-c10a-4c3c-b6d4-a687d3d9d979%40Spark
> <https://groups.google.com/d/msgid/django-developers/d638aa8b-c10a-4c3c-b6d4-a687d3d9d979%40Spark?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJNC_WNqOuymisOD_6tWNSSxu3yoKoJCt0KiugFX77Tm7Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Add Python 3.7 support for Django 1.11?

2018-11-16 Thread Tom Forbes
Do we have an idea of how many fixes would need to be backported? If it's
just this one (or a very select few) I would support that. You can work
around most things in Python but a hard syntax error is a big pain, so I
can understand people wanting this.

On 16 November 2018 at 15:24:44, Tim Graham (timogra...@gmail.com) wrote:

We've received a relatively steady stream of requests to add Python 3.7
support for Django 1.11. Is there support or opposition for that?

See comments of
https://github.com/django/django/commit/931c60c5216bd71bc11f489e00e063331cf21f40#commitcomment-31328709
for the stream of "please backport this to 1.11" comments.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/26621bcf-58eb-4c5f-a87b-250f0c2174a8%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJNqLQtq03ee-Sfc5v5z1YzETbxu%3D-bWN9FQk0%3D5Yd1Whg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Pluggable secret key backend

2018-11-11 Thread Tom Forbes
Is it going to be easy to adjust the semantics of SECRET_KEY to support
sequences like that? I’d imagine a lot of third party packages that expect
SECRET_KEY to be a string would break in weird ways (thanks to both strings
and tuples of strings being iterables that yield strings).

Here’s a quick search on GitHub:
https://github.com/search?q=%22settings.SECRET_KEY%22=Code

To ease the backward compatibility concerns we could use SECRET_KEYS, then
make SECRET_KEY (if it is not explicitly defined) map to SECRET_KEYS[0]?
Third party packages using would not necessarily work with the backwards
verification but they would at least not break and continue to work as
expected.

Tom




On 11 November 2018 at 07:38:15, Aymeric Augustin (
aymeric.augus...@polytechnique.org) wrote:

Hello,

I think this is a great idea.

As suggested by others, an even better default implementation would be:

class SecretKeysBackend:

def get_signing_key(self):
if isinstance(settings.SECRET_KEY, (list, tuple)):
return settings.SECRET_KEY[0]
else:
return settings.SECRET_KEY

def get_verification_keys(self):
if isinstance(settings.SECRET_KEY, (list, tuple)):
return settings.SECRET_KEY
else:
return [settings.SECRET_KEY]

Once Django is updated to take advantage of this feature, hat would make
key rotation practical for every Django user!

(And it seems easier to adjust the semantics of SECRET_KEY than to
introduce a SECRET_KEYS settings.)

Best regards,

-- 
Aymeric.



On 10 Nov 2018, at 11:12, Andreas Pelme  wrote:

Hi,

settings.SECRET_KEY can be used for sessions, password resets, form wizards
and
other cryptographic signatures via the signing APIs. Changing SECRET_KEY
means
that all of those will be invalidated and the users will be affected in
weird
ways without really knowing what happened. (Why am I logged out? Where did
my
form submission go? Why does not this password reset link work?). This is
desirable in case the key is compromised and swift action must be taken.

There are other situations when it would be nice to change the SECRET_KEY
when
this sudden invalidation is not desirable:

- When someone leaves a project/company that had access to the production
 system. After SSH keys/login credentials is revoked the developer could
 potentially have a copy of the secret key. It is essentially a backdoor
with
 full remote access. It would be wise to rotate the key in those cases.

- Periodic and automatic rotations of keys to make it less useful in the
 future.

The current situation of a single SECRET_KEY makes key rotation
impractical. If
you run a busy site with active users 24/7, there is never a nice time to
change the SECRET_KEY.

A solution for this problem would be sign new secrets with a new key while
still allow signatures made with the old key to be considered valid at the
same
time. Changing keys and having a couple of hours of overlap where signatures
from both keys are accepted would mitigate most of the user facing problems
with invalidating sessions, password reset links and form wizard progress.

You could do this today by implementing your own session backend, message
storage backend and password reset token generator but that is cumbersome
and
does not work across reusable apps that directly use low level Django
signing
APIs unless they too provide hooks to provide your own secret.

I propose a pluggable project wide secret key backend
(settings.SECRET_KEY_BACKEND maybe?) with an API something like:

class SecretKeyBackend:
 def get_signing_key(self): …
 def get_verification_keys(self): ...

The default (and backward compatible) backend would then be implemented as
something like:

class SecretKeySettingsBackend:
 def get_signing_key(self):
   return settings.SECRET_KEY
 def get_verification_keys(self):
   return [settings.SECRET_KEY]

django.core.signing.Signer.{sign,unsign} would need to be updated to use
this
backend instead of directly using settings.SECRET_KEY.

That would solve the problem project wide and work across any third party
application that uses django.core.signing directly.

This would open the door for third party secrets backend packages that
retrieves keys from systems such as Hashicorp Vault, AWS Secrets Manager,
Google Cloud KMS, Docker Secrets etc.

Having a method that retrieves the key would allow changes to secret key
during
run time instead of relying on a hard coded setting would allow the key to
change without restarting the server process.

Would something like this be worth pursuing? Could it be designed in som
other
way? I could not find any previous discussion/tickets on this and thought it
would be a good idea to discuss it here before opening a ticket or making an
attempt at a PR. :)

Cheers,

Andreas


--
You received this message because you are subscribed to the Google Groups
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving 

Re: Password reset token safety

2018-11-07 Thread Tom Forbes
Would you consider the *secret* key to not be unpredictable?

On Wed, 7 Nov 2018, 21:22 Alex Toussaint 
> Hello,
>
> I'd like to discuss about Django's password reset token functionality.
>
> I've been able, with a simple Python script, from having read-only access
> to my Django webserver to a full read-write by crafting a reset token.
> Isn't it one of the main goals of hashing passwords ? Protecting from
> attackers having read only access to the database ?
> The only "Unpredictable" data that can be needed if the attacker's  last
> database access is old is the last_login, which can be very easily bypassed
> with a simple script bruteforce (as I did here
> )
>
> I would like to know, is there an other way the password reset mechanism
> could work that wouldn't enable such problem ? I haven't found one alone.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/96fd4144-450e-4d9a-ab7d-eb308339056c%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOasT-mn1u5djEWz23UoUJzOBfUEQ0%2B5P4%3Dwsn0%3D6RUzw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: backend specific tests

2018-11-07 Thread Tom Forbes
Hey Dan,

I’ve been working on a project called django_docker_box (
https://github.com/orf/django-docker-box) that might help with this. Docker
is pretty good at spinning up various databases without needing to clutter
your local machine, spend time configuring authentication or dealing with
issues like this.

You might find this helps with you. You can run the entire test suite with
it, or you can just spin up a Postgres database and connect to it from your
local machine (docker-compose run postgres-db) with django:django.

Alternatively you could run:

docker run -p 5432:5432 -e POSTGRES_PASSWORD=django postgres:9.6

And connect to it with postgres:django on localhost:5432

Mysql can be done in the same way:

docker run -p 3306:3306 -e MYSQL_PASSWORD=django mysql:5.7

Hope this helps,

Tom




On 7 November 2018 at 03:51:47, Dan Davis (dansm...@gmail.com) wrote:

What about mysql?   I have 5.7 installed, and tried to run with root
privileges, which is what worked for postgresql.  It still failed quite a
number of times.

On Tue, Nov 6, 2018 at 7:41 PM charettes  wrote:

> Exactly.
>
> Given you should be running tests against a throwaway or at least
> non-critical
> PostgreSQL cluster anyway I don't think requiring superuser privileges is
> an
> issue.
>
> Simon
>
> Le mardi 6 novembre 2018 18:43:47 UTC-5, Dan Davis a écrit :
>>
>> So, a developer using PostgreSQL doesn't need superuser privileges, but
>> you do to run Django's unit tests, because it will test these contributed
>> postgres operations.
>>
>> On Monday, November 5, 2018 at 5:48:20 PM UTC-5, Josh Smeaton wrote:
>>>
>>> I don't think there's a full list of extensions the test suite uses, but
>>> https://docs.djangoproject.com/en/2.1/ref/contrib/postgres/operations/
>>> would be close to a full set I'd imagine.
>>>
>>> On Sunday, 4 November 2018 14:43:23 UTC+11, Dan Davis wrote:

 So, the contributor guidelines page about unit tests mentions running
 database specific tests:


 https://docs.djangoproject.com/en/2.1/internals/contributing/writing-code/unit-tests/#testing-other-python-versions-and-database-backends

 I am working on ticket 29984, and it seems to me that since the
 TruncDay is attempting to cast to the timezone on the database level, it is
 working, and its job is done.  So, the fix should be at the backend level,
 and the ticket provides one for Postgres.

 Following the unit test advice, I created a settings file.   But not
 using my Postgresql admin user.   I get the following errors:

 psycopg2.ProgrammingError: permission denied to create extension
 "btree_gin"
 HINT:  Must be superuser to create this extension.

 I'm not going to do that - do you have any list of required extensions?

>>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/d252a3e0-f8f4-4026-b1eb-9dfd79993618%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CAFzonYYhELA0RRpM07KGsYX08CmM09UjeN8vZRovwfmCUTz_2g%40mail.gmail.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMt_w9p2XXaZqyyKPDQycsNAurb9RGea09TfsdpJgNfkQ%40mail.gmail.com.
For more options, visit 

Re: Idea: Allow queryset.get() and queryset.filter() to accept a positional argument for implicit primary key filtering

2018-11-05 Thread Tom Forbes
I feel this would be a good addition to just .get(), I’ve wanted this while
working with the shell. Model.objects.get(pk) feels very natural to me, and
the common Model.objects.get(pk=pk) always felt overly verbose.




On 5 November 2018 at 22:37:52, Josh Smeaton (josh.smea...@gmail.com) wrote:

I'm in the same boat as Simon - I've wanted this many times in the last few
months, but only when working at the shell. I'd be +1 for get and -1 for
filter also.

On Thursday, 1 November 2018 05:12:53 UTC+11, charettes wrote:
>
> As I've mentioned on the ticket I've been wishing get(pk) could translate
> to get(pk=pk) for a while but I never got to suggest it. Probably because I
> didn't feel like adding (pk=...) was really that long. I'd note that most
> of the
> times I wished this existed was when doing some object manipulations
> through a Django shell.
>
> I'm not convinced this would be as useful for `filter()` though as I don't
> recall wanting to retrieve a set of objects matching a value I know will
> be unique.
>
> Something to keep in mind as well is whether or not we want to allow
> this to be coupled with extra args and kwargs.
>
> e.g.
>
> Book.objects.get(isbn, author__name='Daniel Roy Greenfeld')
>
> I'd be in favor of preventing pk and kwarg or Q args mixing.
>
> Count me +1 for the get() case and -1 for the filter() one.
>
> Simon
>
> Le mercredi 31 octobre 2018 13:13:34 UTC-4, Antwan a écrit :
>>
>> Hi,
>> I'm creating this topic to see if there is interest to implement
>> positional arguments in queryset filtering.
>>
>> Current situation
>>
>> Currently the only way to use positional arguments to filter can be
>> either:
>>
>>- Passing a single or multiple Q objects:
>>
>>MyClass.objects.filter(Q(key=value))
>>MyClass.objects.filter(Q(key=value), Q(other_key=value))
>>
>>
>>
>>- Passing a couple is also working (not sure if this is a happy
>>accident, should it be removed?)
>>
>>MyClass.objects.filter((key, value))
>>
>>
>>
>>- Combination of both is also proven to work
>>
>>MyClass.objects.filter((key, value), Q(other_key=value))
>>
>>
>>
>> Suggestion
>>
>> This feature suggestion is to leverage the case when a non-Q / non couple
>> object is passed, so it implicitly interpreted as the value for the model's
>> pk.
>>
>> This could ease/simplify code by omitting pk when this is the only
>> filter used:
>>
>>
>> MyClass.objects.get(value)
>> # Translates into: MyClass.objects.get(pk=value)
>>
>>
>> or
>>
>>
>> MyClass.objects.filter(value)
>> # Translates into: MyClass.objects.filter(pk=value)
>>
>>
>> or
>>
>>
>> MyClass.objects.filter(Q(value))
>> # Translates into: MyClass.objects.filter(Q(pk=value))
>>
>> Do you think it's worth it? It could be leveraged to simplify many
>> situations.
>> I'd be happy to proceed to the development myself if this is something
>> gathering interest.
>>
> --
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/ba9128d7-9e49-4269-b3a3-9995a998b491%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJP8Ny6gYqQ9ssmCHjK_J_uEfe%2BGFTmcy0BA1fPJPEfQig%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Remove pyinotify autoreloader support

2018-11-04 Thread Tom Forbes
I have done a lot more reading on this and I really feel pyinotify may not
be required, or we could at least switch to a watchman based service for
simplicity right away. Projects like uwsgi use a stat based approach, as do
most other projects I’ve seen and it appears to work ok for them. watchman
has the advantage of being easier to test, as we can mock the actual
watchman service itself, and it supplies a sane, simple API across all
supported platforms. However it suffers from some limitations I need to
work around.

As an example of a fairly large project I took a look at Sentry. After
running the test suite (and excluding test modules) there are 1,000 loaded
Python modules under sentry.* and 4,000 total entries in sys.modules.
Django accounts for about 600, by far the biggest third party module.

During development it seems likely that it’s not worth polling any builtin
python modules at all, and drastically reducing the polling time for site
packages including Django. That would reduce the number of files to poll by
75%. Perhaps we could even just skip watching site-packages entirely - I
think it’s not very common to edit these in a usual development session,
and if the user finds themselves editing them we could add a flag to
include them in the watch list?

If anyone here has a fairly large Django app they work on could I request
that they run this snippet at some point after it has booted and send me
the output, to see if Sentry is representative?

from collections import Counter
counter = Counter(m.split('.')[0] for m in sys.modules.keys())
total = sum(counter.values())
project_total = counter['PROJECT NAME HERE']
print('Project:', project_total)
print('Total:', total)
print(counter.most_common(20))

Thanks,

Tom




On 8 October 2018 at 12:04:08, Tom Forbes (t...@tomforb.es) wrote:

Thanks for the feedback! In the pull request I have re-added support for
pyinotify with tests, it was not as hard to write them as I believed. They
still fail, but I’m working on that!

I’ve found an interesting module confusingly called watchgod. The author
says[1] that with the new os.scandir() method added in python 3.5 a stat
based method can scan a project of 850 files in about 24ms.

While this is watching a single directory tree and not the entirety of
sys.modules I believe we could get similar speedups by being a bit more
intelligent in our approach. For example we really don’t need to stat every
single django.* module every second. I’m guessing that those change very
rarely (unless you’re hacking on Django!), so could maybe stat them every
2–3 seconds or even not at all. We could reduce the stating time on
site-packages modules as well for similar reasons.

If we do this right we could potentially get the overhead down to an
acceptable level that we don’t necessarily need platform specific watchers.

   1.
   
https://github.com/samuelcolvin/watchgod#why-no-inotify–kqueue–fsevent–winapi-support



On 6 October 2018 at 20:56:17, charettes (charett...@gmail.com) wrote:

While I understand the complexity of 1. I think shipping a version of
Django without
an equivalent inotify replacement such as watchdog could be problematic.

>From my personal experience when using Django's development server in Docker
containers sharing local volumes installing pyinotify resulted a
significant performance
CPU and I/O improvement.

Simon

Le samedi 6 octobre 2018 15:32:33 UTC-4, Tom Forbes a écrit :
>
> What do we think about removing the pyinotify functionality in the
> autoreloader? For context, if pyinotify is installed (Linux only) the
> autoreloader will attempt to use it to detect file changes over the
> standard polling-based approach. It is generally much more efficient but is
> not cross platform, and is not well documented currently IMO.
>
> I’m hacking away at my attempt at refactoring the autoreloader (
> https://github.com/django/django/pull/8819) and I’ve made some good
> progress but I am worried about the lack of tests for pyinotify (there are
> none!). The current code in master works but it’s really hard to refactor
> in any meaningful way without them. I would very much like to explore
> adding support for watchdog (https://pythonhosted.org/watchdog/) instead
> of pyinotify and these changes I’m working on are a means to that end.
>
> Wwatchdog is a library that wraps efficient platform-specific filesystem
> notifications in cross-platform way, and it includes an inotify
> implementation.
>
> So I think there are three ways forward:
>
>1.
>
>Spend time and effort adding special tests for PyInotify (testing this
>stuff is not simple, especially with the current code)
>2.
>
>Remove the functionality with an eye to using something like watchdog
>in the near future
>3.
>
>Never really touch the autoreloader much (it is some of the oldest and
>nastiest code we have).
>

Explore integrating django-docker-box in some way?

2018-11-04 Thread Tom Forbes
Hello all,

I’ve been working on a docker-compose based alternative to django-box
(imaginatively named django-docker-box) over the last month and it finally
appears to be mostly complete.

For reference the tool is just a Dockerfile and a docker-compose definition
that is able to run a complete test matrix of every supported Python and DB
version. It’s as simple as docker-compose run sqlite. You can see a full
test run (excluding oracle) here:
https://travis-ci.com/orf/django-docker-box/builds/90167436

Florian suggested I create a thread here to gather feedback and discuss any
potential future directions for the project, so here goes:

Firstly I’d like to know if there is any support for moving this under the
Django project itself, maybe even as a replacement for django-box? I think
the setup is pretty quick compared to django-box and is more flexible in
terms of database version support as well as working with Oracle. I’d also
really like some help improving Oracle support if anyone has the time!

Secondly is there any support for integrating this with our current Jenkins
setup? I think it would be pretty neat to have parity between what runs on
the CI and what we can run locally and have any improvements shared between
both. Perhaps a full matrix run (which right now is 66 different
environments) is out of the question but a smaller subset could be good?

Thirdly, and this is a bit wild, but what about using this to reduce the
burden of running Jenkins by running the tests on a managed CI service like
Travis CI? We would likely still need Jenkins due to issues with Oracle and
running tests on Windows (unless https://github.com/django/django/pull/10259
works with Docker!), but we could offload some of the environments onto a
third party service. Travis gives large OS projects like Django increased
concurrency limits on their accounts so we could end up with pretty speedy
test runs. Also with docker-compose switching between CI services
(including Jenkins) would be very simple.

The repo is here: https://github.com/orf/django-docker-box.

Any feedback on these points or the project itself would be greatly
appreciated,

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJNFvp5ke_SzZtto2JEy%3DUsUSWFHByf-sA_ahUW9Vb_Brg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Python string formatting

2018-10-31 Thread Tom Forbes
In my experience f strings are vastly more expressive than .format, but not
much better than % based formatting.

If we can make this change after we drop 3.5 I think it will be a lot
easier.

On Wed, 31 Oct 2018, 21:09 Adrian Turjak  There was a push to deprecated % formatting but too many people complained
> and that never happened.
>
> While .format and g-strings are superior, % is here to stay for the
> foreseeable future. Too many people still use it (including myself
> sometimes).
>
> On 1 Nov. 2018 08:14, Carlton Gibson  wrote:
>
> We had a bit of a discussion of this on
>
>
> Python 2 docs have got this re format():
>
> > This method of string formatting is the new standard in Python 3, and
> should be preferred to the % formatting described in String Formatting
> Operations in new code.
>
> ​https://docs.python.org/2/library/stdtypes.html#str.format
>
> But it's not there for Python 3 (
> ​https://docs.python.org/3/library/stdtypes.html#str.format) so I'm not
> clear what we're meant to think.
> (I had thought %-formatting deprecated...)
>
> Anyone know?
>
> Regardless, waiting a bit and moving straight to f-strings (as Simon
> suggests) seems sensible. (If we're convinced we do want to change.)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/e7cf5b0d-cff4-4c6d-a9ec-1af4a82d5a56%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/6b09193b-a53f-4eb4-9f36-2c7aa11f47cb%40email.android.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOpL%2BLXx2xQhSb6DtGr90UQ-LhA2Y6S8CpZftjVcObmQg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Widening participation (Thoughts from DjangoCon)

2018-10-26 Thread Tom Forbes
How much of this would you attribute to the current ticketing system
itself, rather than tickets being tagged appropriately?

I know when I started contributing I found trac to be pretty intimidating
in terms of complexity, especially the search. I still prefer to use the
'Search Trac' field in the root code.djangoproject.com page than fiddle
with the myriad of options and drop downs in the browse tickets section.

If we think of getting new people onboard as a conversion funnel we need to
stop dropoff as much as possible, and that extends to the UI of the ticket
tracker as well I believe.

Tom

On Fri, 26 Oct 2018, 22:43 Ian Foote,  wrote:

> Hi Carlton,
>
> I've had similar thoughts sitting in the back of my mind for at least a
> couple of months, so thank you for sharing this. I agree that finding
> tickets is one of the big problems here, both for new contributors and for
> sprint leaders. At Pycon UK I took on the role of sprint leader along with
> Adam Johnson and directing people to appropriate tickets was a definite
> difficulty. I was also unaware of the django core mentorship list and will
> be joining that soon. I'm willing to spend some time mentoring a small
> number of people, life permitting.
>
> Ian
>
> On Fri, 26 Oct 2018 at 14:44, Carlton Gibson 
> wrote:
>
>> Hi All.
>>
>> OK, so last week I was at DjangoCon US in San Diego. (Thank you if you
>> organised that! Hi! if we met and chatted.)
>> I gave a talk ("Your web framework needs you!") inspired by the
>> discussion on the
>> 
>> DSF list and the proposal to dissolve Django Core
>> . (Can’t see the DSF list? Join
>> the DSF
>> 
>> .)
>> I was asking for more participation in general, and participation that is
>> more representative of the wider Django community in particular.
>>
>> There was lots of good input from many people, including (but not, at
>> all, limited to) representatives of groups such Pyladies, DjangoGirls, and
>> so on.
>>
>>
>> The recurring themes seem to me to fit into three categories:
>>
>>1. The importance of *mentoring*.
>>2. The difficulty of *finding tickets*.
>>3. The importance of *sprints*.
>>
>> The rest here is a summary of that. Hopefully it’s useful.
>>
>> Mentoring
>>
>> For whatever reasons, the exiting *Contributing How-To*
>>  doesn’t
>> lead to contributions from a demographic that matches the wider Django
>> Community.
>>
>> The point that came up again and again about this was that *mentoring*
>> is one of the best (perhaps the best?) tool in helping to change this.
>>
>> Django Core Mentorship
>>
>> We don’t have an official mentoring programme but we do have the 
>> django-core-mentorship
>> list .
>>
>> This must be about the best-kept secret in the Django world: it’s gets ≈0
>> traffic, but I told everybody at DjangoCon about it, and that they should
>> use it.
>>
>> If you are not on django-core-mentorship, and you’re willing to help
>> prospective contributors, please sign-up. I’m hoping we can drive some
>> traffic to it.
>>
>> Maybe there’s call for something more formal, but at least until DCM is
>> actually being used, that seems (to me) like something we can postpone.
>>
>> Finding Tickets
>>
>> The next thing was that there’s not enough guidance on what to work on.
>>
>> The guidance is to look for *Easy Pickings*. There are ≈1300 accepted
>> open tickets in TRAC. 13 of these are marked *Easy Pickings*.
>>
>> That’s not enough. I think we’re too tight with it (or need another
>> grade).
>>
>> There are *many* tickets which aren’t super hard: I put it that, most of
>> our community solve harder problems every day *using Django* than most
>> tickets require.
>>
>> Yes, they still require time, love, energy, etc — and maybe some
>> mentoring — but it’s not primary research, in the main.
>>
>> I talked to people who had (at the conference) got the test suite running
>> and such, but been overawed by the (for want of a better phrase) *sheer
>> face* of issue tracker.
>>
>> We would do well to invite people better here. (I don’t have instant
>> solutions.)
>>
>> Sprints
>>
>> I’m not historically a Django-sprinter. (I have too many children for
>> that TBH, but they’re getting older…)
>>
>> I always thought it was for a hard-core to work on hard issues.
>>
>> Shows what I know… 
>>
>> It was strongly impressed upon me that the real benefit of sprints is
>> being able to give new contributors the support they need to make their
>> first (or second or…) contribution.
>>
>> In particular, groups such as Pyladies can organise a sprint event with
>> the specific goal of helping members of the community get across the
>> barriers to contributing. This 

Re: Adding a database-agnostic JSONField into Django

2018-10-20 Thread Tom Forbes
Awesome work! Would you mind elaborating on the differences between mysql
and mariadb?

On Sat, 20 Oct 2018, 05:14 Raphael Michel,  wrote:

> Hi everyone,
>
> I used the sprints at DjangoCon US to work on this issue in form of a
> third-party package. Mainly, I created a subclass of
> django.contrib.postgres.fields.JSONField that
>
>- includes code from django-mysql to work on MySQL 5.7+ as well
>- does some nasty hacks to even work on MariaDB 10.2.7+
>- gracefully falls back to storing JSON strings in a TextField on all
>other databases
>
> Unfortunately, the differences between the JSON implementations
> (especially between MariaDB and MySQL) seem to be quite large. Since my
> primary objective was to prove that it is possible to create a
> multi-database field, I resolved this mostly with some back-and-forth data
> conversion hacks, instead of searching for a clean solution. Not all
> features of the PostgreSQL version are possible on MySQL/MariaDB, but the
> most useful ones all are.
>
> This would probably need a lot of cleaning up before considering to move
> it to Django core, and you'd likely also want to explore integrating
> Oracle/SQLite for that. I will probably not have time to do that, but if
> someone else wants to, this might provide a starting point.
>
> Here's the repository: https://github.com/raphaelm/django-jsonfallback
>
> Best
> Raphael
>
> P.S. Thanks to all organizers and attendees for an amazing conference <3
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/330b1ade-ab27-4135-9952-d6a188789c31%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJO8u3oMwH8Mra_eyGqaG%3DKs%3D5d%3DmF8g9U8YBAxswKxCcQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Remove pyinotify autoreloader support

2018-10-08 Thread Tom Forbes
Thanks for the feedback! In the pull request I have re-added support for
pyinotify with tests, it was not as hard to write them as I believed. They
still fail, but I’m working on that!

I’ve found an interesting module confusingly called watchgod. The author
says[1] that with the new os.scandir() method added in python 3.5 a stat
based method can scan a project of 850 files in about 24ms.

While this is watching a single directory tree and not the entirety of
sys.modules I believe we could get similar speedups by being a bit more
intelligent in our approach. For example we really don’t need to stat every
single django.* module every second. I’m guessing that those change very
rarely (unless you’re hacking on Django!), so could maybe stat them every
2–3 seconds or even not at all. We could reduce the stating time on
site-packages modules as well for similar reasons.

If we do this right we could potentially get the overhead down to an
acceptable level that we don’t necessarily need platform specific watchers.

   1.
   
https://github.com/samuelcolvin/watchgod#why-no-inotify–kqueue–fsevent–winapi-support



On 6 October 2018 at 20:56:17, charettes (charett...@gmail.com) wrote:

While I understand the complexity of 1. I think shipping a version of
Django without
an equivalent inotify replacement such as watchdog could be problematic.

>From my personal experience when using Django's development server in Docker
containers sharing local volumes installing pyinotify resulted a
significant performance
CPU and I/O improvement.

Simon

Le samedi 6 octobre 2018 15:32:33 UTC-4, Tom Forbes a écrit :
>
> What do we think about removing the pyinotify functionality in the
> autoreloader? For context, if pyinotify is installed (Linux only) the
> autoreloader will attempt to use it to detect file changes over the
> standard polling-based approach. It is generally much more efficient but is
> not cross platform, and is not well documented currently IMO.
>
> I’m hacking away at my attempt at refactoring the autoreloader (
> https://github.com/django/django/pull/8819) and I’ve made some good
> progress but I am worried about the lack of tests for pyinotify (there are
> none!). The current code in master works but it’s really hard to refactor
> in any meaningful way without them. I would very much like to explore
> adding support for watchdog (https://pythonhosted.org/watchdog/) instead
> of pyinotify and these changes I’m working on are a means to that end.
>
> Wwatchdog is a library that wraps efficient platform-specific filesystem
> notifications in cross-platform way, and it includes an inotify
> implementation.
>
> So I think there are three ways forward:
>
>1.
>
>Spend time and effort adding special tests for PyInotify (testing this
>stuff is not simple, especially with the current code)
>2.
>
>Remove the functionality with an eye to using something like watchdog
>in the near future
>3.
>
>Never really touch the autoreloader much (it is some of the oldest and
>nastiest code we have).
>
> Does anyone have any strong opinions on this? Does anyone use the
> pyinotify speedup while developing?
>
> Tom
>
>
> --
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/62d4d8da-b5ca-4494-96ef-f58917b44e63%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/62d4d8da-b5ca-4494-96ef-f58917b44e63%40googlegroups.com?utm_medium=email_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJNQvxQM3umqXbfRrRXJLsz6aJoSN1%2BKP7J5nMdeNV4TEg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Remove pyinotify autoreloader support

2018-10-06 Thread Tom Forbes
What do we think about removing the pyinotify functionality in the
autoreloader? For context, if pyinotify is installed (Linux only) the
autoreloader will attempt to use it to detect file changes over the
standard polling-based approach. It is generally much more efficient but is
not cross platform, and is not well documented currently IMO.

I’m hacking away at my attempt at refactoring the autoreloader (
https://github.com/django/django/pull/8819) and I’ve made some good
progress but I am worried about the lack of tests for pyinotify (there are
none!). The current code in master works but it’s really hard to refactor
in any meaningful way without them. I would very much like to explore
adding support for watchdog (https://pythonhosted.org/watchdog/) instead of
pyinotify and these changes I’m working on are a means to that end.

Wwatchdog is a library that wraps efficient platform-specific filesystem
notifications in cross-platform way, and it includes an inotify
implementation.

So I think there are three ways forward:

   1.

   Spend time and effort adding special tests for PyInotify (testing this
   stuff is not simple, especially with the current code)
   2.

   Remove the functionality with an eye to using something like watchdog in
   the near future
   3.

   Never really touch the autoreloader much (it is some of the oldest and
   nastiest code we have).

Does anyone have any strong opinions on this? Does anyone use the pyinotify
speedup while developing?

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJPZiKXhJhyOTGiB4xqLr3UBLAwVszicAAbPmFrLhQAsuw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Adding a database-agnostic JSONField into Django

2018-10-04 Thread Tom Forbes
I would also be very interested in helping out. I don’t know if this is
useful, but I’ve had a brief look at the feature support compared to our
current baseline Postgres features:

MySQL: https://dev.mysql.com/doc/refman/8.0/en/json-search-functions.html -
JSON_CONTAINS_PATH() can implement most of our has_* filtering, and we can
also do array index lookups. Will need to convert lookups to path
expressions. There also appears to be some nastiness relating to quoting we
may need to take care of. Also supports partial updating of JSON columns.
The ->> operator can be used for filtering.

Sqlite: https://www.sqlite.org/json1.html - Some trickiness, sqlite
preserves json objects with duplicate keys. Can do simple queries using
json_extract(), but we may need to use json_each() for some lookups (which
adds a bit of complexity). Supports updating values in-place as well.

Oracle:
https://docs.oracle.com/en/database/oracle/oracle-database/18/adjsn/query-json-data.html#GUID–119E5069–77F2–45DC-B6F0-A1B312945590
- Seems to support nesting lookups without needing a path expression,
i.e SELECT
json_field.some_key.some_value FROM table. Supports functions that take
JSON paths as well so perhaps it’s easier to use those. Supports updating
in place.

It seems while all 4 backends support at least the minimum feature set
Postgres does, they do so in pretty different ways (different functions,
different arguments, etc).




On 4 October 2018 at 19:47:03, Adam Johnson (m...@adamj.eu) wrote:

I'd be up for helping with a database-agnostic one. One thing to note about
MySQL world is that MariaDB diverges more from MySQL here and I haven't
found the time to fix the differences here in Django-MySQL's JSONField. I'd
get on with it if I knew someone needed the work for a DB-agnostic field
though ;)

On Thu, 4 Oct 2018 at 12:31, Carlton Gibson 
wrote:

> Sorry, what I meant was a Django field. (I wasn't clear enough.)
>
> Charles Leifer has good posts covering SQLite+JSON
> http://charlesleifer.com/
> His Peewee ORM has a JSONField
> http://docs.peewee-orm.com/en/latest/peewee/sqlite_ext.html#sqlite-ext
>
> I was just wondering if someone knew if someone had already wrapped up a
> Django equivalent?
>
> (If so we'd have reference implementations for all the supported DBs...)
>
> On Thursday, 4 October 2018 13:12:42 UTC+2, Jon Dufresne wrote:
>>
>> > Anyone know of an SQLite implementation?
>>
>> A quick search shows the JSON1 extensions exists:
>> https://www.sqlite.org/json1.html
>>
>> According to the release history (https://sqlite.org/changes.html) this
>> was added in version 3.9.0 (2015-10-14).
>>
>> But I have no direct experience working with it.
>>
>> On Thu, Oct 4, 2018 at 12:19 AM Carlton Gibson 
>> wrote:
>>
>>> This has come up again, with an example implementation for Oracle:
>>>
>>> https://code.djangoproject.com/ticket/29821
>>>
>>> Anyone know of an SQLite implementation?
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/django-developers/215a0e1c-6947-4375-ba1d-38d310c32411%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/593cb04e-6de6-4ccd-9b8d-d175f95d970f%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>


--
Adam
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit

Re: Adding a bulk_save method to models

2018-09-16 Thread Tom Forbes
Thank you all for the feedback, I’ve changed the method to be bulk_update()
as this seems to be the most liked option. Naming things is hard, and while
bulk_update() isn’t perfect I think it’s a bit better than
bulk_update_fields() or just update_fields().




On 16 September 2018 at 10:19:40, Tobias Kunze (r...@rixx.de) wrote:

On 18-09-15 23:15:10, Adam Johnson wrote:
>> I agree bulk_save() maybe is not the best name as people might expect
>> signals to be sent, but are there any suggestions other than
bulk_update()?
>> Maybe something more accurate, like bulk_update_fields()? Or
>> bulk_save_fields()?
>
>bulk_update_fields also sounds good, the longer method name is probably
>balanced by the lower frequency of use.

bulk_update_fields() sounds fine to me, as it makes clearer what
happens. With bulk_update() alone, I'd expect the exact analogous
action to update() to occur, since we're already used to that pattern
from create() vs bulk_create().

update_fields() alone may also work. Upside: it's shorter. Downside:
it's not immediately clear that it takes an iterable and not an
instance. I'd be happy with both options.

Best regards,
Tobias

-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/20180916091931.7hlmh2xc5eo5z7ws%40ronja.localdomain.

For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOHYTA-%3D6BYVm7Doe%3DE2%3DAWgnJ-F7%2B_UKgzAsiH%2B0bm-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Adding a bulk_save method to models

2018-09-15 Thread Tom Forbes
My original reasoning was that Queryset.update() already bulk updates rows,
so the bulk prefix seems a bit redundant here (how do you bulk something
that already does something in bulk?). .save() however operates on a single
object, so the bulk prefix seems more appropriate and easier to understand.

I agree bulk_save() maybe is not the best name as people might expect
signals to be sent, but are there any suggestions other than bulk_update()?
Maybe something more accurate, like bulk_update_fields()? Or
bulk_save_fields()?




On 14 September 2018 at 16:32:05, Raphael Michel (m...@raphaelmichel.de)
wrote:

Hi,

I'd be very careful about calling it bulk_save(), since calling
it something with save() very strongly suggests that it calls pre_save
or post_save signals.

Best
Raphael


Am Fri, 14 Sep 2018 07:56:38 -0700 (PDT)
schrieb Tim Graham :

>
>
> I wanted to ask about naming of the new method. Currently the
> proposed name is "QuerySet.bulk_save()" but I think it's a bit
> confusing since it uses QuerySet.update(), not Model.save(). It works
> similarly to QuerySet.bulk_update() from
> https://github.com/aykut/django-bulk-update but the arguments are a
> bit different.
>
>
> Josh's comment on the PR: "Since this only works for instances with
> an pk, do you think that bulk_update would be a better name? The
> regular save() method can either create or update depending on pk
> status which may confuse users here."
>
> And Tom's reply: "I considered this, but queryset.update() is the
> best 'bulk update' method. I didn't want to confuse the two, this is
> more about saving multiple model fields with multiple differing
> values, gene bulk_save. Open to changing it though."
>
>
> On Tuesday, January 23, 2018 at 7:38:18 AM UTC-5, Neal Todd wrote:
> >
> > Hi Tom,
> >
> > That's great, should be a helpful addition to core. Will follow the
> > ticket and PR.
> >
> > Neal
> >
> > (Apologies - I hadn't spotted that you'd already referenced
> > django-bulk-update in your ticket when I left my drive-by comment!)
> >
> > On Monday, January 22, 2018 at 7:41:11 PM UTC, Tom Forbes wrote:
> >>
> >> Hey Neal,
> >>
> >> Thank you very much for pointing that out, I actually found out
> >> about this package as I was researching the ticket - I wish I had
> >> known about this a couple of years ago as it would have saved me a
> >> fair bit of CPU and brain time!
> >>
> >> I think that module is a good starting point and proves that it’s
> >> possible, however I think the implementation can be improved upon
> >> if we bring it inside core. I worked on a small PR to add this
> >> <
https://github.com/django/django/pull/9606/files#diff-5b0dda5eb9a242c15879dc9cd2121379R473>

> >> and the implementation was refreshingly simple. It still needs
> >> docs, a couple more tests and to fix a strange error with sqlite
> >> on Windows, but overall it seems like a lot of gain for a small
> >> amount of code.
> >>
> >> Tom
> >>
> >>
> >> On 22 January 2018 at 15:10:53, Neal Todd (ne...@torchbox.com)
> >> wrote:
> >>
> >> Hi Tom,
> >>
> >> A built-in bulk save that's more flexible than update would
> >> certainly be nice. Just in case you haven't come across it though,
> >> there is a package called django-bulk-update:
> >>
> >> https://github.com/aykut/django-bulk-update
> >>
> >> I've found it very useful on a number of occassions where update
> >> isn't quite enough but the loop-edit-save pattern is too slow to
> >> be convenient.
> >>
> >> Probably some useful things in there when considering the API and
> >> approach.
> >>
> >> Cheers, Neal
> >>
> >> On Friday, January 19, 2018 at 5:49:48 PM UTC, Tom Forbes wrote:
> >>>
> >>> Hello all,
> >>>
> >>> I’d love for some feedback on an idea I’ve been mulling around
> >>> lately, namely adding a bulk_save method to Dango.
> >>>
> >>> A somewhat common pattern for some applications is to loop over a
> >>> list of models, set an attribute and call save on them. This
> >>> unfortunately can issue a lot of database queries which can be a
> >>> significant slowdown. You can work around this by using
> >>> ‘.update()’ in some cases, but not all.
> >>>
> >>> It seems it would be possible to use a CASE statement in SQL to
> >>> handle bulk-updating many rows with differing values. For example:
>

Re: Add autocomplete attribute to contrib.auth fields?

2018-08-25 Thread Tom Forbes
I don’t have much to add other than it’s pretty common for pentests to flag
autocomplete being enabled on sensitive fields (email/password) and
recommend disabling it (autocomplete=off). While I’m not sure if I agree
with that recommendation in some situations you have little choice but to
follow it.




On 25 August 2018 at 16:54:08, Tim Graham (timogra...@gmail.com) wrote:

Browser support looks somewhat limited, so I wanted to ask if there are any
concerns or drawbacks with adding
autocomplete=username/email/current-password/new-password to contrib.auth's
forms?


Pull request: ​https://github.com/django/django/pull/9921


>From the ticket [https://code.djangoproject.com/ticket/29379]:


The most useful one is autocomplete=new-password, which prevents browsers
prefill with current password, Chrome will also suggest a random strong
password for users who turned on account sync.
Related docs:
​
https://html.spec.whatwg.org/multipage/form-control-infrastructure.html#autofill
​
https://www.chromium.org/developers/design-documents/form-styles-that-chromium-understands
​
https://developer.mozilla.org/en-US/docs/Web/Security/Securing_your_site/Turning_off_form_autocompletion#The_autocomplete_attribute_and_login_fields
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/d398c554-3fe2-4e0f-9deb-a61dabc4cbf3%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMektU_6pQ6vXnRrBHEisGbUzp8sm-cE2m0-597dShHGA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Django and Cython

2018-08-03 Thread Tom Forbes
The people showing you benchmarks comparing Django to ‘NodeJS’ are
comparing apples to oranges. Django is not an asynchronous framework (yet!)
so you cannot fairly compare the number of raw requests per second the two
handle and present that as evidence one is better than the other. Each one
has it’s strengths and weaknesses but Django wins hands down in a number of
places. Plus it’s not JavaScript, so that’s one thing Django has going for
it!

I’ve had immense success with uvloop and asyncio which are much more
comparable to NodeJS. Here are some synthetic benchmarks showing it handing
over 100,000 requests per second:
https://magic.io/blog/uvloop-blazing-fast-python-networking/. I would
recommend playing around with asyncio and aiohttp to build some small
services and benchmark them yourself, you might be surprised at the speed.

Regarding Cython: while some very specific parts of Django could
potentially benefit from the speedup Cython would bring we would need to
maintain two separate versions: A Cython one and a fallback Python one.
This would add a maintenance burden for an uncertain gain and even if you
implemented this the raw requests per second Django would reach would not
be close to that of an asynchronous framework with the same resources.
Typically Django apps are not CPU bound by the Django code itself, and a
lot of time is spent waiting on the network for the database or other
services.




On 3 August 2018 at 18:18:59, Daniel Anechitoaie (
daniel.anechito...@gmail.com) wrote:

So the fact that frameworks like Vibora and webapps developed in other
faster programming languages like GO/NodeJS/etc. support so many more req/s
than Django it's relevant only in benchmarks?
And in real life/live web apps the difference in performance won't be
noticeable?

I have to mention again that I really don't try to start any flame war and
I'm legit interested in this.
As a huge Python fan and unfortunately lone Python dev among my peers they
all flame me when I bring Python up and show me all kind of benchmarks how
fast NodeJS (as this is what they use) is vs Python.
I'm just trying to make sure I have ground to stand upon.


On Thursday, August 2, 2018 at 3:31:37 PM UTC+3, Jason Johns wrote:
>
> There was a discussion a while back about this https://groups.google.
> com/forum/#!searchin/django-developers/cython/django-
> developers/Fi4U602GxHA/mE50LOPkBgAJ
>
> tl;dr not sure what benefits Django would get from it, since the
> bottlenecks you experience are most likely non-Django/Python parts of your
> project, such as networking latency, db queries, connection initiation,
> etc.  In addition, pypy is an alternative interpreter that you can drop in
> for up to 3.5.x Python versions.
>
> On Wednesday, August 1, 2018 at 5:21:38 PM UTC-4, Daniel Anechitoaie
> wrote:
>>
>> I'm looking at frameworks like https://vibora.io/ that make use of
>> Cython to greatly improve the performance and I'm just wondering if
>> something like this would work with Django?
>> Was there any discussion (taken into consideration) about using Cython to
>> boost the performance of certain aspects of the framework?
>> Would it help make it faster?
>>
>> I really love Python and Django for me is the best web framework out
>> there, but you have to notice how fast other frameworks from other
>> programming languages are.
>> Would using Cython help boost the performance?
>>
> --
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/24d4d03d-d9ed-4d7f-9a69-8066846f090e%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJNkcVcjHmuzJ35NAN2mLGhHbWopWribGj3_mGU4hDqcyg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Integrate dj-database-url into Django

2018-07-28 Thread Tom Forbes
So in the PR I proposed I only bits I took verbatim from dj-database-url
are the tests. The rest is re-implemented. I think it's a pretty good POC
but I haven't touched it in a while.

In any case we have to implement our own parsing for backends that do not
support passing in a URL to the connection library.

Making postgres skip our parsing step and passing it in directly is an
implementation detail and there are much more important questions around
the API design to answer before this has any chance of being included.

On Sat, 28 Jul 2018, 12:57 Maciej Urbański,  wrote:

> I would agree that DSN support seems like a nicer alternative to just
> copying dj-database-url, because it not only focuses on 12factor
> configuration in enviroment variables, but also enables some additional
> flexibility for the database connection option passing.
>
> As for 12factor, I think https://gist.github.com/telent/9742059 is a good
> read as to why exposing as enviroment variables maybe not the best
> motivation. Having to accommodate settings, like database connection
> information, just so they can be fitted into single string put conveyable
> by enviroment variable is a case in point. We likely can do the same for
> Cache addresses as mentioned previously, although we may end up inventing
> new URI schemes do to that.., but django overall has multitude of other
> options that are not as easy to stringify.
>
> On Friday, 27 July 2018 19:14:12 UTC+2, gw...@fusionbox.com wrote:
>>
>> I'd like to approach this as 'support database urls in django', rather
>> than 'copy/paste dj-database-url into django'. For postgres (I'm not sure
>> about other backends), a database url can be passed directly to psycopg2.
>> The postgres connection string format actually supports more features than
>> is possible with django's HOST/USER/PORT... style settings. For example,
>> you can pass multiple hosts and psycopg2 will attempt to connect to one of
>> them: https://paquier.xyz/postgresql-2/postgres-10-multi-host-connstr/.
>> Any attempt to parse the url in django will result in a loss of those
>> features.
>>
>> The only problem I see is that we have to parse the database backend out
>> of the url, and you can't pass a url like 'postgis://' to psyscopg2.
>> I'd like to be able to do something like:
>>
>> DATABASES = {
>> 'default': {
>> 'DSN': 'postgres://',
>> 'ENGINE': 'django.contrib.gis.db.backends.postgis',
>> },
>> }
>>
>> And let psycopg2 handle the DSN.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/1a55cc1c-ba9c-4950-ab94-50da8eec7d06%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMa1xSzBeGUYygG7nxfCVz8jUPNEiSEJhbAqLDTwga9BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Optionally using a custom queryset for Paginator and MultipleObjectMixin (ListView)

2018-07-17 Thread Tom Forbes
Indeed, I had an attempt at doing this here (
https://github.com/django/django/pull/8928/), but it seems a hard problem.
I think there is huge potential here but I have no time to work on this
anymore.

On Tue, 17 Jul 2018, 19:00 Ramiro Morales,  wrote:

> On Sat, Jun 30, 2018 at 12:40 PM Tom Forbes  wrote:
>
>> Are you sure it is the prefetches that is causing this? As Adam pointed
>> out these are correctly ignored. Annotations however are not, which can
>> cause unnecessary work and longer execution times. i.e:
>>
>> Book.objects.annotate(something=Max('pages__word_count')).count()
>>
>> We have enough information to be able to strip the pages join when using
>> count() it certain conditions are met, and this might be preferable than
>> adding workarounds to various places.
>>
>
> FYI: This had already been reported in #23771 (
> https://code.djangoproject.com/ticket/23771) (and a couple of
> similar/duplicates tickets)
>
> Regards,
>
> --
> Ramiro Morales
> @ramiromorales
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAO7PdF_Y_vQqYNst5ZR1d9vxWaZMqhZwfF3fcZ4PC7tQv0hGJw%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-developers/CAO7PdF_Y_vQqYNst5ZR1d9vxWaZMqhZwfF3fcZ4PC7tQv0hGJw%40mail.gmail.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOf-rN%3DMmOMTxjSEjjrC7FOGwuOLF7zXGqP79aeg0-68g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: MariaDB, official support

2018-07-05 Thread Tom Forbes
I think that given Adams plan it might be possible to get this done before
2.2 as there are not too many failures. I’m not entirely convinced that
including MariaDB logic inside the MySQL backend is a permanent solution.

It might be the path of least resistance right now, and a good idea to get
this in before 2.2, but if we care enough to make a fully supported backend
then we might also care enough to remove any workarounds we have for MySQL
(like 27676  + and others we
discover), plus support any additional features or syntax that MariaDB
includes. This could get pretty complex if we shoehorn it into one backend
because we are going to be supporting a larger compatibility matrix.




On 5 July 2018 at 14:45:35, Carlton Gibson (carlton.gib...@gmail.com) wrote:

This seems a bit more pressing now: 2.0.7 broke MariaDB compatibility.

https://code.djangoproject.com/ticket/29544

I'm sure that can be fixed. But we're in a tricky position now.

Claude has marked the ticket as a Release Blocker, which is understandable,
but I can't see how we can
both support and not-support MariaDB, especially when we don't have test
coverage on CI to support it.

Adam, you came to mind immediately when I saw the ticket here, since you've
talked about this.
(So I'm glad you're already on this thread.)

Where are we on MariaDB support? What timescale can we add it (if at all)?
What can we realistically do in the interim?

v2.2 is scheduled for the end of 2018. That's the earliest we can get it in
now.
What's a realistic workflow if we're going to consider MariaDB breaking
changes as out of bounds before then?

Thanks for the input all!

C.
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/4cf1e564-e0ad-4576-ba53-4d9276633cff%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJNb6n9Yp81KixvubewqwVe62erTw-WAvfNMycB3UFAm_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Optionally using a custom queryset for Paginator and MultipleObjectMixin (ListView)

2018-06-30 Thread Tom Forbes
Are you sure it is the prefetches that is causing this? As Adam pointed out
these are correctly ignored. Annotations however are not, which can cause
unnecessary work and longer execution times. i.e:

Book.objects.annotate(something=Max('pages__word_count')).count()

We have enough information to be able to strip the pages join when using
count() it certain conditions are met, and this might be preferable than
adding workarounds to various places.


On Fri, 29 Jun 2018, 22:00 Jakub Kleň,  wrote:

> Hi guys, I came across a problem today while testing my webapp with a full
> migrations applied from my legacy site for the first time.
> The problem is I'm using Prefetch in my view query (get_queryset), and it
> works fine when I just fetch it, but the problem comes with the .count()
> query which generates sql with subqueries, and it takes really long to
> process by MySQL.
> The nicest solution I'm thinking about would be to use 2 different queries
> for object_list / pagination count, because for count the query would be
> really simple, and wouldn't need the prefetches.
> I think this scenario may be pretty likely to come across also for other
> Django users, so I'm thinking if it wouldn't be nice to have it directly in
> Django.
> From my point of view, it seems like the implementation would be pretty
> straightforward and easy, and I would even like to look into it and
> contribute to the Django project if possible and if you think it would be
> useful.
> I would like to add count_queryset param to Paginator.__init__, and def
> get_count_queryset to MultipleObjectMixin. Happy to hear your opinions on
> this :)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/a29e6b81-f05c-46af-a479-04e1f4bce705%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJNKOE-8QA4w-74U42FCnX%2B685W9Wm8MJN_L0b7QVZYVKQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: A Django Async Roadmap

2018-06-05 Thread Tom Forbes
> Unfortunately even if you have context variables, you simply can't await
inside of an attribute reference because there's a synchronous call in your
stack. I even chatted to some Python core devs at PyCon US about this and
we couldn't really think of a way out of this problem without some very
serious changes to the language.

I’m just spitballing here but could we not conditionally return a
future/promise if some conditions are met, then implement some kind of
chaining based on the attribute accesses? So
`instance.related_field.other_related_field` breaks down into the
`related_field` returning some kind of awaitable instance, and we return a
nested instance when `other_related_field` is accessed? This last future
would resolve to ’the right thing’. Perhaps just throwing an exception if
we are in an async loop is an acceptable and simpler trade off however just
as long as the existing synchronous implementation works as-is.

>  keyword arguments (like "my_method(..., async=True)"), suffixes
("my_method_async()", as you suggest), automatically changing based on if
you're in an async thread (which is a bit... magic) or different module
namespaces ("from django.db.async import foo”).

If we go with a `_async` suffix we could reduce a lot of boilerplate with a
bit of magic. We could potentially auto-generate the sync versions of the
methods with a decorator, seeing as they would just call the async version
and wait for the result? I’m not sure if we could do the same nicely with a
different namespace, but that might be cleaner.

On 5 June 2018 at 07:36:49, Andrew Godwin (and...@aeracode.org) wrote:


> I think getting rid of the related field references could be a big issue
> here and cause a lot of headaches for existing applications. Could we do
> this in a backwards compatible way at all? I wonder if PEP 567
>  could help here, could we
> define some kind of ‘Django async context’ and do different things
> depending on if this is true or false?
>
Unfortunately even if you have context variables, you simply can't await
inside of an attribute reference because there's a synchronous call in your
stack. I even chatted to some Python core devs at PyCon US about this and
we couldn't really think of a way out of this problem without some very
serious changes to the language.

What will work, though, is attribute access from a sync context - the
related field references will only error out of they know they're in an
async loop, and we can detect that by looking for an active event loop on
the current thread (no context varaible needed).


> Regarding exposing an async interface alongside a synchronous one: are you
> envisaging something like appending _async to methods or having some kind
> of wrapper class that could be optionally included to go from async->sync?
> I guess it would have to be appending _async, as a wrapper class could be
> used in different contexts.
>

I'm not quite sure on this one. For backwards compatibility, we have to
keep sync methods working with the same name, but there are several options
as to how to separate them - keyword arguments (like "my_method(...,
async=True)"), suffixes ("my_method_async()", as you suggest),
automatically changing based on if you're in an async thread (which is a
bit... magic) or different module namespaces ("from django.db.async import
foo").

I want to sketch out what all of these look like as part of this project
and then work out which is best for Django.


> Async templates seem particularly powerful if we work out the details. We
> could eventually render different parts of the template concurrently, i.e
> example each iteration of a for loop could be it’s own future resolved
> independently, but this is likely a pipe dream.
>

That would very much be a long-term thing, and honestly something I might
consider handing off to something like Jinja (which already has full async
support, I learnt yesterday)

Andrew
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CAFwN1upPVQCjHTxgJmzJSUzaevsfQ7evwi89hFt4OvHmMe5m2A%40mail.gmail.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

Re: A Django Async Roadmap

2018-06-04 Thread Tom Forbes
Hey Andrew, thank you for the very exciting proposal and the ongoing work
on Django channels! There is a lot to consider here and some very
interesting problems to work through - I’d love to help out wherever I can.

Do we think async is worth going after?

I think this is very much worth doing. Async interfaces to the ORM and
templates would be a huge, huge improvement if we can get it right. I think
Tom Christie summed it up really well however I would add that I think
teams excluding Django as a contender may happen sooner rather than later
if it’s not happening already.

Are the proposed modifications to how Django runs sensible?

I had a few random thoughts while reading your proposal:

I think getting rid of the related field references could be a big issue
here and cause a lot of headaches for existing applications. Could we do
this in a backwards compatible way at all? I wonder if PEP 567
 could help here, could we
define some kind of ‘Django async context’ and do different things
depending on if this is true or false?

Regarding exposing an async interface alongside a synchronous one: are you
envisaging something like appending _async to methods or having some kind
of wrapper class that could be optionally included to go from async->sync?
I guess it would have to be appending _async, as a wrapper class could be
used in different contexts.

Async templates seem particularly powerful if we work out the details. We
could eventually render different parts of the template concurrently, i.e
example each iteration of a for loop could be it’s own future resolved
independently, but this is likely a pipe dream.




On 4 June 2018 at 14:18:19, Andrew Godwin (and...@aeracode.org) wrote:

Hello everyone,

For a while now I have been working on potential plans for making Django
async-capable, and I finally have a plan I am reasonably happy with and
which I think we can actually do.

This proposed roadmap, in its great length, is here:

https://www.aeracode.org/2018/06/04/django-async-roadmap/

I'd like to invite discussion on this potential plan - including:

 - Do we think async is worth going after? Note that this is just async
HTTP capability, not WebSockets (that would remain in Channels)

 - Can we do this in a reasonable timeframe? If not, is there a way around
that?

 - Are the proposed modifications to how Django runs sensible?

 - How should we fund this?

There's many more potential questions, and I really would love feedback on
this. I'm personally pretty convinced that we can and should do this, but
this is a decision we cannot take lightly, and I would love to hear what
you have to say.

Andrew
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/CAFwN1up4%2BP%2B4bCSSiG6t845idoM8dvktnuTs4dH2R5aAuCz3iA%40mail.gmail.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMTTXPK5vjE7uKm8nKcZJOJTSyWA3xGVtxaGEtrQuUkfA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Proposal: security enhancements

2018-05-04 Thread Tom Forbes
Hey James,

I think these ideas are fantastic.

I used EmberJS for a project and the development server contained a built
in CSP report URL which just printed what the browser sent to the console.
This was very useful during development as you could immediately see CSP
errors that were triggered rather than perhaps having to navigate to
another page and seeing the issues without context. It would be great if we
could add this functionality to Django, perhaps even in production - if we
provided a route that would output CSP errors to a specified logger users
could configure the logger to send the errors to a variety of places
without too much configuration or complexity?

The rel=noopener would be pretty hard to implement IMO, and while noble I’m
not sure how we could even do this (outside of the Django admin)?

I don’t have much else to add, other than I’d love to help with the
implementation of any of these if I’m able.

Tom




On 1 May 2018 at 08:28:18, James Bennett (ubernost...@gmail.com) wrote:

I've written this up in pseudo-DEP format partly for ease of
organization, and partly because I'm unsure whether it would require a
DEP. Right now I'm just throwing it out here as a proposal, and
offering to work on implementing it; if you have questions, concerns,
or suggestions for things to add to it, please let me know :)


Introduction
-

Django has acquired a reputation for being helpful with/good at
"security". Which has been a focus since at least 1.0; Django goes out
of its way to protect you out of the box against common security
problems, and includes a lot of tools geared toward making it easy to
do the right thing when building apps.

But security is a Red Queen problem; we can't ever pause and rest,
since as soon as we do we'll fall behind.

In light of which, I'd like to propose a few security-related
improvements/additions, and offer to put in the time to help implement
them. My baseline goal here is to set things up such that it's
possible, using only things that ship with Django itself and an
SSL-enabled hosting service, to score an A+ rating on
securityheaders.com. Beyond that, some stretch goals are possible.


Content Security Policy


CSP[1] is one of the more important defenses available against XSS and
a few other types of attacks. Django does not currently have
out-of-the-box support for it; the usual solution is to install
django-csp[2], configure it and set up its middlware to emit the CSP
header.

I'd like to see support for CSP built in to Django. This could be done
by integrating django-csp, which is open source and uses a compatible
license, though there'd need to be some updates to its documentation
(which seems to lag a bit behind what the code supports) and it might
also be good to switch over to using dictionary-based configuration.


Referrer-Policy
-

The Referrer-Policy header[3] is relatively new, but gaining support
in major browsers. It helps enhance both security and privacy.

I wrote a package earlier this year[4] that supports sending the
Referrer-Policy header, and BSD-licensed it. It would be relatively
easy to integrate this into Django.


Subresource integrity
-

Subresource integrity[5] allows the author of an HTML document to
supply the expected hashes of resources (scripts, stylesheets, images,
etc.) referenced in the document. This helps to harden against not
only local breaches, but also breaches of third-party scripts and
CDNs.

I don't have a clear vision yet of how it would work, but I'd like to
see the form media system and the staticfiles app grow support for
SRI. In the case of form media, the output already generates the full
HTML and could include the hash. In the case of the {% static %}
template tag, I'm open to suggestions for how it could
generate/include the hash. The admin should use this by default.


CORS
-

Cross-Origin Resource Sharing is increasingly important on the modern
web, and Django does not currently support setting CORS headers. There
is an open-source package for CORS in Django[6], which could be
integrated.


rel="noopener"
-

Setting rel="noopener" on links which will open in new tabs/windows is
an easy way to head off certain types of surprising attacks[7]. Like
SRI, I don't yet have a great plan for how this could be supported
easily/automatically in Django, but I'd like to give it a try.


Improved system-check integration
---

Currently the security system-check is OK, but still a bit
minimal. I'd like to add support for checking at least everything I've
proposed adding above, and probably several more items. In my ideal
world, a service like securityheaders.com could be replicated by the
security check.

In my magical stretch-goal land, I'd also figure out a way to support
the pyup safety library[8] to scan for a requirements file and any
dependencies in 

Re: please help me i am trying to solve it from 5 days but still did not get the solution so please help me here is problem

2018-04-15 Thread Tom Forbes
This mailing list is for the development of Django itself, not for support.
Use the django-users mailing list for that, or IRC #django on freenode, or
a site like Stack Overflow.



On 15 April 2018 at 14:53:50, rahul verma (rahulverma6...@gmail.com) wrote:

https://github.com/rahul6612/flying-store/issues/1?
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/4edfde6b-c4db-4cb8-b0f5-fe87912bc89b%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJNdvrb8q3z9cwGw_5eR_Hf66A0QiLjFTMATO8zR54-FZw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Shouldn't manage.py call python3 instead of python?

2018-04-08 Thread Tom Forbes
It may be an obstacle but I believe it’s better than having them nuke their
base systems by accident by installing a package that conflicts with their
base system. This isn’t such a huge issue on MacOS but on Linux it is and
I’ve seen it happen a few times. Not to mention the issue of multiple
conflicting dependencies across projects - all in all it’s really not a
recommended and we should not look to make it easier IMO.

People have different setups and whatever works, works, but things like
pipenv are maturing rapidly and solve the convenience issue you describe. I
personally use virtualenvwrapper which is really simple to set up and
displays the current virtual environment in the prompt, and makes it really
easy to switch between them/create new ones.

Tom



On 8 April 2018 at 15:00:46, Bobby Mozumder (bmozum...@gmail.com) wrote:

I never really liked the idea of using VirtualEnv or HomeBrew over the
default installation in Mac OS.  (FreeBSD has the same naming issues).

Having beginners use VirtualEnv or HomeBrew always struck me as a huge
obstacle to getting a beginners Django developer's environment operational,
as well as being a huge pain-in-the-ass of always setting VirtualEnvs for
each shell.  So, I personally don’t use them anymore, and just use the base
system now.

I wish there was a process of running Django out-of-the-box from a default
Mac OS install.

-bobby

On Apr 8, 2018, at 8:27 AM, Tom Forbes <t...@tomforb.es> wrote:

This only seems to be an issue when you are using the base system
interpreter to run manage.py. installing Django and other dependencies
there is not recommended for a variety of reasons, and this isn't a problem
when using a virtualenv, it doesn't seem like there is much to fix IMO.


On Sun, 8 Apr 2018, 08:19 Bobby Mozumder, <bmozum...@gmail.com> wrote:

> Is it OK to reopen that ticket?
>
> The problem is that python2 and python3 need to coexist in most systems,
> and you can’t just rename python3 to python.
>
> -bobby
>
> On Apr 6, 2018, at 8:30 PM, Tim Graham <timogra...@gmail.com> wrote:
>
> It was tried in https://code.djangoproject.com/ticket/27878 but it caused
> problems, particularly on Windows.
>
> On Friday, April 6, 2018 at 6:35:50 PM UTC-4, Josh Smeaton wrote:
>>
>> I think you're right and PEP394 is the relevant text:
>> https://www.python.org/dev/peps/pep-0394/
>>
>> TL;DR
>>
>> For now, *python* should refer to python2 and *python3* should be used
>> to refer to python 3.
>>
>> On Saturday, 7 April 2018 07:07:35 UTC+10, Bobby Mozumder wrote:
>>>
>>> The header of manage.py has: #!/usr/bin/env python
>>>
>>> Shoudn’t it be: #!/usr/bin/env python3
>>>
>>> Since 2.0 is now only Python3. Both my Mac OS & FreeBSD environments
>>> have Python 3.5+ as “python3". (I’m not sure about Linux or other
>>> environments).
>>>
>>> Is that a bug I need to file?
>>>
>>> -bobby
>>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/7cdf48bb-ab0b-449d-8f33-a4c6d369%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/7cdf48bb-ab0b-449d-8f33-a4c6d369%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/E1881F92-2D8C-45D8-8315-E5D72D0D7B6E%40gmail.com
> <https://groups.google.com/d/msgid/django-developers/E1881F92-2D8C-45D8-8315-E5D72D0D7B6E%40gmail.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop 

Re: Shouldn't manage.py call python3 instead of python?

2018-04-08 Thread Tom Forbes
This only seems to be an issue when you are using the base system
interpreter to run manage.py. installing Django and other dependencies
there is not recommended for a variety of reasons, and this isn't a problem
when using a virtualenv, it doesn't seem like there is much to fix IMO.


On Sun, 8 Apr 2018, 08:19 Bobby Mozumder,  wrote:

> Is it OK to reopen that ticket?
>
> The problem is that python2 and python3 need to coexist in most systems,
> and you can’t just rename python3 to python.
>
> -bobby
>
> On Apr 6, 2018, at 8:30 PM, Tim Graham  wrote:
>
> It was tried in https://code.djangoproject.com/ticket/27878 but it caused
> problems, particularly on Windows.
>
> On Friday, April 6, 2018 at 6:35:50 PM UTC-4, Josh Smeaton wrote:
>>
>> I think you're right and PEP394 is the relevant text:
>> https://www.python.org/dev/peps/pep-0394/
>>
>> TL;DR
>>
>> For now, *python* should refer to python2 and *python3* should be used
>> to refer to python 3.
>>
>> On Saturday, 7 April 2018 07:07:35 UTC+10, Bobby Mozumder wrote:
>>>
>>> The header of manage.py has: #!/usr/bin/env python
>>>
>>> Shoudn’t it be: #!/usr/bin/env python3
>>>
>>> Since 2.0 is now only Python3. Both my Mac OS & FreeBSD environments
>>> have Python 3.5+ as “python3". (I’m not sure about Linux or other
>>> environments).
>>>
>>> Is that a bug I need to file?
>>>
>>> -bobby
>>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/7cdf48bb-ab0b-449d-8f33-a4c6d369%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/E1881F92-2D8C-45D8-8315-E5D72D0D7B6E%40gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJORs_f%3D70fkjdCRX%2BHwY4%2B%3DTk2Ns8TZwU-m7zboTY8Ssg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: SafeExceptionReporterFilter should obfuscate all variables in every stack frame below a function decorated with sensitive_variables

2018-04-06 Thread Tom Forbes
My only concern is that it would greatly reduce the usefulness of the
exception reporter, and might lead to people removing the
sensitive_variables decorator in order to see exception tracebacks.

Perhaps rather than obscure all local variables below the decorated
function, we could obscure any variable with the same *value* as a
sensitively marked variable in the call stack? This would work better with
strings than say dictionaries, but it could be a good middle ground.

Tom

On Wed, 4 Apr 2018, 21:05 ncvc via Django developers (Contributions to
Django itself),  wrote:

> SafeExceptionReporterFilter obfuscates variables in the function decorated
> with sensitive_variables, but it does not obfuscate variables lower in the
> call stack, which could result in sensitive data being leaked in exception
> reports.
>
> For instance:
>
> @sensitive_variables('sensitive')
> def decorated_function():
> sensitive = 'something sensitive'
> undecorated_function(sensitive)
>
> def undecorated_function(var):
> raise Exception()
>
>
> In this code, the "sensitive" variable will be obfuscated in
> the decorated_function stack frame, but "var" in the undecorated_function
> stack frame will not, resulting in the sensitive data being leaked in the
> report. If we wrote undecorated_function, then we can just decorate the
> function ourselves, but if it's from a third-party package, we are unable
> to decorate it.
>
> The solution here is to obfuscate _all_ variables in all stack frames
> below a function decorated with sensitive_variables, since these functions
> can do arbitrary things with the sensitive data. I've written a custom
> SafeExceptionReporterFilter that does this for the company I work for, and
> I think it would be a good behavior to adopt in core Django.
>
> Any thoughts or concerns with this approach?
>
>
> This message, including any attachments, is a PRIVATE communication, which
> may contain confidential, legally privileged, and/or proprietary
> information.  If you are not the intended recipient, you are hereby
> notified that any dissemination, disclosure, copying, distribution or use
> of the information contained in or attached to this message is strictly
> prohibited.  Please notify the sender of the delivery error by replying to
> this message, and then permanently delete it from your system.  Unless
> explicitly stated to the contrary, nothing contained in this message shall
> constitute an offer to buy or sell, or a solicitation of an offer to buy or
> sell, any security, property interest or other asset, nor shall it
> constitute a binding obligation of any kind, an official confirmation of
> any transaction or an official statement of Cadre.
>
> Cadre may monitor, review and retain email communications traveling
> through its networks or systems, AND CADRE IS NOT OBLIGATED TO RESTRICT THE
> USE OR DISCLOSURE OF ANY INFORMATION SENT TO IT BY YOU VIA E-MAIL
> COMMUNICATION.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/7376ab0c-530c-42d8-9cfb-6c829af21098%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJNPoU0S2_44S3frL4%2BdkjYqOXYEw%2BRP4-tT1z_Lx%2BuRqg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: GSoC 2018

2018-03-16 Thread Tom Forbes
Hey Manasvi,

I'm perhaps a bit biased but I would be very interested in anything that
can make JavaScript a real first class citizen in Django. There are a
number of third party packages out there that attempt something like this
(i.e django-webpack-loader). Django is classically quite conservative, and
up until relatively recently the JavaScript ecosystem has been in flux with
a relatively high amount of churn when it comes to build tools and best
practices.

Things have somewhat stabalized recently with webpack achieving widespread
adoption so perhaps it's time to evaluate the landscape and see if we can
improve in this area. I'd love to see some kind of pluggable JavaScript
build system that would make modern JavaScript a first class citizen
alongside templates and other static assets, separate or as an extension to
the classic static assets functionality in Django.

All in all I am quite jealous of how Rails handles JavaScript
integrations[1]. Again, this is just my personal opinion and I cannot say
if pursuing this will in any way effect your chances of being accepted but
this seems likes really interesting direction to investigate and more
practically useful than yet another way of generating static server side
HTML.

Tom
×

1. https://medium.com/@hpux/rails-5-1-loves-javascript-a1d84d5318b

On 17 Mar 2018 03:14, "Manasvi Saxena" <msmana...@gmail.com> wrote:

Hello Sir,

On Saturday, March 17, 2018 at 5:39:34 AM UTC+5:30, Tom Forbes wrote:

> Hey Manasvi,
> I don't have any say in the choice of a GSOC student, but I'd like to add
> my two cents nonetheless.
>
> I can see the logic behind your proposal, but I'm skeptical about the
> usefulness of such a project. Libraries that implemented something similar
> to this have come and gone, and got good reason, they where not very
> practical and other solutions where much easier to use.
>
> In the time since frontend frameworks have proliferated and JavaScript has
> become much more mature. Anything we do in this space seems like it will be
> dead on arrival compared to something like React or Ember, which already do
> this but with huge huge advantages over your approach.
>
> In my humble opinion anything in this space will have to treat JavaScript
> as a first class citizen - i.e something that makes the interoperability
> between a JavaScript frontend and a Django backed easier would have some
> benefits, but generating HTML in pure python on the backend by writing
> components in Python does not have a bright future IMO.
>
> Tom
>
> On 16 Mar 2018 22:36, "Manasvi Saxena" <msma...@gmail.com> wrote:
>
>> Hello Sir,
>>
>> Okay, so the idea is to render HTML by generally defining it in Python. I
>>> could've sworn that I'd seen something like this years ago, but I'm failing
>>> to find it. That stated, I think this is basically a generating version of
>>> BeautifulSoup (https://www.crummy.com/software/BeautifulSoup/bs4/doc/)
>>> as opposed to a parsing version.
>>>
>>> It's roughly like the Storm ORM (https://storm.canonical.com/) but for
>>> HTML instead of database queries.
>>>
>>> It's interesting, but I'll ask if one can get 90% of the functionality
>>> from xml.etree, which is in the standard Python Library.
>>>
>>> import xml.etree.ElementTree as ET
>>>
>>> a = ET.Element('p', style='{color: red;}')
>>> a.text = "hello world"
>>> ET.dump(a) # will match yours
>>>
>>> Note that this gets even weirder with something like "hello ignore
>>> this world".
>>>
>>> b = ET.Element('p')
>>> b.text = "hello "
>>> c = ET.Element('i')
>>> c.text = "ignore this"
>>> c.tail = " world"
>>> b.append(c)
>>> ET.dump(b) # will match above
>>>
>>
>>
>> I'm sure xml.etree has its drawbacks which I will improve in my library.
>> The whole idea here is to bring python closer in the picture of
>> generating HTML content.
>> Its applications are vast and not only limited to this. If time permits I
>> intend to write a complete framework like Bootstrap for python where things
>> like cards, navbars etc will be inbuilt and purely written on the basis of
>> my library.
>> And that is the new feather I intend to add to Django's hat.
>> Just like we have some inbuilt  'forms' in Django we'll have a lot of
>> other front-end related objects written in python and easily manipulative.
>> And thus I intend to extend Django's reign in the front-end domain as
>> well. And make it a full-stack framework.
>> Surely designing is something that changes very frequently in today's
&g

Re: GSoC 2018

2018-03-16 Thread Tom Forbes
Hey Manasvi,
I don't have any say in the choice of a GSOC student, but I'd like to add
my two cents nonetheless.

I can see the logic behind your proposal, but I'm skeptical about the
usefulness of such a project. Libraries that implemented something similar
to this have come and gone, and got good reason, they where not very
practical and other solutions where much easier to use.

In the time since frontend frameworks have proliferated and JavaScript has
become much more mature. Anything we do in this space seems like it will be
dead on arrival compared to something like React or Ember, which already do
this but with huge huge advantages over your approach.

In my humble opinion anything in this space will have to treat JavaScript
as a first class citizen - i.e something that makes the interoperability
between a JavaScript frontend and a Django backed easier would have some
benefits, but generating HTML in pure python on the backend by writing
components in Python does not have a bright future IMO.

Tom

On 16 Mar 2018 22:36, "Manasvi Saxena"  wrote:

> Hello Sir,
>
> Okay, so the idea is to render HTML by generally defining it in Python. I
>> could've sworn that I'd seen something like this years ago, but I'm failing
>> to find it. That stated, I think this is basically a generating version of
>> BeautifulSoup (https://www.crummy.com/software/BeautifulSoup/bs4/doc/)
>> as opposed to a parsing version.
>>
>> It's roughly like the Storm ORM (https://storm.canonical.com/) but for
>> HTML instead of database queries.
>>
>> It's interesting, but I'll ask if one can get 90% of the functionality
>> from xml.etree, which is in the standard Python Library.
>>
>> import xml.etree.ElementTree as ET
>>
>> a = ET.Element('p', style='{color: red;}')
>> a.text = "hello world"
>> ET.dump(a) # will match yours
>>
>> Note that this gets even weirder with something like "hello ignore
>> this world".
>>
>> b = ET.Element('p')
>> b.text = "hello "
>> c = ET.Element('i')
>> c.text = "ignore this"
>> c.tail = " world"
>> b.append(c)
>> ET.dump(b) # will match above
>>
>
>
> I'm sure xml.etree has its drawbacks which I will improve in my library.
> The whole idea here is to bring python closer in the picture of generating
> HTML content.
> Its applications are vast and not only limited to this. If time permits I
> intend to write a complete framework like Bootstrap for python where things
> like cards, navbars etc will be inbuilt and purely written on the basis of
> my library.
> And that is the new feather I intend to add to Django's hat.
> Just like we have some inbuilt  'forms' in Django we'll have a lot of
> other front-end related objects written in python and easily manipulative.
> And thus I intend to extend Django's reign in the front-end domain as
> well. And make it a full-stack framework.
> Surely designing is something that changes very frequently in today's
> world but if this is successfully implemented we can bring developers from
> front-end world to contribute to it too.
> Logic + Design.
>
> Regards,
> Manasvi Saxena
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/2cf482cc-0f2e-4182-ae34-
> 951a74b43731%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOwW59_TYcg8w1JVBOFXoNP5SjC%2BxDZCg49oH%2Bh%2BUZAOQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: “Abusing BinaryField” warning about binary files in DB

2018-02-25 Thread Tom Forbes
Hey Antonie,

Personally I’m quite against changing that warning. I have only ever seen
one application where the use of an in-database file is appropriate and
they where using the FILESTREAM type in SQL Server

which offers some pretty advanced semantics compared to other databases
(more akin to Django’s file storage than a BLOB column).

I’ve seen a lot of beginners use BLOB/byte fields where it’s really not
needed and struggle with some insane performance issues due to it -
especially with Django fetching all columns in a model by default. Also the
link you gave (and thanks for linking, it’s an interesting read) is
obviously Postgres specific, the issues you might face doing this are very
vendor specific and non-portable - sqlite recommends against storing
anything larger than 100kb in a row
 for example.

I feel like the warning should implicitly say “do not do this, really
don’t, but if you’re super super super sure you 100% need to then you’re
going to disregard this warning anyway”, which the current one does quite
well. To put it another way, if you’re at the point where you need to do
this you’re way past reading the warning in the Django docs, and we should
deter people who might make the wrong choice at the start.

Tom



On 25 February 2018 at 23:21:39, Antoine Pietri (antoine.piet...@gmail.com)
wrote:

Hi!

In the documentation, the BinaryField has a warning called “Abusing
BinaryField” that states:

> Although you might think about storing files in the database, consider
that
> it is bad design in 99% of the cases. This field is not a replacement for
> proper static files handling.

https://docs.djangoproject.com/en/2.0/ref/models/fields/#
django.db.models.BinaryField

I agree with the intention of this warning: we don't want people to
start using their database for image uploads, large static files, or
thinking they can completely replace proper static file serving with a
databse.

That said, I think this warning is a huge overstatement. I think the
moment you're wondering "maybe this would be a good usecase to store
it in my database", your case for storing files in database might not
be absurd at all. There are tradeoffs, that are documented here, for
instance: https://wiki.postgresql.org/wiki/BinaryFilesInDB . It's
definitely not as clear-cut as "don't do it". People should be aware
of the tradeoffs instead of just dismissing the possibility.

Can I suggest replacing the warning by something like this?:

> Although you might think about storing files in the database, consider
that
> it might be a bad design choice. This field is not a replacement for
proper
> static files handling.
>
> That said, there might be cases where you do want the guarantees that the
> database offers you for binary files. Be sure to be aware of the
> trade-offs[1] before you decide to do so.
> [1]: https://wiki.postgresql.org/wiki/BinaryFilesInDB

As I'm not subscribed to this mailing-list, I would appreciate to be
CC'd to the responses :-)

Cheers,

-- 
Antoine Pietri
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/28cec919-ae57-4eed-960b-d598a01c2711%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJP967XiTqQBURfPN6W3KCOBpgiMEyCXw7aXbCSgWbNuhg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Consider renaming `mark_safe` to `dangerously_trust_html` (etc)

2018-02-22 Thread Tom Forbes
You're right, trusted_html is a better name than trust_html. I think either
is superior to mark_safe however.

Regarding deprecations, deprecating mark_safe will cause a lot of churn
even if the migration path is as simple as a rename. We should keep the old
alias around for a long time, but promote the new name wherever possible.

On 22 Feb 2018 20:46, "Josh Smeaton" <josh.smea...@gmail.com> wrote:

Yes I'm not a fan of the *dangerously...* names either. I'm still somewhat
wary of *trust_html* which is a *verb* and could still confuse users in a
similar way (does the api make it trustworthy?). I think I'd prefer
something more descriptive like *trusted_html*.

{{ content|trusted_html }}
vs
{{ content|trust_html }}


On Friday, 23 February 2018 01:07:12 UTC+11, Adam Johnson wrote:

> I am also in favour of a rename without deprecating the old name.
>
> I like 'trust_html' - it's still similarly short but as Tom says it
> implies more than 'mark_safe' does.
>
> On 22 February 2018 at 08:30, Tom Forbes <t...@tomforb.es> wrote:
>
>> What about just 'trust_html'? The dangerous part is quite context
>> dependent (and a bit of mouth-full), but at the core you are trusting the
>> HTML. Hopefully it follows that you should not trust html with user input
>> that hasn't been escaped.
>>
>>
>> On 22 Feb 2018 13:10, "Anthony King" <anthon...@gmail.com> wrote:
>>
>> I entirely agree with renaming `mark_safe`. Though it's name is correct,
>> it doesn't convey the gravity of what this actually does.
>> However I'm unsure on the `dangerously_trust_html` name. It wouldn't be
>> dangerous to trust the literal "Some Content", for example.
>>
>> Perhaps it could be something a bit more explicit. `no_escape(string)`?
>> This assumes that most have at least heard of escaping.
>>
>>
>> On 22 February 2018 at 12:16, Josh Smeaton <josh.s...@gmail.com> wrote:
>>
>>> The concern isn't overusing an API. It's not understanding the proper
>>> use case for it.
>>>
>>> "mark safe" can sound like the API is doing sanitation so it can
>>> encourage developers to use it incorrectly. I'm fairly sure I've done this
>>> myself.
>>>
>>> The intended meaning is "this output is **already** safe" but the name
>>> doesn't convey that meaning clearly enough.
>>>
>>> What the proposal is designed to do is convey the "I trust this output"
>>> meaning of the API. I'm just wary of enforcing users to change code when
>>> they already use the API correctly.
>>>
>>> On Thursday, 22 February 2018 21:08:31 UTC+11, Florian Apolloner wrote:
>>>>
>>>> Yeah, I am also worried about the churn for no gain in my eyes. If
>>>> users overuse mark_safe, they will overuse dangerously_trust_html too…
>>>>
>>>> On Wednesday, February 21, 2018 at 10:41:15 PM UTC+1, Josh Smeaton
>>>> wrote:
>>>>>
>>>>> I agree that the names are misleading and we should probably provide
>>>>> better names. I'm wary of deprecating the old names because it'll create a
>>>>> lot of churn (some of which would be the right thing to do). Maybe we 
>>>>> could
>>>>> just alias and warn when using the old name, leaving a decision on
>>>>> deprecation until some time in the future.
>>>>>
>>>>> On Monday, 29 January 2018 03:14:27 UTC+11, Stuart Cox wrote:
>>>>>>
>>>>>> In my experience, misuse of mark_safe() — i.e. marking stuff safe
>>>>>> which *isn’t* actually safe (e.g. HTML from a rich text input) — is
>>>>>> one of the biggest causes of XSS vulnerabilities in Django projects.
>>>>>>
>>>>>> The docs warn to be careful, but unfortunately I think Django devs
>>>>>> have just got too used to mark_safe() being *the way* to insert HTML
>>>>>> in a template. And it’s easy for something that was safe when it was
>>>>>> authored (e.g. calling mark_safe() on a hard-coded string) to be
>>>>>> copied / repurposed / adapted into a case which is no longer be safe 
>>>>>> (e.g.
>>>>>> that string replaced with a user-provided value).
>>>>>>
>>>>>> Some other frameworks use scary sounding names to help reinforce that
>>>>>> there are dangers around similar features, and that this isn’t something
>>>>>> you should use in everyday work — e.g. React’s dange

Re: Consider renaming `mark_safe` to `dangerously_trust_html` (etc)

2018-02-22 Thread Tom Forbes
What about just 'trust_html'? The dangerous part is quite context dependent
(and a bit of mouth-full), but at the core you are trusting the HTML.
Hopefully it follows that you should not trust html with user input that
hasn't been escaped.

On 22 Feb 2018 13:10, "Anthony King"  wrote:

I entirely agree with renaming `mark_safe`. Though it's name is correct, it
doesn't convey the gravity of what this actually does.
However I'm unsure on the `dangerously_trust_html` name. It wouldn't be
dangerous to trust the literal "Some Content", for example.

Perhaps it could be something a bit more explicit. `no_escape(string)`?
This assumes that most have at least heard of escaping.


On 22 February 2018 at 12:16, Josh Smeaton  wrote:

> The concern isn't overusing an API. It's not understanding the proper use
> case for it.
>
> "mark safe" can sound like the API is doing sanitation so it can encourage
> developers to use it incorrectly. I'm fairly sure I've done this myself.
>
> The intended meaning is "this output is **already** safe" but the name
> doesn't convey that meaning clearly enough.
>
> What the proposal is designed to do is convey the "I trust this output"
> meaning of the API. I'm just wary of enforcing users to change code when
> they already use the API correctly.
>
> On Thursday, 22 February 2018 21:08:31 UTC+11, Florian Apolloner wrote:
>>
>> Yeah, I am also worried about the churn for no gain in my eyes. If users
>> overuse mark_safe, they will overuse dangerously_trust_html too…
>>
>> On Wednesday, February 21, 2018 at 10:41:15 PM UTC+1, Josh Smeaton wrote:
>>>
>>> I agree that the names are misleading and we should probably provide
>>> better names. I'm wary of deprecating the old names because it'll create a
>>> lot of churn (some of which would be the right thing to do). Maybe we could
>>> just alias and warn when using the old name, leaving a decision on
>>> deprecation until some time in the future.
>>>
>>> On Monday, 29 January 2018 03:14:27 UTC+11, Stuart Cox wrote:

 In my experience, misuse of mark_safe() — i.e. marking stuff safe
 which *isn’t* actually safe (e.g. HTML from a rich text input) — is
 one of the biggest causes of XSS vulnerabilities in Django projects.

 The docs warn to be careful, but unfortunately I think Django devs have
 just got too used to mark_safe() being *the way* to insert HTML in a
 template. And it’s easy for something that was safe when it was authored
 (e.g. calling mark_safe() on a hard-coded string) to be copied /
 repurposed / adapted into a case which is no longer be safe (e.g. that
 string replaced with a user-provided value).

 Some other frameworks use scary sounding names to help reinforce that
 there are dangers around similar features, and that this isn’t something
 you should use in everyday work — e.g. React’s dangerouslySetInnerHTML.

 Relatedly, this topic
 
  suggested
 making it more explicit that mark_safe() refers to being safe for use
 in *HTML* contexts (rather than JS, CSS, SQL, etc).

 Combining the two, it would be great if Django could rename mark_safe() to
 dangerously_trust_html(), |safe to |dangerously_trust_html,
 @csrf_exempt to @dangerously_csrf_exempt, etc.

 Developers who know what they’re doing with these could then be
 encouraged to create suitable wrappers which handle their use case safely
 internally — e.g.:

 @register.filter
 def sanitize_and_trust_html(value):
 # Safe because we sanitize before trusting
 return dangerously_trust_html(bleach.clean(value))


 --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/ms
> gid/django-developers/db4ac958-89e1-4286-a616-99e9854c9bbb%
> 40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To 

Re: #28398: Allow management command invocation to suggest for incomplete commands?

2018-02-16 Thread Tom Forbes
Did we reach a consensus on this? Would it change things if this could be
implemented with a very small changeset?

People brought up that it seems similar to the bash completion - perhaps we
can share code between the two features? I think it's a great idea from a
usability standpoint, and seeing another junior engineer start to use
manage.py I think it could make a difference to beginners.

I've made a simple PR that uses the stdlib 'difflib.get_close_matches'
which seems to be pretty great at suggesting the right command and it's
also less code to maintain: https://github.com/django/django/pull/9703

With this the output 'manage.py rnserver' would be:

Unknown command 'rnserver'. Did you mean runserver?
Type 'manage.py help' for usage.

On 17 Jul 2017 10:32, "Adam Johnson" <m...@adamj.eu> wrote:

> I feel like this mostly duplicates the bash completion logic we have,
> which is also more standard across other CLI's
>
> However I agree with Tom that correcting for typos is the main use case,
> using levehshtein distance is a good idea.
>
> Another thing that some CLI's have, like npm or arc is auto-correction
> for common typos, e.g. ./manage.py mgriate could print out a message
> 'Assuming you meant migrate' and then continue executing migrate, not
> sure if this is the best for Django though where commands can be
> arbitrarily added by third party apps.
>
>
>
> On 17 July 2017 at 09:47, Tom Forbes <t...@tomforb.es> wrote:
>
>> Vlada:
>> I think this is a great idea for improving the usability of manage.py,
>> especially for newcomers. When I looked your current implementation used a
>> simple 'in' to find suggestions, but this is not great for the most
>> obvious/common use case: typos.
>>
>> I would strongly advocate for using the levenshtein distance algorithm if
>> this does get merged, there are some simple and succinct python
>> implementations we can use, and the algorithm itself is what git uses to
>> suggest commands (and I've always found that quite good).
>>
>> Brice:
>> You are right in that this is a case that could happen in part due to
>> this feature, but it's a long shot and IMO the benefits outweigh the risks.
>> This can happen without this feature anyway - if someone is typing in
>> random manage.py commands without thinking then issues can and will arise.
>> The only preventative measure we could take would be to add an "are you
>> sure?" prompt after every manage.py invocation which is not a great idea
>> overall.
>>
>> For newcomers though this could be great, seeing the 'command not found'
>> message can be confusing and unhelpful, any help we can add to that is a
>> good thing IMO. Adding stars to the command invocation can be even more
>> confusing as users have to escape them (and if they don't they could end up
>> running random commands) and it's not terribly nice UI wise.
>>
>> Built in shell autocompletion would be the best way forward in this case,
>> at least for non-windows users. Kubectl has a nice way of doing this:
>> source <(kubectl completion bash|zsh). Maybe something like this could be
>> adapted for manage.py?
>>
>>
>> On 17 Jul 2017 08:49, "Brice PARENT" <cont...@brice.xyz> wrote:
>>
>> Hi!
>>
>> I'm not sure how I feel about that. It feels like a good idea at first,
>> but it might lead to dangerous behaviours.
>>
>> Let me explain my thought: having such a feature would encourage people
>> to use it (of course). Doing so can lead so side effects. For example,  if,
>> in a project you're working on, you want to use a custom management command
>> named "migrate_data_to_other_server", you might end up typing
>> "./manage.oy migrate" in hope for the system to display the exact name that
>> you probably have forgotten. But it won't, it will migrate your database
>> instead. What I mean is that executing commands that shouldn't work on
>> purpose might lead to executing the wrong command instead. And management
>> commands might be dangerous if not used at the right time (I've seen
>> management commands being used to push code to production for example.
>> Executing them by error in a dev environment might be a real issue!).
>>
>> I'd prefer to encourage the use of "./manage.py help", which lists all
>> available commands, or use masks when searching for a command ("./manage.py
>> migrate*"). When you want to look for the name of a command, you'd know
>> that adding a star somewhere won't execute anything other than listing
>> available commands matching the value you just gave. And every d

Re: DEP Pre-posal: Re-Designing Django Forms

2018-02-05 Thread Tom Forbes
> Perhaps we should just be able to swap Forms with WTForms or another
python library and bake in ElementUI,

There are a plethora of UI frameworks with different tradeoffs, I really
don't think Django sound pick one.

However a stronger integration with the JS-build tools of the day like
Yarn, webpack et al would be a good step in the right direction, and more
agnostic than just 'use this UI framework we picked for you'.

> perhaps it's time for npm to become a first class citizen.

I think if Django is going to stay relevant to building most/all kinds of
web apps this is something that will need to happen. To be fair, npm and
the whole frontend ecosystem has only really 'matured' in the last couple
of years, it's come a long way since Django started.

On 4 Feb 2018 00:36, "Jamesie Pic"  wrote:

On Thu, Feb 1, 2018 at 12:46 PM, Marc Tamlyn  wrote:
> This is a huge project to achieve everything you mentioned in your email,
and it has implications across a large number of Django packages (not least
the admin). I don't want to discourage you, but don't underestimate how
much work it would be to get a good replacement for forms for the modern
web.

Perhaps we should just be able to swap Forms with WTForms or another python
library and bake in ElementUI, even if that means replacing template_name
with vue_name in the view generic class, but if we're talking about "modern
web" then perhaps it's time for npm to become a first class citizen.

> Your next steps should be to research, spec and potentially write a DEP.

In my recent research it seemed ElementUI the most feature complete UI. It
includes ajax file upload which every user expects in the modern web which
seems to be the feature which defines feature-completion of a UI framework,
compared to what HTML offers out of the box.

Thanks a lot for doing something about this Robert, forms in django
definitely needs a major refactoring sprint, typically remove the
field/widget layer and rely on one level inheritance that will help a lot
for example with material design which displays field.name inside the
widget, not possible with current object design.

-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/CAC6Op1_ESqUA6tUwQxwgastH4XzQ%
3D-PBybtq__2yWEuc0OH4BA%40mail.gmail.com

.

For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOJE4me_7nw0K3_hpXWN_vzsCAip_OvgmuLHABmuFgDvw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Integrate dj-database-url into Django

2018-02-04 Thread Tom Forbes
I spent some time working on this today and fixed up my branch that
implements this. I think I’ve got a nice API
 that passes all of the
dj-database-url tests.

As there is a bit of database-specific shenanigans we have to perform on
URLs I added a config_from_url static method to database adapters that
takes the URL and returns a valid django configuration dictionary. As it’s
a method inside the adapter itself we can override it to apply any
db-specific fixups

and third party backends can hook into the process as well via the same
mechanism.

It expands on dj-database-url by also allowing the configuration of caches
using the same method, which I think is pretty neat.

It uses a registration process described below, users just call
register_db_backend or register_cache_backend with a path to their third
party engine in the settings to link a scheme to a backend dotted path.

I had one question that I’m not sure the answer to: for databases I added
the config_from_url to the .base.DatabaseWrapper class inside each built in
adapter. I’ve kind of assumed each third party database would have the same
structure, how much of a safe assumption is this?



On 8 June 2017 at 18:40:07, Kenneth Reitz (kenn...@heroku.com) wrote:

My thoughts are the same as Shai's below.

On Monday, May 29, 2017 at 4:00:09 PM UTC-4, Shai Berger wrote:
>
> If I understand things correctly, registration of new url-schemas to enable
> something like dj-database-url to support 3rd parties should be a trivial
> matter; just add an entry to some dictionary, with the schema as key, and
> the
> dotted-path of the backend as value.
>
> IMO, scanning INSTALLED_APPS, or relying on ready(), is a lot of over-
> engineering for this. Just do the registration from the settings file,
> before
> invoking the url-parsing mechanism:
>
> from django.database_url import register_backend, configure_db
>
> register_backend('pyodbc-azure', 'sql_server.pyodbc')
>
> DATABASES = {'default' : configure_db() }
>
> The only barrier I see is namespacing -- the functions mentioned above
> cannot
> live in django.conf (would cause a circular import if imported from the
> settings module) nor django.db (which shouldn't be imported too early), and
> that's why I wrote `django.database_url` above.
>
> What am I missing?
>
--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/065601c3-9488-42aa-8b23-c7f906c26b5a%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOpvhXUQCoLpt0Om%2BCgg1VgnWGvAHLQFPnqp%3DC1p%3DxVeQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Adding a bulk_save method to models

2018-01-22 Thread Tom Forbes
Hey Neal,

Thank you very much for pointing that out, I actually found out about this
package as I was researching the ticket - I wish I had known about this a
couple of years ago as it would have saved me a fair bit of CPU and brain
time!

I think that module is a good starting point and proves that it’s possible,
however I think the implementation can be improved upon if we bring it
inside core. I worked on a small PR to add this
<https://github.com/django/django/pull/9606/files#diff-5b0dda5eb9a242c15879dc9cd2121379R473>
and the implementation was refreshingly simple. It still needs docs, a
couple more tests and to fix a strange error with sqlite on Windows, but
overall it seems like a lot of gain for a small amount of code.

Tom


On 22 January 2018 at 15:10:53, Neal Todd (n...@torchbox.com) wrote:

Hi Tom,

A built-in bulk save that's more flexible than update would certainly be
nice. Just in case you haven't come across it though, there is a package
called django-bulk-update:

https://github.com/aykut/django-bulk-update

I've found it very useful on a number of occassions where update isn't
quite enough but the loop-edit-save pattern is too slow to be convenient.

Probably some useful things in there when considering the API and approach.

Cheers, Neal

On Friday, January 19, 2018 at 5:49:48 PM UTC, Tom Forbes wrote:
>
> Hello all,
>
> I’d love for some feedback on an idea I’ve been mulling around lately,
> namely adding a bulk_save method to Dango.
>
> A somewhat common pattern for some applications is to loop over a list of
> models, set an attribute and call save on them. This unfortunately can
> issue a lot of database queries which can be a significant slowdown. You
> can work around this by using ‘.update()’ in some cases, but not all.
>
> It seems it would be possible to use a CASE statement in SQL to handle
> bulk-updating many rows with differing values. For example:
>
> SomeModel.object.filter(id__in=[1,2]).update(
> some_field=Case(
> When(id=1, then=Value('Field value for ID=1')),
> When(id=2, then=Value('Field value for ID=2'))
> )
> )
>
> I’ve made a ticket for this here: https://code.djangoproject.
> com/ticket/29037
>
> I managed to get a 70x performance increase using this technique on a
> fairly large table, and it seems it could be applicable to many projects
> just like bulk_create.
>
> The downsides to this is that it can produce very large SQL statements
> when updating many rows (I had MySQL complain about a 10MB statement once),
> but this can be overcome with batching and other optimisations (i.e the
> same values can use WHEN id IN (x, y, z) rather than 3 individual WHEN
> statements).
>
> I’m imagining an API very similar to bulk_create, but spend any time on a
> patch I thought I would ask if anyone have any feedback on this suggestion.
> Would this be a good addition to Dango?
>
>
> --
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/5988d579-7843-4c42-a6f9-1e389c58ece6%40googlegroups.com
<https://groups.google.com/d/msgid/django-developers/5988d579-7843-4c42-a6f9-1e389c58ece6%40googlegroups.com?utm_medium=email_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJO_yxQ-Kj4V6Ps2NtL7wfNMSkzeAu_H1c26NB8%3DgJzqug%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Adding a bulk_save method to models

2018-01-19 Thread Tom Forbes
Hello all,

I’d love for some feedback on an idea I’ve been mulling around lately,
namely adding a bulk_save method to Dango.

A somewhat common pattern for some applications is to loop over a list of
models, set an attribute and call save on them. This unfortunately can
issue a lot of database queries which can be a significant slowdown. You
can work around this by using ‘.update()’ in some cases, but not all.

It seems it would be possible to use a CASE statement in SQL to handle
bulk-updating many rows with differing values. For example:

SomeModel.object.filter(id__in=[1,2]).update(
some_field=Case(
When(id=1, then=Value('Field value for ID=1')),
When(id=2, then=Value('Field value for ID=2'))
)
)

I’ve made a ticket for this here:
https://code.djangoproject.com/ticket/29037

I managed to get a 70x performance increase using this technique on a
fairly large table, and it seems it could be applicable to many projects
just like bulk_create.

The downsides to this is that it can produce very large SQL statements when
updating many rows (I had MySQL complain about a 10MB statement once), but
this can be overcome with batching and other optimisations (i.e the same
values can use WHEN id IN (x, y, z) rather than 3 individual WHEN
statements).

I’m imagining an API very similar to bulk_create, but spend any time on a
patch I thought I would ask if anyone have any feedback on this suggestion.
Would this be a good addition to Dango?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOmmRDzZv9jMDxnp3-Wp%3Dg5F1dR_Gga3d51kARKGbrrzQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: ease patching Django object behavior in another django packages

2018-01-10 Thread Tom Forbes
Django has managed this before by examining the function signature. Adding
a property directly onto the instance could cause some issues, for example
if there is already a user property. I guess this could be detected and a
deprecation warning issued though.

Anyway, this is all academic unless there is consensus.

On 10 Jan 2018 19:00, "Sergey Glazyrin" <sergey.glazyrin@gmail.com>
wrote:

We use function load_backend in django about 5 times in production code,
so, it shouldn't be a big change
About signals idea: yes, I can implement it using signals abstraction
though I prefer to be tied to the "Builder" idea, there would no big
difference between signals and Builder implementation in this case because
load_backend always returns a new instance of the class, so, I expect no
threading problems, etc, it's just a way to distribute process of building
objects
About adding request to get_user, I don't like it because then all
dependent of django projects will need to change backends, it would be
worst for community

середа, 10 січня 2018 р. 19:52:06 UTC+1 користувач Tom Forbes написав:
>
> I would be in favour of a mechanism to help with this use case, I ran into
> a somewhat similar issue when using JWTs and a non-model backed user.
>
> Adding a user parameter seems like the easiest solution and quite simple,
> whereas adding a builder class into this particular section of Django seems
> like it would be harder to get consensus.
>
> You could maybe get more traction if you suggested firing a signal that is
> passed the auth instance as a parameter when it is initialized, which is
> akin to your suggestion, however IMO that's still not a great idea.
>
> On 10 Jan 2018 18:18, "Tim Graham" <timog...@gmail.com> wrote:
>
>> Without studying the openstack code much it's hard for me to say if the
>> solution there is the best approach and that a more elegant solution
>> doesn't exist. It looks like if we added 'request' to the signature of the
>> authentication backend get_user() method, that would remove the need for
>> monkey patching. We did a similar change for the authenticate() method [1],
>> I'm not sure if there would be consensus to make the change.
>>
>> [1] https://code.djangoproject.com/ticket/25187
>>
>> On Wednesday, January 10, 2018 at 1:12:55 PM UTC-5, Sergey Glazyrin wrote:
>>>
>>> Btw, I see no way how do I use this auth_user.create_user_from_token to
>>> solve this problem.
>>> It uses django contrib auth get_user function, so the proper place is to
>>> to use django auth backend logic.
>>>
>>> середа, 10 січня 2018 р. 14:17:50 UTC+1 користувач Tom Forbes написав:
>>>>
>>>> I think Tim’s assessment in the ticket is on point, a
>>>> DjangoObjectBuilder would look very strange and out of place if included
>>>> (it’s not particularly pythonic either).
>>>>
>>>> Seems like there might be a legitimate issue here (or maybe just bad
>>>> designs in OpenStack?), but unless I’m misunderstanding something couldn’t
>>>> you call ‘auth_user.create_user_from_token’ yourself and set it on the
>>>> request object rather than monkeypatch Django?
>>>>
>>>> On 10 January 2018 at 12:22:01, Sergey Glazyrin (sergey.gl...@gmail.com)
>>>> wrote:
>>>>
>>>> Hello guys!
>>>> I faced a situation when auth backend needs access to request object
>>>> inside of get_user auth backend function
>>>>
>>>> https://code.djangoproject.com/ticket/29005
>>>>
>>>> I can patch it following way (function to be patched is
>>>> django.contrib.auth.get_user)
>>>>
>>>>
>>>> def get_user(request):
>>>>>
>>>>
>>>>
>>>> ..
>>>>> code
>>>>> .
>>>>>  backend = load_backend(backend_path)
>>>>>  backend,request = request
>>>>> .
>>>>> code
>>>>> .
>>>>
>>>>
>>>>
>>>> But I don't like this solution because I'll need to keep my eyes on
>>>> this monkey patch while django upgrade, etc, and it's very dirty hack.
>>>>
>>>> Instead I propose to extend django behaviour using design pattern
>>>> Builder to simplify integration of another apps into django object
>>>> internals (it sounds hacky, but it's safe and simple to implement)
>>>>
>>>> with change I proposed, the patch would be done on django level, we
>>>> need to add
>>

Re: ease patching Django object behavior in another django packages

2018-01-10 Thread Tom Forbes
I would be in favour of a mechanism to help with this use case, I ran into
a somewhat similar issue when using JWTs and a non-model backed user.

Adding a user parameter seems like the easiest solution and quite simple,
whereas adding a builder class into this particular section of Django seems
like it would be harder to get consensus.

You could maybe get more traction if you suggested firing a signal that is
passed the auth instance as a parameter when it is initialized, which is
akin to your suggestion, however IMO that's still not a great idea.

On 10 Jan 2018 18:18, "Tim Graham" <timogra...@gmail.com> wrote:

> Without studying the openstack code much it's hard for me to say if the
> solution there is the best approach and that a more elegant solution
> doesn't exist. It looks like if we added 'request' to the signature of the
> authentication backend get_user() method, that would remove the need for
> monkey patching. We did a similar change for the authenticate() method [1],
> I'm not sure if there would be consensus to make the change.
>
> [1] https://code.djangoproject.com/ticket/25187
>
> On Wednesday, January 10, 2018 at 1:12:55 PM UTC-5, Sergey Glazyrin wrote:
>>
>> Btw, I see no way how do I use this auth_user.create_user_from_token to
>> solve this problem.
>> It uses django contrib auth get_user function, so the proper place is to
>> to use django auth backend logic.
>>
>> середа, 10 січня 2018 р. 14:17:50 UTC+1 користувач Tom Forbes написав:
>>>
>>> I think Tim’s assessment in the ticket is on point, a
>>> DjangoObjectBuilder would look very strange and out of place if included
>>> (it’s not particularly pythonic either).
>>>
>>> Seems like there might be a legitimate issue here (or maybe just bad
>>> designs in OpenStack?), but unless I’m misunderstanding something couldn’t
>>> you call ‘auth_user.create_user_from_token’ yourself and set it on the
>>> request object rather than monkeypatch Django?
>>>
>>> On 10 January 2018 at 12:22:01, Sergey Glazyrin (sergey.gl...@gmail.com)
>>> wrote:
>>>
>>> Hello guys!
>>> I faced a situation when auth backend needs access to request object
>>> inside of get_user auth backend function
>>>
>>> https://code.djangoproject.com/ticket/29005
>>>
>>> I can patch it following way (function to be patched is
>>> django.contrib.auth.get_user)
>>>
>>>
>>> def get_user(request):
>>>>
>>>
>>>
>>> ..
>>>> code
>>>> .
>>>>  backend = load_backend(backend_path)
>>>>  backend,request = request
>>>> .
>>>> code
>>>> .
>>>
>>>
>>>
>>> But I don't like this solution because I'll need to keep my eyes on this
>>> monkey patch while django upgrade, etc, and it's very dirty hack.
>>>
>>> Instead I propose to extend django behaviour using design pattern
>>> Builder to simplify integration of another apps into django object
>>> internals (it sounds hacky, but it's safe and simple to implement)
>>>
>>> with change I proposed, the patch would be done on django level, we need
>>> to add
>>>
>>> def get_user(request):
>>>>
>>>> ...
>>>> code
>>>> ...
>>>>
>>>>  backend = load_backend(backend_path)
>>>> DjangoObjectBuilder.do_initialize_object(backend, request)
>>>> 
>>>> code
>>>> 
>>>
>>>
>>> and in another django package we subscribe to this object initialization:
>>>
>>>
>>> def add_request_to_backend(obj, request):
>>>> obj.request = request
>>>> DjangoObjectBuilder.add_custom_initializer(lambda obj: isinstance(obj,
>>>> openstack_auth.Backend), add_request_to_backend)
>>>
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/django-developers/5dffbda9-7239-489e-9530-564df9ab578e%
>>> 40

Re: ease patching Django object behavior in another django packages

2018-01-10 Thread Tom Forbes
I think Tim’s assessment in the ticket is on point, a DjangoObjectBuilder
would look very strange and out of place if included (it’s not particularly
pythonic either).

Seems like there might be a legitimate issue here (or maybe just bad
designs in OpenStack?), but unless I’m misunderstanding something couldn’t
you call ‘auth_user.create_user_from_token’ yourself and set it on the
request object rather than monkeypatch Django?

On 10 January 2018 at 12:22:01, Sergey Glazyrin (
sergey.glazyrin@gmail.com) wrote:

Hello guys!
I faced a situation when auth backend needs access to request object inside
of get_user auth backend function

https://code.djangoproject.com/ticket/29005

I can patch it following way (function to be patched is
django.contrib.auth.get_user)


def get_user(request):
>


..
> code
> .
>  backend = load_backend(backend_path)
>  backend,request = request
> .
> code
> .



But I don't like this solution because I'll need to keep my eyes on this
monkey patch while django upgrade, etc, and it's very dirty hack.

Instead I propose to extend django behaviour using design pattern Builder
to simplify integration of another apps into django object internals (it
sounds hacky, but it's safe and simple to implement)

with change I proposed, the patch would be done on django level, we need to
add

def get_user(request):
>
> ...
> code
> ...
>
>  backend = load_backend(backend_path)
> DjangoObjectBuilder.do_initialize_object(backend, request)
> 
> code
> 


and in another django package we subscribe to this object initialization:


def add_request_to_backend(obj, request):
> obj.request = request
> DjangoObjectBuilder.add_custom_initializer(lambda obj: isinstance(obj,
> openstack_auth.Backend), add_request_to_backend)




--
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit
https://groups.google.com/d/msgid/django-developers/5dffbda9-7239-489e-9530-564df9ab578e%40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJM6JqGC6CgxfdJDpMp2GutC8h%3DuCoWXiAb6xZ0f9om04g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Refactoring the autoreloader

2017-12-10 Thread Tom Forbes
Thank you all for your feedback! I have fixed up my PR (
https://github.com/django/django/pull/8819) to add support for
watchman-based reloading, and it seems to be working well locally. After
further investigation I don't think using the Watchdog library is suitable
for Django, it seems to be un-maintained right now (which is a real shame).
Watchman seems to be faster even with constant reloading though, so I think
it's fine to just add support for that.

If anyone is interested they could take the changes to
`django/core/management/commands/runserver.py` and
`django/utils/autoreload.py`, and apply them to their installations. You
just need to install the `pywatchman` package + watchman itself, then run
manage.py with '--watchman'. I'd love some real-world feedback as to if
this improves reload speed and reduces CPU usage. Unfortunately Watchman
won't work on NFS volumes or virtualbox folders, so if you're using Vagrant
there isn't much that will help you here, which is also why we cannot make
this a default option - watchman will appear to work, but not pick up any
changes, if its run on such a volume.

I'll start work on adding documentation and tests next, but I'm quite lost
as to how I should test the Watchman integration. Should I be testing using
a real Watchman service, or should I mock it and hope that's enough?


On Thu, Oct 12, 2017 at 12:42 AM, Josh Smeaton <josh.smea...@gmail.com>
wrote:

> I'd just like to second (or third) an appetite for this or a similar
> change. Lately I've started to notice very poor performance (high cpu
> usage, slow shutdowns, etc) with the django dev server on my project. I've
> moved to runserver_plus/werkzeug with watchdog in the mean time but it'd be
> good to have similar improvements via django natively.
>
> On Sunday, 8 October 2017 05:24:31 UTC+11, Aymeric Augustin wrote:
>>
>> Hello Tom,
>>
>> I think the PR is on the right track. Now it needs to be taken beyond the
>> "proof of concept" stage.
>>
>> Once you have code, docs, and if possibly some tests — currently there
>> are few due to the difficulty of testing auto reloading, especially edge
>> cases — for steps 1 and 2, I can review changes again and help you get it
>> merged.
>>
>> Since this is a major overhaul, we'll have to think carefully about how
>> the behavior changes and possible backwards incompatibilities. That said,
>> the autoreloader is a dev tool. Changing it isn't going to break anyone's
>> production. This makes the big-bang approach viable.
>>
>> Thanks for working on this!
>>
>> --
>> Aymeric.
>>
>>
>>
>> On 29 Sep 2017, at 21:03, Tom Forbes <t...@tomforb.es> wrote:
>>
>> Hello,
>> I've been thinking on and off about how to improve the autoreloader
>> implementation and I wanted to gather some feedback on potential solutions.
>>
>> For some background, Django uses a fairly basic autoreload implementation
>> that simply polls the last modified time for loaded Python files once a
>> second. While this isn't the most efficient, it does work and has worked
>> quite well for a long time. When running manage.py runserver, the
>> autoreloader will launch a child "manage.py" with the same arguments and
>> this child process actually runs Django and serves requests. To reload, the
>> child process exits with exit code 3 and the parent restarts it. The code
>> is some of the oldest in Django, with a fair bit of it not touched in 9-12
>> years.
>>
>> While it works (and I'm a believer in "if it isn't broke don't fix it")
>> there are some architectural and performance issues with the current code:
>>
>> - Polling every second is not very efficient
>> - Detecting when the child process has exited during startup (i.e problem
>> in settings.py) is problematic and the code is rather nasty
>> - i18n files are 'reloaded' when they change in a rather hacky way
>> (resetting private attributes in another module)
>> - There is limited support for extending the current implementation, and
>> there are cases during development where the parent autoreloader will
>> terminate.
>>
>> I don't want this email to be too long, so I'm going to summarize what I
>> think would be a good approach to tackling these problems.
>>
>> 1. Refactor the current implementation by removing `pyinotify`, redundant
>> python 2 checks and implement a 'file_changed' signal so other parts of
>> Django can react to file changes (i.e the i18n module can handle resetting
>> it's own state).
>> 2. Add support for the "watchdog" library as a replacement for pyinotify.
>> Watchdog implements file system notif

Re: models.CalculatedField feature

2017-11-16 Thread Tom Forbes
I think without client-side Python execution of these functions it greatly
limits the use of this feature. In the cases I've seen you have a bunch of
`qs.filter(age__gt=18)` scattered around everywhere and a separate
`can_drink` function of property. Without the `can_drink` function
returning the correct result when the model is updated it's kind of
useless, you'd still end up with people still writing their own `can_drink`
functions to do so.

If we can make simple cases work well (i.e functions that do arithmetic,
and/or, simple comparisons and on simple fields) I think it would be pretty
useful. Sure, when you get into quite complex database functions, complex
fields with differing support it can get quite hard, but for
string/integer/boolean columns with simple operators that different between
databases we support?

On Thu, Nov 16, 2017 at 12:08 PM, Sjoerd Job Postmus 
wrote:

> I disagree with not being able to calculate it locally (without the
> database): such a calculation depends on other fields. What if a dependent
> field is updated, but the object is not yet saved. What should the value be?
>
> >>> c.age, c.is_adult
> 17, False
> >>> c.age = 40
> >>> c.age, c.is_adult
> 40, False
>
> Would not make sense to me.
>
> As a first step, maybe it would make sense to make it easier to support
> custom filter attribute. Right now you have to create a custom query set
> subclass implementing .filter where you intercept the value
>
> def filter(self, *args, **kwargs):
> if 'is_adult' in kwargs:
> value = kwargs.pop('is_adult')
> if value:
> kwargs['age__gte'] = 18
> else:
> kwargs['age__lt'] = 18
> return super().filter(*args, **kwargs)
>
> And that's ignoring exclude and Q, and probably a lot of other cases.
>
> Could we make that simpler first?
>
> On 16 Nov 2017 07:43, Shai Berger  wrote:
>
> On Thursday 16 November 2017 07:21:38 Josh Smeaton wrote:
> >
> > I don't agree with reimplementing lookups/transforms/funcs to produce
> the
> > python version of a SQL concept. It's a leaky abstraction. Instead, I'd
> > prefer the model author provides their python calculation:
> >
> > is_adult = models.CalculatedField(Q(age__gte=18), lambda c: c.age
> >= 18)
>
> (name shortened to fit in a 80-char line)
>
> I disagree. The way I see it, these calculated fields should essentially
> provide a declarative way to add an annotate() that's always going to be
> there. There should be only one source of truth, and that's the database;
> parallel implementations tend to disagree on fringe cases with
> disappointing
> results (e.g. consider if age above is nullable -- the database then makes
> the
> calculated field null, the python above makes it raise an exception). Or
> consider calculated fields which are aggregates -- a Python version could
> very
> easily become painfully inefficient.
>
> >
> > There may be destructuring/serialisation issues (migrations/pickle)
> using
> > lambda, unless we can ignore these model fields during deserialisation.
> >
>
> That's another (though minor) argument against lambdas here.
>
> No, the way I see it, the API should just use expressions. Of note, I
> think it
> should also support the output_field keyword. At the hand-waving,
> brainstorming
> level, I think we might, at some point, actually try to implement this by
> creating objects in the database (views on Postgres, computed columns on
> MySql, etc) -- so that the field is really available in all contexts.
>
> My 2 cents,
>
> Shai.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/fdf87ea3-aa91-452a-8336-
> d73dc92efe8d%40email.android.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMQkAbonjFi%2BSjpDzEq_azpdKXJEkM3nCG0h_an%2BE0XBQ%40mail.gmail.com.
For more options, visit 

RE: models.CalculatedField feature

2017-11-15 Thread Tom Forbes
SQLAlchemy has this, through a feature called Hybrid Attributes:
http://docs.sqlalchemy.org/en/latest/orm/extensions/hybrid.html

Every Django project i've worked on has had to tackle this problem, often
poorly. You usually end up with an instance method and a kind of duplicate
manager method, or some specific queries spread throughout the code.

The annotate feature is quite cool, but I think Django would benefit from
something like this.


On 15 Nov 2017 16:49, "Matthew Pava"  wrote:

I handle that situation quite differently.

I have a model manager that annotates different fields to the queryset
depending on when I need them.

For your example, I would have something like this:



class CustomerQuerySet(models.QuerySet):
   def with_allowed_to_drink(self):
  return self.annotate(allowed_to_drink=Case(When(age__gte=18,
then=True), default=False))





Then I actually created a decorator that applies the manager to instances
on a model.  It is not as generic as it could be since it depends on
function names having the word “with” in them, but I’m sure we could make
it more general.

def queryset_on_instance(model: Model):







*"""Takes the queryset of a model and creates new properties for each
instance of the model based on the custom queryset class.  Some code is
copied from django.models.managers; most of this was inspired by it.
The queryset methods that are copied must have 'with' in their name.
Tip: Use this as a decorator on model classes """ *def create_method
(method_name):
def the_method(self):
return getattr(model._default_manager, method_name)().get(pk=
self.pk)

return the_method

queryset_class = getattr(model._default_manager, "_queryset_class", None
)
predicate = inspect.isfunction
for name, method in inspect.getmembers(queryset_class, predicate
=predicate):
if 'with' in name:
# Only copy missing attributes.
if hasattr(model, name):
continue
# Only copy public methods or methods with the attribute
`queryset_only=False`.
queryset_only = getattr(method, 'queryset_only', None)
if queryset_only or (queryset_only is None and name.startswith(
'_')):
continue
# Copy the method onto the model.
setattr(model, name, create_method(name))
return model





And then on your Customer model:

@queryset_on_instance
class Customer(models.Model):
   age = models.IntegerField()
   objects = CustomerQuerySet.as_manager()





In your code or template, you would access the value like so:

customer.with_allowed_to_drink.allowed_to_drink



And you can filter by the value like so:

Customers.objects.with_allowed_to_drink().filter(allowed_to_drink=True)



I do agree that it would be nice to have something out of the box that
could handle custom properties in the QuerySet calls, or, as you put it,
calculated fields.





*From:* django-developers@googlegroups.com [mailto:django-developers@
googlegroups.com] *On Behalf Of *ilya.kazakev...@jetbrains.com
*Sent:* Wednesday, November 15, 2017 7:21 AM
*To:* Django developers (Contributions to Django itself)
*Subject:* models.CalculatedField feature



Hello.



I think I just invented very useful feature.

I have the following model



class Customer(models.Model):

age = models.IntegerField()



I want to show special link in template for customer if she is allowed to
drink.



class Customer(models.Model):

age = models.IntegerField()



@property

def allowed_to_drink(self):

return self.age >= 18



And in template I can do



{% if customer.allowed_to_drink %}.



But in another view I want to show only those customers, who are allowed to
drink.

So, in view I write



drinkers = [customer for customer in models.Customer.objects.all() if
customer.allowed_to_drink].



But what if I have one million customers? It is insane to check this field
on Python side instead of database side.

I use ORM:

models.Customer.objects.filter(age__gte=18).all()



And DRY principle is now violated.

See, I have knowledge "allowed_to_drink -> (age >= 18) " and I want to
write in once, and then use both in ORM queries and object method.



And I am not alone here:

https://stackoverflow.com/questions/31658793/django-
filter-query-on-with-property-fields-automatically-calculated

https://stackoverflow.com/questions/2143438/is-it-possible-to-reference-a-
property-using-djangos-queryset-values-list



Here is what I suggest



class Customer(models.Model):

age = models.IntegerField()

allowed_to_drink = models.CalculatedField(Q(age__gte=18))



# Using in ORM

Customer.objects.filter(allowed_to_drink=True)

# It actually converted to

Customer.objects.filter(Q(age__gte=18))



# Using it in code

Customer().allowed_to_drink

# This part a little bit tricky, since you will need to check in on Python
side.

# "age__gte" should be converted to

funcs 

Re: Database agnostic EnumField

2017-10-20 Thread Tom Forbes
There is a maybe/someday ticket for this here: https://code.djangoproject.
com/ticket/24342

If your interested you could work on adding this into Django through that
ticket, it seems like an interesting idea.

When removing an enum value, how do you handle existing records? Are all
supported databases consistent in throwing errors if existing rows are
still present with a removed enumeration value?

On Fri, Oct 20, 2017 at 12:55 PM, Ashley Waite 
wrote:

> I've been working a bit on an EnumField implementation because it'll save
> me a lot of future time in a project, and existing implementations seem to
> be fragile, non-reversible, or one-database-only. Wondering why there isn't
> a PEP435 based EnumField in Django itself, I didn't find many answers with
> a search on the mailing list.
>
> Is this a feature that would be considered, and if so, what would the
> expectations on it be?
>
> I was a bit reluctant on all the implementations I could find because they
> seem to reduce to these issues:
> * Using an int/char instead of database enum
> * Being database vendor specific
> * Requiring a non-standard enum or sub-class of it
> * Not allowing enum reuse
> * Not migrating changes statefully (ie, injecting type declaration on
> connection in postgres)
> * Not allowing enum changes (add/remove/rename)
> * Not parametising enum values (mysql)
> * Not handling data consistency on enum changes
>
> And realised, maybe that's why it's not a standard field.
>
> I've done a POC implementation that works for mysql and postgres, and
> should be able to easily generalise to work on any database via two flags
> (has_enum, and requires_enum_declaration) that determine how to deal with
> changes to it.
>
> It addresses all of these issues, migrates, and with a little more work
> can handle data effects as well.
>
> So where should I go with this from here?
> https://github.com/ashleywaite/django-more/tree/master/django_enum
>
> - Ashley
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/ms
> gid/django-developers/317b5aea-b68f-467b-886d-68a49c7194c7%
> 40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOqGuKL_eP0TxbL2TaH__j%2BYLKaOJAMQPhFdFWXya7zOg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: django project avoid reload page where work algorithm

2017-10-07 Thread Tom Forbes
This is the mailing list for the development of Django itself, I would post
this to the Django users mailing list instead if you want any answers to
your question.

It sounds like an ideal use case for celery though.

On 8 Oct 2017 01:41, "Xristos Xristoou"  wrote:

I have a website where the user can be put three numbers on my html
template and get some results from my personal mathematical algorithm. the
result save at user personal table on my database and can see in specific
tab in my website.

my problem is where the algorithm to calculate result maybe take time
between 5-10 minutes in this time the browser stay on reload. if user
change tab or close browser or maybe have problem with internet connection
then loose that request and need again to put the numbers and again wait
for results.

I want after the user request from my form in html to keep this request and
work my algorithm without reload the page and where the algorithm finish
then to send the user some email or message or just need the user visit the
tab of results to see new results.

that I want to avoid the problems where the algorithm is in running and
user loose data or request or time.

is easy to do that using suproccess,celery or RabbitMQ ?

any idea ?

here the code

views.py

def math_alg(request):
if request.method == "POST":
test = request.POST.get('no1')
test = request.POST.get('no3')
test = request.POST.get('no3')
#start algorith
calc_math(no1,no1,no3,result)
instance = rmodel.objects.create(user=request.user,rfield=result)
instance.save
return render(request, 'page.html', {'result': result})

html :

{% csrf_token %}
  op math calculate:
  
  
  
  {{result }}

-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/937d9c18-85e2-4077-a1fe-06ef992ce47e%40googlegroups.
com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMVcZ5%3Dby0B%3Dh83OAgy7Og6Oo61h1GXE-8e%2BSxBRZWHgw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Refactoring the autoreloader

2017-10-07 Thread Tom Forbes
Thanks for the feedback! Watchdog does include a stat watcher, and I think
we can delegate to that. Currently we turn on the file system notifications
if the library is available, but we should add a '--stat' flag to disable
this it you are running on filesystems that don't support this. At which
point we can delegate to the watchdog stat watcher if available, else
fallback to our current implementation.

There hasn't been much feedback on this, so I'm guessing there isn't much
appetite for these changes. I think improving the current code is a good
idea though, even without the filesystem notifications there is some work
that could be done.

On 3 Oct 2017 00:57, "François Freitag" <francois.frei...@gmail.com> wrote:

Hi Tom,

Thank you for the great summary.


> 2. Add support for the "watchdog" library as a replacement for
> pyinotify. Watchdog implements file system notifications for all major
> platforms using a fairly simple API, so we can remove polling and have
> instant reloading. Also support Watchman, a notification Daemon from
> Facebook.

Filesystem polling is required for some setup, such as mounting code using
NFS or rsync, for example using vagrant synced folders[1]. Although it does
not prevent from using the watchdog library, which provides a
PollingObserver[2], I think it's worth keeping that use case in mind. The
PR has a StatReloader which seem able to handle filesystem polling, I would
suggest either keeping it or delegating the polling to watchdog.

[1] https://www.vagrantup.com/docs/synced-folders/nfs.html
[2] https://pythonhosted.org/watchdog/api.html#module-watchdog.
observers.polling

Cheers,
François


On 09/29/2017 12:03 PM, Tom Forbes wrote:

> Hello,
> I've been thinking on and off about how to improve the autoreloader
> implementation and I wanted to gather some feedback on potential solutions.
>
> For some background, Django uses a fairly basic autoreload implementation
> that simply polls the last modified time for loaded Python files once a
> second. While this isn't the most efficient, it does work and has worked
> quite well for a long time. When running manage.py runserver, the
> autoreloader will launch a child "manage.py" with the same arguments and
> this child process actually runs Django and serves requests. To reload, the
> child process exits with exit code 3 and the parent restarts it. The code
> is some of the oldest in Django, with a fair bit of it not touched in 9-12
> years.
>
> While it works (and I'm a believer in "if it isn't broke don't fix it")
> there are some architectural and performance issues with the current code:
>
> - Polling every second is not very efficient
> - Detecting when the child process has exited during startup (i.e problem
> in settings.py) is problematic and the code is rather nasty
> - i18n files are 'reloaded' when they change in a rather hacky way
> (resetting private attributes in another module)
> - There is limited support for extending the current implementation, and
> there are cases during development where the parent autoreloader will
> terminate.
>
> I don't want this email to be too long, so I'm going to summarize what I
> think would be a good approach to tackling these problems.
>
> 1. Refactor the current implementation by removing `pyinotify`, redundant
> python 2 checks and implement a 'file_changed' signal so other parts of
> Django can react to file changes (i.e the i18n module can handle resetting
> it's own state).
> 2. Add support for the "watchdog" library as a replacement for pyinotify.
> Watchdog implements file system notifications for all major platforms using
> a fairly simple API, so we can remove polling and have instant reloading.
> Also support Watchman, a notification Daemon from Facebook.
> 3. Add support for more advanced features, like proper handing of startup
> errors and socket sharing.
>
> I've got a merge request that implements all three stages as a proof of
> concept, but I think it's far too much a change to be done at once and
> should be done carefully stage by stage. One and two are fairly simple to
> implement, but three requires see careful consideration as to the best
> approach (this message is long enough already, I don't want to describe
> them here).
>
> Does anyone have any feedback on these ideas? Is it worth persuing even if
> the current implementation works ok-ish?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com  django-developers+unsubscr...@googlegroups.com>.
> To post to this group, send email to 

Re: WOULD LIKE TO CONTRIBUTE TO DJANGO

2017-10-03 Thread Tom Forbes
Your idea about ignoring conflicts in bulk_create is a great idea, i've
made a merge request that attempts to implements this. If anyone has any
time to review it it would be much appreciated.

I couldn't find a very elegant way of adding it though - Sqlite and mysql
need something added just after the INSERT, whereas postgres needs
something added after the values but before the RETURNING statement. Oracle
doesn't support this as far as I'm aware.

> Qui tacet consentire videtur.

Silence is rarely an agreement on a mailing list, certainly not regarding a
proposed new feature or potentially breaking change. Code speaks louder
than words though :)

On 3 Oct 2017 20:17, "Дилян Палаузов"  wrote:

Hello,

Qui tacet consentire videtur.

I agree there is still a concurrency problem with the approach I
suggested.  INSERT ON CONFLICT DO UPDATE might not work, if the
simultaneously inserted object has for some reason other default values,
that the current object, so that DO UPDATE will overwrite the data.  I am
not talking about other backends.

Another idea to start with:

1. Add on_conflict=IGNORE parameter to bulk_create() -
https://code.djangoproject.com/ticket/28668

The problem is that in order to insert a lot of data in a database fast, a
single INSERT must be sent, but it fails if any of the records were already
in the database, so if this could happen, one has to iterate over the input
and do a separate INSERT for each object, but this is slower than a single
INSERT.

However with something like INSERT ... ON CONFLICT DO NOTHING (which varies
in the different RDBMs) the database is instructed to absorb all the data
that is not already there.

2. Afterwards extend bulk_create(..., on_conflict=IGNORE) to detect for
Postgresql which objects were actually added, as described at
https://groups.google.com/forum/#!topic/django-developers/wdHIYdQHO_0 , and
return only those.

Regards
  Дилян


On 10/03/2017 06:00 PM, Aymeric Augustin wrote:

> Hello,
>
> Since I haven't seen positive feedback from a committer, I'm not convinced
> there's consensus about this change.
>
> Also this doesn't look like a particularly easy topic for a beginner. It
> raises the following questions:
>
> - INSERT ON CONFLICT UPDATE would likely be a better option on Postgres ≥
> 9.5, wouldn't it?
> - what about other databases with built-in backends? third-party backends?
> - is this implementation really appropriate at any isolation level?
> - what are the consequences for performance?
>
> Best regards,
>
> --
> Aymeric.
>
>
> 2017-10-03 17:23 GMT+02:00 Дилян Палаузов >:
>
>
> Hello Muhereza,
>
> I assume you understand by now Django quite well and are willing to
> give something "in return".
>
> Currently QuerySet.get_or_create() consists of two SQL commands:
> SELECT + optional INSERT.  They cause a concurrent problem, if
> another get_or_create() is called between the SQL statements.
>
> With the Postgresql backend it is possible to reduce the queries to
> a single one.
>
> Consider this table:
>
> CREATE TABLE t (
> id SERIAL PRIMARY KEY,
> name VARCHAR(10) NOT NULL UNIQUE,
> comment VARCHAR(10));
>
> The following query can do what get_or_create() currently achieves:
>
> WITH
>maybe_found AS (SELECT * FROM t WHERE t.name
> ='nameD'),
>
>to_be_inserted AS (SELECT 'nameD' as "name", 'comment13' as
> "comment"),
>just_inserted AS (
>   INSERT INTO t (name, comment) SELECT * FROM to_be_inserted
>  WHERE NOT EXISTS(SELECT * FROM maybe_found)
>   RETURNING *)
> SELECT *, FALSE as "created" FROM maybe_found UNION ALL
> SELECT *, TRUE AS "created" FROM just_inserted LIMIT 2;
>
> where "to_be_inserted' contains the values for the new object
> ('default' parameter of get_or_create) and 'nameB' in maybe_found is
> the criterion passed to get().
>
> The challenge is to integrate the WITH ... SELECT query in Django.
>  As guidance I can only suggest looking at the existing code.
>
> Regards
> Дилян
>
>
> On 10/03/2017 10:24 AM, Muhereza Herman wrote:
>
> Hello, anyone reading this please help me out.
> am a new developer but i would like to contribute to_*django*_.
> please guide me on how to do that. thank you.
>
> -- You received this message because you are subscribed to
> the
> Google Groups "Django developers (Contributions to Django
> itself)" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to
> django-developers+unsubscr...@googlegroups.com
> 
>  >.
>
>   

Refactoring the autoreloader

2017-09-29 Thread Tom Forbes
Hello,
I've been thinking on and off about how to improve the autoreloader
implementation and I wanted to gather some feedback on potential solutions.

For some background, Django uses a fairly basic autoreload implementation
that simply polls the last modified time for loaded Python files once a
second. While this isn't the most efficient, it does work and has worked
quite well for a long time. When running manage.py runserver, the
autoreloader will launch a child "manage.py" with the same arguments and
this child process actually runs Django and serves requests. To reload, the
child process exits with exit code 3 and the parent restarts it. The code
is some of the oldest in Django, with a fair bit of it not touched in 9-12
years.

While it works (and I'm a believer in "if it isn't broke don't fix it")
there are some architectural and performance issues with the current code:

- Polling every second is not very efficient
- Detecting when the child process has exited during startup (i.e problem
in settings.py) is problematic and the code is rather nasty
- i18n files are 'reloaded' when they change in a rather hacky way
(resetting private attributes in another module)
- There is limited support for extending the current implementation, and
there are cases during development where the parent autoreloader will
terminate.

I don't want this email to be too long, so I'm going to summarize what I
think would be a good approach to tackling these problems.

1. Refactor the current implementation by removing `pyinotify`, redundant
python 2 checks and implement a 'file_changed' signal so other parts of
Django can react to file changes (i.e the i18n module can handle resetting
it's own state).
2. Add support for the "watchdog" library as a replacement for pyinotify.
Watchdog implements file system notifications for all major platforms using
a fairly simple API, so we can remove polling and have instant reloading.
Also support Watchman, a notification Daemon from Facebook.
3. Add support for more advanced features, like proper handing of startup
errors and socket sharing.

I've got a merge request that implements all three stages as a proof of
concept, but I think it's far too much a change to be done at once and
should be done carefully stage by stage. One and two are fairly simple to
implement, but three requires see careful consideration as to the best
approach (this message is long enough already, I don't want to describe
them here).

Does anyone have any feedback on these ideas? Is it worth persuing even if
the current implementation works ok-ish?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMT9qDk-4pKKXSJysEQCmd6CGxMZBYZs_7BQs_WbAqL6g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: bulk_create on Postgresql: on conflict do nothing / post_save signal

2017-09-28 Thread Tom Forbes
I've been in similar situations before, you can usually get away with using
a single query to fetch existing records and only pass data that doesn't
exist to bulk_create. This works great for a single identity column, but if
you have multiple it gets messy.

It seems all supported databases offer at least ON CONFLICT IGNORE in some
form or another, with pretty similar syntax.

On 28 Sep 2017 18:11, "Дилян Палаузов"  wrote:

>
> Hello,
>
> I want after a user request to be sure that certain objects are stored in
> a Postgres database, even if before the request some of the objects were
> there.
>
> The only way I can do this with django, not talking about raw sql, is with
> "for obj in objects: Model.objects.get_or_create(obj)".  It works, but
> creates several INSERTs, and is hence suboptimal.
>
> I cannot use bulk_create(), which squeezes all the INSERTs to a single
> one, as it does not work, if any of the to-be-inserted rows was already in
> the database.
>
> In Postgresql this can be achieved by sending "INSERT ... ON CONFLICT DO
> NOTHING".
>
> I propose changing the interface of QuerySet.bulk_create to accept one
> more parameter on_conflict, that can be a string e.g. "DO NOTHING" or
> Q-objects (which could be used to implement ON CONFLICT DO UPDATE SET ...
> WHERE ...
>
> def bulk_create(self, objs, batch_size=None, on_conflict=None): ...
>
> What are the arguments against or in favour?
>
> The further, bulk_create() does not send post_save signal, because it is
> difficult to implement with the standard backends, except with postgresql.
>
> I propose extending the implementation to send the signal:
>
> https://code.djangoproject.com/ticket/28641#comment:1
>
> when Postgresql is used.  I assume there a no users, who want to get a
> (post_save) signal on save() but not on bulk_create().
>
> Combining ON CONFLICT DO NOTHING with sending post_save gets however
> nasty, as "INSERT ... ON CONFLICT DO NOTHING RETURNING id;" does not return
> anything on unchanged rows, hence the system knows at the end how much rows
> were changed, but not which, so it cannot determine for which objects to
> send post_save.  At least I have not found a way how to figure out which
> rows were inserted/not inserted.
>
> However, this can be achieved by RETURNING * and then comparing the
> returned objects to the sent objects, eventually making bulk_create()
> return the objects actually inserted in the database.
>
> These changes will allow a switch to a single INSERT on Postgresql.
>
> Regards
>   Дилян
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/ms
> gid/django-developers/daa88462-c095-dfcc-2ce7-6d34f6bbc2f6%40aegee.org.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOUGzYixc4cvPF2%2B_VTo2YwqwVDk%2BTkErxW3hjXvpaXbQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Templates: __html__ method support

2017-09-26 Thread Tom Forbes
The problem with something more complex like yaml or json is it's not easy
to combine the output. If those methods return a string, as in actual json,
it's not easy to do anything with them (like combine them into an array or
another object). YAML is also whitespace sensitive. If they return a dict
or some python object that can be combined and serialised as a whole, then
that's kind of confusing and could be more generic.

XML and HTML don't suffer as much from this I think.

On 26 Sep 2017 21:18, "Brice Parent"  wrote:

> I like the idea a lot. But wouldn't it be better to have it as a separate
> library? It seems to be something that could be quite common, and have many
> use cases both inside and outside Django (we could sometimes benefit from
> having an html exception message for example).
>
> And I guess that would mean we should also have an html() function which
> would call .__html__() if it exists, and fallback to .__str__() if not.
>
> Django's template would call this html() function in its templates, while
> models and other objects could just declare this __html__() method.
>
> And if it gets adopted by more than just Django, it could be integrated
> into the standard Python library and benefit to many more projects.
>
> Only question: Where should it stop? Should there also be a __json__, a
> __yaml__ and an __xml__ methods? Those are also quite common
> representations we could want from a class, even more nowadays that a big
> tendency is to develop microservices which communicate through APIs, and
> frontends being more and more delegated to javascript libraries.
>
> Le 26/09/17 à 14:34, Jonas H a écrit :
>
> Proposal: Support the __html__ method as an alternative/addition to the
> __str__ for turning objects into strings in the template layer.
>
> If this has been discussed before, please point me to it; I couldn't find
> anything with the search function.
>
> Some custom classes may have, in addition to a __str__ representation, a
> natural representation that is better suited for HTML output. Example:
>
> class Money:
> def __init__(self, amount, currency):
> self.amount = amount
> self.currency = currency
>
> def __str__(self):
> return '%s %s' % (self.currency, self.amount)
>
> def __html__(self):
> # Always show amount and currency on same line
> return '%s\xa0%s' % (self.currency, self.amount)
>
> `conditional_escape` and friends already consider the __html__ method, and
> this works out well:
>
> >>> str(Money(1, '$'))
> '$ 1'
> >>> conditional_escape(Money(1, '$'))
> '$\xa01'
>
> In templates however it doesn't work that way because variables are always
> turned into strings before stuffing them into `conditional_escape` (see
> https://github.com/django/django/blob/98706bb35e7de0e445cc336f669919
> 047bf46b75/django/template/base.py#L977). My suggestion is to change the
> behaviour of that function so that it works as follows:
>
> - Given I write {{ foo }}
> - Does foo have a __html__ method? If yes, return `foo.__html__()`
> - Otherwise, return `conditional_escape(str(foo))`
>
> Do think that's a good idea?
>
> Jonas
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/ea088de2-e538-4808-a7fd-
> 8726929e2b91%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/b3d37420-052d-74a2-5a39-e10d4c180f17%40brice.xyz
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

Re: CONTRIBUTION TO DJANGO

2017-09-22 Thread Tom Forbes
Hey Heba,
For a few popular packages on Github it could be as simple as making a
merge request and changing their tox.ini or travis.yml file. Once the
django 2.0 alpha is published on PyPi (sometime very soon) you could make a
merge request to run the projects tests with the alpha, and inspect the CI
output for any failures, then fix those. Alternatively, clone each project
and run the tests locally (but this can get tiring for lots of individual
different projects).

For example, django-reversion has a tox.ini here:
https://github.com/etianen/django-reversion/blob/master/tox.ini - If you
make a merge request to that project adding Django 2.0 alpha to the `deps`
section (line ~16) and the `test` section (line 4) it will run the tests on
Travis CI and report any failures. You could post on the django-users
mailing list for some ideas of what packages to make changes for (and for
more help with specific things like tox/travis), but I would recommend:
django-reversion, django-mptt, silk and django-debug-toolbar:

https://github.com/etianen/django-reversion/blob/master/tox.ini
https://github.com/django-mptt/django-mptt/blob/master/tox.ini
https://github.com/jazzband/silk/blob/master/.travis.yml
https://github.com/jazzband/django-debug-toolbar/blob/master/tox.ini

An initial merge request testing on Django 2.0 alpha is the first step,
then fixing any issues that are discovered is the next (and harder) one.

On Thu, Sep 21, 2017 at 6:55 PM, Heba Khan  wrote:

> Can you suggest a way of how to test Django projects ad third party
> packages please?
>
> On Thursday, 21 September 2017 21:00:36 UTC+5:30, Adam Johnson wrote:
>>
>> There's a whole documentation page on this: https://docs.djangoproje
>> ct.com/en/dev/internals/contributing/
>>
>> There aren't many easy pickings tickets, plus most of the effort right
>> now is being put into features for the 2.0 feature freeze. I'd suggest the
>> biggest contribution you can make right now is testing Django projects or
>> third party packages with 2.0 and finding bugs or helping them become
>> compatible.
>>
>> On 21 September 2017 at 15:17, Heba Khan  wrote:
>>
>>> Hello!
>>>
>>> I'm an undergrad student of B.Tech. in Computer Science and we've been
>>> assigned a project to contribute in an open source project. My team members
>>> and I decided to pick Django since it is one of the most well known and
>>> widely used open source projects. We need help in deciding what
>>> contributions to make to the repository and how to go about it. Please keep
>>> the following in mind:
>>>
>>> 1. We're students with Intermediate coding skills and intermediate
>>> knowledge of Python along with a good hold on HTML, CSS, JavaScript and
>>> JQuery.
>>> 2. We need some easy contribution ideas which can be executed in a short
>>> span of time.
>>> 3. We will be needing guidance and future help from the community as
>>> well.
>>>
>>> Thank you in advance.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit https://groups.google.com/d/ms
>>> gid/django-developers/236511d3-54de-441a-a423-57cc01143be0%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Adam
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/c3feb2a3-4cec-4ac2-9bc1-
> e61e90165913%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at 

Re: New Feature: Allow password reset token to expire in under a day

2017-09-21 Thread Tom Forbes
It also seems odd to express it as seconds, it's often going to be a large
value between an hour and several days and the lowest resolution for the
value anyone would need is minutes.

On 22 Sep 2017 01:29, "Tom Forbes" <t...@tomforb.es> wrote:

> I would still vote for a timedelta, im not sure if there is a strong
> consensus in the thread.
>
> Representing the time as seconds always irks me, you can make it more
> readable by using multiplication but you often end up with a comment anyway
> and it doesn't scan nearly as well. Having to do 'timedelta.seconds' is OK,
> but it seems a bit like busywork.
>
> It's a small thing but I don't see any practical problem with just
> accepting a timedelta, they are nicer to work with in the settings file
> itself and within Django, especially if the TimestampSigner accepts them
> natively and we start to use that.
>
> On 21 Sep 2017 22:41, "Zhiqiang Liu" <zachliu...@gmail.com> wrote:
>
> If most agree, I will proceed with using seconds.
>
> It is a good idea for the potential documentation Dylan!
>
> Zach
>
>
> On Thursday, September 21, 2017 at 10:09:50 AM UTC-4, Dylan Reinhold wrote:
>
>> I still think seconds are the way to go, but maybe the documentation
>> could give a clue that timedelta().seconds can be used for readability
>> PASSWORD_RESET_TIMEOUT = datetime.timedelta(hours=6, minutes=30).seconds
>>
>> Dylan
>>
>> On Thu, Sep 21, 2017 at 6:14 AM, Zhiqiang Liu <zachl...@gmail.com> wrote:
>>
>>> Yeah I don't think float number of days is a good choice because the
>>> calculation will be weird with precision issues.
>>>
>>> I think it makes sense to use PASSWORD_RESET_TIMEOUT. For timedelta vs.
>>> integer seconds. Timedelta has the benefit of readability, but integer has
>>> the benefit of simplicity. I think in SETTINGS everything should be as
>>> simple as possible, so I think integer seconds is a better choice here. And
>>> it is used in most applications too.
>>>
>>>
>>> On Thursday, September 21, 2017 at 8:56:36 AM UTC-4, charettes wrote:
>>>>
>>>> That's what I proposed on the ticket but I feel like it felt odd to me,
>>>> the setting name does't suggest this is possible and it might be hard to
>>>> achieve exact second precious because of float rounding?
>>>>
>>>> In my opinion introducing PASSWORD_RESET_TIMEOUT with timedelta support
>>>> would be the best option.
>>>>
>>>> Simon
>>>>
>>>> Le jeudi 21 septembre 2017 05:26:23 UTC-4, Adam Johnson a écrit :
>>>>>
>>>>> Why not just keep PASSWORD_RESET_TIMEOUT_DAYS and allow floats? Then
>>>>> you can just do 1/24 for an hour.
>>>>>
>>>>> On 21 September 2017 at 09:50, Eddy C <coupo...@chicheng.me> wrote:
>>>>>
>>>>>> I think Minute, with default value 30 or 60, is the best unit for
>>>>>> this setting.
>>>>>>
>>>>>> 3 minutes (even 1) is short enough for edge case and 720 (12 hours)
>>>>>> also looks good.
>>>>>>
>>>>>> On Thursday, September 21, 2017 at 6:22:20 PM UTC+10, Tom Forbes
>>>>>> wrote:
>>>>>>>
>>>>>>> I think we shouldn't shoe-horn a timedelta into the existing
>>>>>>> setting, so my vote is with the second option, but I think a timedelta 
>>>>>>> is
>>>>>>> much more readable than just an integer.
>>>>>>>
>>>>>>> Also, the existing 3 day timeout for password links is quite
>>>>>>> surprising from a security point of view. The consultants I work with 
>>>>>>> would
>>>>>>> flag up a token that lasts longer than 12 hours as an issue during a
>>>>>>> pentest.
>>>>>>>
>>>>>>> IMO a new, far shorter default should be added to this setting.
>>>>>>>
>>>>>>> On 21 Sep 2017 03:56, "Zhiqiang Liu" <zachl...@gmail.com> wrote:
>>>>>>>
>>>>>>> I need general consensus on how to proceed with supporting password
>>>>>>> expire time to be under a day. Currently it is not possible because we 
>>>>>>> use
>>>>>>> PASSWORD_RESET_TIMEOUT_DAYS.
>>>>>>>
>>>>>>> In ticket 28622 <https://code.djangoproject.com/ticket/28622&g

Re: New Feature: Allow password reset token to expire in under a day

2017-09-21 Thread Tom Forbes
I would still vote for a timedelta, im not sure if there is a strong
consensus in the thread.

Representing the time as seconds always irks me, you can make it more
readable by using multiplication but you often end up with a comment anyway
and it doesn't scan nearly as well. Having to do 'timedelta.seconds' is OK,
but it seems a bit like busywork.

It's a small thing but I don't see any practical problem with just
accepting a timedelta, they are nicer to work with in the settings file
itself and within Django, especially if the TimestampSigner accepts them
natively and we start to use that.

On 21 Sep 2017 22:41, "Zhiqiang Liu" <zachliu...@gmail.com> wrote:

If most agree, I will proceed with using seconds.

It is a good idea for the potential documentation Dylan!

Zach


On Thursday, September 21, 2017 at 10:09:50 AM UTC-4, Dylan Reinhold wrote:

> I still think seconds are the way to go, but maybe the documentation could
> give a clue that timedelta().seconds can be used for readability
> PASSWORD_RESET_TIMEOUT = datetime.timedelta(hours=6, minutes=30).seconds
>
> Dylan
>
> On Thu, Sep 21, 2017 at 6:14 AM, Zhiqiang Liu <zachl...@gmail.com> wrote:
>
>> Yeah I don't think float number of days is a good choice because the
>> calculation will be weird with precision issues.
>>
>> I think it makes sense to use PASSWORD_RESET_TIMEOUT. For timedelta vs.
>> integer seconds. Timedelta has the benefit of readability, but integer has
>> the benefit of simplicity. I think in SETTINGS everything should be as
>> simple as possible, so I think integer seconds is a better choice here. And
>> it is used in most applications too.
>>
>>
>> On Thursday, September 21, 2017 at 8:56:36 AM UTC-4, charettes wrote:
>>>
>>> That's what I proposed on the ticket but I feel like it felt odd to me,
>>> the setting name does't suggest this is possible and it might be hard to
>>> achieve exact second precious because of float rounding?
>>>
>>> In my opinion introducing PASSWORD_RESET_TIMEOUT with timedelta support
>>> would be the best option.
>>>
>>> Simon
>>>
>>> Le jeudi 21 septembre 2017 05:26:23 UTC-4, Adam Johnson a écrit :
>>>>
>>>> Why not just keep PASSWORD_RESET_TIMEOUT_DAYS and allow floats? Then
>>>> you can just do 1/24 for an hour.
>>>>
>>>> On 21 September 2017 at 09:50, Eddy C <coupo...@chicheng.me> wrote:
>>>>
>>>>> I think Minute, with default value 30 or 60, is the best unit for this
>>>>> setting.
>>>>>
>>>>> 3 minutes (even 1) is short enough for edge case and 720 (12 hours)
>>>>> also looks good.
>>>>>
>>>>> On Thursday, September 21, 2017 at 6:22:20 PM UTC+10, Tom Forbes wrote:
>>>>>>
>>>>>> I think we shouldn't shoe-horn a timedelta into the existing setting,
>>>>>> so my vote is with the second option, but I think a timedelta is much 
>>>>>> more
>>>>>> readable than just an integer.
>>>>>>
>>>>>> Also, the existing 3 day timeout for password links is quite
>>>>>> surprising from a security point of view. The consultants I work with 
>>>>>> would
>>>>>> flag up a token that lasts longer than 12 hours as an issue during a
>>>>>> pentest.
>>>>>>
>>>>>> IMO a new, far shorter default should be added to this setting.
>>>>>>
>>>>>> On 21 Sep 2017 03:56, "Zhiqiang Liu" <zachl...@gmail.com> wrote:
>>>>>>
>>>>>> I need general consensus on how to proceed with supporting password
>>>>>> expire time to be under a day. Currently it is not possible because we 
>>>>>> use
>>>>>> PASSWORD_RESET_TIMEOUT_DAYS.
>>>>>>
>>>>>> In ticket 28622 <https://code.djangoproject.com/ticket/28622> we
>>>>>> have two options.
>>>>>>
>>>>>> One is to continue to use the same setting
>>>>>> PASSWORD_RESET_TIMEOUT_DAYS, but change the value to non-integer (such as
>>>>>> timedelta) so we can send hours, minutes, etc to it.
>>>>>>
>>>>>> The other one is to create a new setting like PASSWORD_RESET_TIMEOUT
>>>>>> which takes seconds.To support backward compatibility, I think we should
>>>>>> keep PASSWORD_RESET_TIMEOUT_DAYS and its default value of 3. Only use
>>>>>> P

Re: There should be a way to make Templates substitution to raise an exception on error

2017-09-21 Thread Tom Forbes
You could perhaps emulate that with a template tag, it seems it would be
best if this was kept in the template rather than in some associated Python
file:

@register.simple_tag(takes_context=True)
def requires(context, *names):
for name in names:
if name not in context:
raise RuntimeError('Missing context: {0}'.format(name))

{% requires "name" "other" %}

On Thu, Sep 21, 2017 at 2:37 PM, Zhiqiang Liu  wrote:

> To continue the previous comment.
>
> template can raise error give warning if required contexts are not
> provided or the types are not correct. You can have something not
> isRequired in contextTypes too but types can be check if the context is
> actually passed to template.
>
>
> On Thursday, September 21, 2017 at 9:35:05 AM UTC-4, Zhiqiang Liu wrote:
>>
>> This is not 100% related to the ticket, but something to think about.
>>
>> In ReactJS, there a concept called propTypes, which will check all props
>> (they are similar to context in concept I think) listed with types in UI
>> component.
>>
>> So maybe we can have something similar in django template system that a
>> template can have an attribute called contextTypes and it can be a dict of
>> context the template may have. It doesn't need to have all possible context
>> values, only include those you want to make sure the context are passed and
>> value types are correct.
>>
>> A simple example can be
>>
>>
>> contextTypes = {
>> "name": contextTypes.string.isRequired,
>>
>> }
>>
>>
>>
>> On Thursday, September 21, 2017 at 7:05:28 AM UTC-4, Shreyas Pandya wrote:
>>>
>>> Hi All,
>>>
>>> What is your opinion on having an option to raise an error in template
>>> if variable is not found in context. This may be useful for automated
>>>  tests as discussed in ticket.
>>>
>>> reference ticket #28618  ;
>>>
>>> Thanks
>>>
>>> regards
>>> Shreyas
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-developers/0e82ce82-3a0d-44a6-9f68-
> f359a7354eb9%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJM2mPKcPD-9wpDMA1-MNe029Wtzz%2B2U8t1wzLzT_UOYUA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: There should be a way to make Templates substitution to raise an exception on error

2017-09-21 Thread Tom Forbes
You could perhaps emulate something like that with a template tag, couldn't
you?

@register.simple_tag(takes_context=True)
def requires(context, *names):
for name in names:
if name not in context:
raise RuntimeError('{0} is not in the template
context'.format(name))

And in the template:

{% requires "name" "other" %}

Seems that it would be best if that kind of data lived with the template,
rather than outside in a Python file.

On 21 Sep 2017 14:37, "Zhiqiang Liu"  wrote:

> To continue the previous comment.
>
> template can raise error give warning if required contexts are not
> provided or the types are not correct. You can have something not
> isRequired in contextTypes too but types can be check if the context is
> actually passed to template.
>
>
> On Thursday, September 21, 2017 at 9:35:05 AM UTC-4, Zhiqiang Liu wrote:
>>
>> This is not 100% related to the ticket, but something to think about.
>>
>> In ReactJS, there a concept called propTypes, which will check all props
>> (they are similar to context in concept I think) listed with types in UI
>> component.
>>
>> So maybe we can have something similar in django template system that a
>> template can have an attribute called contextTypes and it can be a dict of
>> context the template may have. It doesn't need to have all possible context
>> values, only include those you want to make sure the context are passed and
>> value types are correct.
>>
>> A simple example can be
>>
>>
>> contextTypes = {
>> "name": contextTypes.string.isRequired,
>>
>> }
>>
>>
>>
>> On Thursday, September 21, 2017 at 7:05:28 AM UTC-4, Shreyas Pandya wrote:
>>>
>>> Hi All,
>>>
>>> What is your opinion on having an option to raise an error in template
>>> if variable is not found in context. This may be useful for automated
>>>  tests as discussed in ticket.
>>>
>>> reference ticket #28618  ;
>>>
>>> Thanks
>>>
>>> regards
>>> Shreyas
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/ms
> gid/django-developers/0e82ce82-3a0d-44a6-9f68-f359a7354eb9%
> 40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJOcTUqeaBbeMy8OC%3D_Kf9_1D_Qz2x%3DyTBNSxEBKGgkDgA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: New Feature: Allow password reset token to expire in under a day

2017-09-21 Thread Tom Forbes
I think we shouldn't shoe-horn a timedelta into the existing setting, so my
vote is with the second option, but I think a timedelta is much more
readable than just an integer.

Also, the existing 3 day timeout for password links is quite surprising
from a security point of view. The consultants I work with would flag up a
token that lasts longer than 12 hours as an issue during a pentest.

IMO a new, far shorter default should be added to this setting.

On 21 Sep 2017 03:56, "Zhiqiang Liu"  wrote:

I need general consensus on how to proceed with supporting password expire
time to be under a day. Currently it is not possible because we use
PASSWORD_RESET_TIMEOUT_DAYS.

In ticket 28622  we have two
options.

One is to continue to use the same setting PASSWORD_RESET_TIMEOUT_DAYS, but
change the value to non-integer (such as timedelta) so we can send hours,
minutes, etc to it.

The other one is to create a new setting like PASSWORD_RESET_TIMEOUT which
takes seconds.To support backward compatibility, I think we should keep
PASSWORD_RESET_TIMEOUT_DAYS and its default value of 3. Only use
PASSWORD_RESET_TIMEOUT when provided.

I'm unsure which one is better, so inputs are welcome.

-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/ms
gid/django-developers/c8e96008-eb95-4924-8e5e-9b02d6b90c99%
40googlegroups.com

.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMiAMnOefVvoX1ewp_%2B05%2B4y%2BOzRrpq9nEC7vO%2Bt57kGQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: ConnectionResetError in test_closes_connection_without_content_length

2017-09-20 Thread Tom Forbes
I added this test for the keep-alive release blocker. It's possible that
the exceptions raised in that test vary by OS at least on 3.4, the docs
suggest that it should throw BadStatusLine instead.

ConnectionResetError is also a correct exception to raise in that specific
case, we should just add that to the try/catch statement I think.

On 20 Sep 2017 10:14 am, "Adam Johnson"  wrote:

Hi Zhiqiang

It's hard to tell what the problem is with more details. From your
traceback it looks like you're running Python3.4 on MacOS but there's no
indication of your database or other settings.

If you run the tests on the 1.11 branch, does it pass? If so, you can
bisect to find the regression and report that as a proper bug, instructions
here: https://docs.djangoproject.com/en/dev/internals/contributing/
triaging-tickets/#bisecting-a-regression

Otherwise, more information is needed to debug.

On 20 September 2017 at 03:46, Zhiqiang Liu  wrote:

> I run the test suits for master and get the following error.
>
> Traceback (most recent call last):
>   File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework
> /Versions/3.4/lib/python3.4/unittest/case.py", line 58, in
> testPartExecutor
> yield
>   File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework
> /Versions/3.4/lib/python3.4/unittest/case.py", line 577, in run
> testMethod()
>   File "/Users/ZachLiu/CS/Python/Projects/Django/django/django/test/utils.py",
> line 371, in inner
> return func(*args, **kwargs)
>   File 
> "/Users/ZachLiu/CS/Python/Projects/Django/django/tests/servers/tests.py",
> line 80, in test_closes_connection_without_content_length
> conn.getresponse()
>   File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework
> /Versions/3.4/lib/python3.4/http/client.py", line 1171, in getresponse
> response.begin()
>   File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework
> /Versions/3.4/lib/python3.4/http/client.py", line 351, in begin
> version, status, reason = self._read_status()
>   File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework
> /Versions/3.4/lib/python3.4/http/client.py", line 313, in _read_status
> line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
>   File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework
> /Versions/3.4/lib/python3.4/socket.py", line 374, in readinto
> return self._sock.recv_into(b)
> ConnectionResetError: [Errno 54] Connection reset by peer
>
> Is there anything I missed? Anyone can help?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/ms
> gid/django-developers/9b97b2a3-160f-4c34-b4fd-c447cd1f6895%
> 40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/CAMyDDM0nGnmWyjatkYhk695O%
3DPc-bJVAbKGqaXV4xVQxF96vGg%40mail.gmail.com

.

For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJP264dGhLfVVVxqc%3Dnn5PmFtSdDGOBwrkRU8_2RSJ%2BxeA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Having a MongoDB connector for Django

2017-09-15 Thread Tom Forbes
You're right for pointing out that some elements of a relational database
can be emulated in a non relational one, like Mongodb. The in-python joins
sound s bit scary though.

However I would argue that if you find yourself needing such things your
data is relational, and therefore a relational database is perhaps a better
fit than a non relational one.

On 15 Sep 2017 18:44, "Nes Dis"  wrote:

I would like to thank everyone for their valuable comments. Simultaneously
I would like to comment on some conceptions regarding using MongoDB. Its
not accurate to state that relational joins cannot happen in MongoDB. It
can be done at the application level. LEFT JOIN and INNER JOIN. A detailed
description
 of this
is available.

Admin modules rely heavily on SQL, So cannot be ported to MongoDB. But
admin porting to MongoDB is working successfully in djongo.

Regards
Nes Dis


On Friday, 8 September 2017 18:33:59 UTC+5:30, Nes Dis wrote:

> Hello
>
> I am wondering what is the state of the art on Django having a backend
> connector for MongoDB database backend. There are a few solutions out there
> but they don't work as expected.
>
> A possible solution for this is to have a connector which translates SQL
> queries created in Django, into MongoDB queries.
>
> I would like to hear the *expert opinion *from the esteemed members of
> this group on this concept.
>
> A working solution for this can be found here: djongo
> . (Django + Mongo = Djongo) The project
> is hosted on github.
>
> Regards
> Nes Dis
>
-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/e0ac0919-0ae6-4a42-ba7c-c558bc42d141%40googlegroups.
com

.

For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJP1GNXO_7%3Di5NdqL-gqvtGMdi6UCyD6W1BB79u72wTmRQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Having a MongoDB connector for Django

2017-09-08 Thread Tom Forbes
JSON support in postgres is a great middle ground for storing unstructured
documents.

I don't know the specifics of your application, but I would be surprised if
a document orientated database is a perfect fit. The majority of such apps
are semi-structured, where you have some relations but also some
unstructured data. Postgres is an absolutely perfect fit for this.

On 8 Sep 2017 17:57, "Adam Johnson"  wrote:

I agree, I think forcing Django's ORM to work on MongoDB is not a great
idea. Django relies heavily on transactions and other relational goodness.

Have you tried storing JSON in your Postgres/MySQL database? Django can
work with that with contrib.postgres/django-mysql 

On 8 September 2017 at 16:51, 'Tom Evans' via Django developers
(Contributions to Django itself)  wrote:

> Short answer: always use the appropriate tool
>
> Relational databases and document stores have different uses and
> purposes. Using a document store like a relational database (eg, with
> an ORM (emphasis on the R)) is a bad idea, and using a relational
> database as a document store is similarly foolish.
>
> Work out what questions you want to ask of your data, then structure
> the data in a way that allows you to query it efficiently.
>
> If the format desired is a document store, I wouldn't attempt to
> shoehorn that in to an ORM wrapper, I'd use something like mongothon.
>
> Cheers
>
> Tom
>
> On Fri, Sep 8, 2017 at 8:50 AM, Nes Dis  wrote:
> > Hello
> >
> > I am wondering what is the state of the art on Django having a backend
> > connector for MongoDB database backend. There are a few solutions out
> there
> > but they don't work as expected.
> >
> > A possible solution for this is to have a connector which translates SQL
> > queries created in Django, into MongoDB queries.
> >
> > I would like to hear the expert opinion from the esteemed members of this
> > group on this concept.
> >
> > A working solution for this can be found here: djongo. (Django + Mongo =
> > Djongo) The project is hosted on github.
> >
> > Regards
> > Nes Dis
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Django developers (Contributions to Django itself)" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to django-developers+unsubscr...@googlegroups.com.
> > To post to this group, send email to django-developers@googlegroups.com.
> > Visit this group at https://groups.google.com/group/django-developers.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/django-developers/b0ce04d1
> -62cb-4765-b850-06c4a5b0607f%40googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/ms
> gid/django-developers/CAFHbX1%2BNpZuWrAgcMwkcJQ4Xg4TrM700JRs
> YV_AnW05_9L3joA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/CAMyDDM1YMFUT_oSGYuj-2Uqu_BSteqFbb_84Uz-zMauoE-m6Kg%
40mail.gmail.com

.

For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJPwC91aBLto5%3DzjPnGPV9qUYWtT2vsywza5%2BvtxGbNFKQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Add support for multiple file fields

2017-08-31 Thread Tom Forbes
(I wrote a reply earlier and sent it, but it appears to have disappeared
into /dev/null. Apologies if it comes through at some later date.)

It seems adding a multiple kwarg and retrofitting it onto the existing
FileField might not be the best idea. It might cause confusion with
existing FileField APIs like size and name which don't make sense with
multiple files.

We could add a MultiFileField that had a files attribute, which returns a
list of individual objects analogous to a single FileField?

I guess that MultiFileField could be hidden between a 'multiple' keyword
argument, but I'm not sure a migration from true to false would be possible
which could also be confusing.


On 31 Aug 2017 17:31, "Johannes Hoppe"  wrote:

I tried to exclude my personal opinion and preference from the ticket
description, to have an open discussion.
So here goes my personal opinion:

It seems odd to me that the FileField is limited to 100 characters. I could
not find any reference to why the field was limited in the first place.
Furthermore I do not know of any file system with a 100 char limitation nor
are URLs limited to only 100 chars.

Therefore I would suggest basing the FileField upon the TextField.

I would recommend splitting the issue tho. I would first add support form
multiple files to form. This is a nice feature in itself and requires
little work and good documentation. I would presume that such a feature
would spawn a larger discussion on how to store those files. As multiple
model instances? As an Array in Postgres? As CSV or JSON in a TextField? I
think we can figure this out later or not at all if we don't come to a
conclusion.

I would love to work on that. I do maintain [django-s3file](​https://
github.com/codingjoe/django-s3file/) and [django-stdimage](​https://
github.com/codingjoe/django-stdimage/) and have some experience with custom
the FileInput and FileField. Let me know if I can be of any help.

Cheers
-joe

On Thursday, August 31, 2017 at 6:30:37 PM UTC+2, Johannes Hoppe wrote:
>
> Hi there!
>
> I already created a ticket regarding this matter, but I think this thicket
> requires some discussion prior to crafting a solution.
> https://code.djangoproject.com/ticket/28554
>
> The django.db.models.FileField currently allows only to store a single
> file URL.
> This behavior seems outdated considering that input[type=file] supports a
> multiple attribute, which is supported by all major Browsers.
> See: ​https://www.w3schools.com/tags/att_input_multiple.asp
>
> I would suggest to have a similar attribute on the database field itself,
> as well as on the form field and widget.
>
> The major point for discussion would be how to store the data. The easiest
> would be to coma separate the paths in a single char field. The major
> concern here would be the lack of atomicity. For databases like Postgres I
> would be sensical to use the ArrayField as an underlying structure. Another
> way for serialisation would be the use of JSONs in a char field.
>
> All solutions based on a text or char field have an issue related to the
> compatibility with the current FileField which is limited to 100
> characters. The type would need to be migrated to a text type or the length
> would need to be increased. The latter options bares the question to which
> point.
>
> Another approach would be to defer the storage issue and only provide
> support for multiple files on form level.
> This would mean to add a multiple attribute to the form field and widget.
> Users would be able to create forms with multiple files in a single field
> and handle storage according to their preference.
>
-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/2b9bb49f-1cbc-4ea6-94df-774dfad43114%40googlegroups.
com

.

For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 

Re: Optimizing out unused annotations from count queries

2017-08-21 Thread Tom Forbes
> Interestingly enough, just doing a .filter(m2m_relation__foo='bar') might
change the results

Is that not because it's a filter? Would ".
annotate(x=F('m2m_relation__foo'))" change the results in any way? Sorry if
I'm not following.



On 21 Aug 2017 06:58, "Anssi Kääriäinen"  wrote:

On Sunday, August 20, 2017 at 2:48:23 AM UTC+3, Josh Smeaton wrote:

> Thanks for making me elaborate, because I'm probably wrong. I'll step
> through what my thinking was though.
>
> '.annotate(local=F('related__field'))' will join to the related table. If
> `related` is a nullable foreign key, then a join would do an implicit
> filter, removing the row from the result set if it had no relation, and
> reducing the count.
>
> But Django models a nullable foreign key join as a LEFT JOIN which
> wouldn't filter the row out and wouldn't reduce the count.
>
> Multivalue relationships (reverse foreign key and m2m joins) would need
> some care though I think. I'm not sure if the same annotation above works
> on m2m joins or not.
>

Interestingly enough, just doing a .filter(m2m_relation__foo='bar') might
change the results. For m2m relations, any non-aggregate annotation is not
safe to remove.

I believe our test cases are not fully covering here, so removing the
annotations will be a bit risky. If there are cases where removal is known
to be safe, it's of course a good idea to remove anything extra from the
query.

 - Anssi

-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/51a5c1ad-e071-482b-aa3a-184dd4145413%40googlegroups.
com

.

For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJNfBAw5HzzT%2BZTZcYxTRTWSFBPSM%3DHSQ%3Dh4HBUK71JUwg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Optimizing out unused annotations from count queries

2017-08-19 Thread Tom Forbes
Thanks for your reply Josh. Can you elaborate on why optimizing out '
.annotate(local=F('related__field'))' would not be safe?

On 20 Aug 2017 00:10, "Josh Smeaton" <josh.smea...@gmail.com> wrote:

I'd like to see this provided all bases were covered. I'll just list below
the cases where I think it wouldn't be safe.

- Filtered annotations
- Annotations that join to anything other than a non-null foreign key:
.annotate(local=F('related__field'))
- Annotations that have a GROUP BY on fields that are not just the PK or
entire field set (with .values())

The group by one I'm unsure of, because I forget if a count is called with
an aggregate query as a subquery or if it's an error.

I think an optimisation is available if:

1. No filters ref an annotation
2. values() has not been called
3. There are no joins at all

Further optimisation scenarios may be available with stricter rules (such
as join types), but we could look at doing that separately if it was
difficult.

On Sunday, 20 August 2017 03:11:13 UTC+10, Tom Forbes wrote:
>
> Hello,
> I think there is potential room for improvement with how Django handles
> count() queries, specifically relating to annotations. Currently any
> annotations that are added to a queryset are included in the SQL statement
> that is generated in the count() query, including all joins and SQL
> function calls - I've personally been bitten by this with DRF and a query
> with a particularly complex set of annotations where the count query took
> longer than fetching the data itself, but was semantically equivalent to a
> much faster, plain Model.objects.count() call.
>
> I think in most cases annotations can be stripped, for example in the two
> queries below the annotation is not filtered on so it can be safely removed
> from the query without affecting the result:
>
> one = Book.objects.annotate(Count('chapters')).count()
> two = Book.objects.count()
>
> I wanted to gather some feedback from the group as to whether this always
> holds true: if an annotation is not filtered (or ordered on) it can always
> be safely removed from a count query? I wouldn't be surprised if there is a
> case where this isn't safe, but I cannot think of one.
>
> I've made an initial merge request that removes annotations from querysets
> that have no filter or ordering that depends on annotations here:
> https://github.com/django/django/pull/8928. It seems like Django also has
> all the information to optimize out annotations that are not directly or
> indirectly used by filters, but I wanted to gather some feedback before I
> attempted this.
>
> Tom
>
-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/506d0f2c-51e8-400f-a4fb-66c2b8597aeb%40googlegroups.
com
<https://groups.google.com/d/msgid/django-developers/506d0f2c-51e8-400f-a4fb-66c2b8597aeb%40googlegroups.com?utm_medium=email_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJP%3DwknS2hDLT2XE4qxKn6MXrMNK2C4AHMLpY7T0ywi%3D9A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Optimizing out unused annotations from count queries

2017-08-19 Thread Tom Forbes
Hello,
I think there is potential room for improvement with how Django handles
count() queries, specifically relating to annotations. Currently any
annotations that are added to a queryset are included in the SQL statement
that is generated in the count() query, including all joins and SQL
function calls - I've personally been bitten by this with DRF and a query
with a particularly complex set of annotations where the count query took
longer than fetching the data itself, but was semantically equivalent to a
much faster, plain Model.objects.count() call.

I think in most cases annotations can be stripped, for example in the two
queries below the annotation is not filtered on so it can be safely removed
from the query without affecting the result:

one = Book.objects.annotate(Count('chapters')).count()
two = Book.objects.count()

I wanted to gather some feedback from the group as to whether this always
holds true: if an annotation is not filtered (or ordered on) it can always
be safely removed from a count query? I wouldn't be surprised if there is a
case where this isn't safe, but I cannot think of one.

I've made an initial merge request that removes annotations from querysets
that have no filter or ordering that depends on annotations here:
https://github.com/django/django/pull/8928. It seems like Django also has
all the information to optimize out annotations that are not directly or
indirectly used by filters, but I wanted to gather some feedback before I
attempted this.

Tom

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJM27OnuBSKWY8%3DAUkPa2vv75ASmL9yuohkH4heN8QUz2A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Automatic prefetching in querysets

2017-08-16 Thread Tom Forbes
Some quicker changes that have been brought up here could be implemented in
the interim, like adding intelligent warnings.

Someone brought up a great point about how the ORM hides most SQL
terminology from the user, but still requires the knowledge of the
difference between prefetch and select related.

Perhaps a good idea would be to create a generic 'related' method on the
queryset that handles both cases? Hiding the differences between how m2m
and foreign keys are fetched could make them seem less complicated and more
approachable to new users?

On 16 Aug 2017 14:04, "Marc Tamlyn"  wrote:

As an opt in feature, it does sound quite interesting. If over the course
of a few releases, it proves to be reliable and we don't get tons of
support requests of it either not working, or causing the opposite problem,
we can consider moving it to an opt out feature. This makes it easy for
people to "turn on the magic", but keeps the default safer. It would be
very easy to write a one line monkeypatch to make it default behaviour.

Django is conservative when it comes to changes to default behaviour, and
doubly so when the ORM is considered. This will need to have been used
widely in the wild before it would be considered safe as a default
behaviour for all projects. This whole process would only take a couple of
years, which isn't much time in Django release land! I don't mean to
undermine the belief of the people who think it's definitely an improvement
in most cases, I just want us to be cautious in how we explore that
hypothesis.

It's also likely to require a significant patch, and need careful analysis
before merging. I'd suggest it's a good candidate for a DEP to discuss the
motivation for the project, and to find a shepherd from the core team
(Josh? Adam?) to help it land.

On 16 August 2017 at 13:48, Luke Plant  wrote:

> Hi Josh,
>
> On 16/08/17 02:26, Josh Smeaton wrote:
>
> I believe we should be optimising for the **common** use case, not
> expecting everyone to be experts with the ORM.
>
> > If I can come up with a single example where it would significantly
> decrease performance (either memory usage or speed) compared to the default
> (and I'm sure I can), then I would be strongly opposed to it ever being
> default behaviour.
>
> The status quo is already one where thousands of users and sites are doing
> the non-optimal thing because we're choosing to be conservative and have
> users opt-in to the optimal behaviour.
>
>
> I wouldn't say it is "optimal behaviour". It is behaviour that is an
> optimization for some use cases - common ones, I'd agree - but not all. In
> fact it is not even the best optimization - it performs less well in many
> cases than a manual `select_related`.
>
> We don't know how many sites would be affected by the opposite default,
> because we aren't putting people in that situation. Generating potentially
> large queries (i.e. ones that return lots of results) is going to make
> someone's life a lot harder, and can even crash processes (out of memory
> errors), or cause massive slowdown due to insufficient memory and swapping.
> I have had these kind of issues in production systems, and sometimes the
> answer is to prefetch *less* - which is why things like `iterator()` exist,
> because sometimes you *don't* want to load it all into memory upfront.
>
> A massive complaint against Django is how easy it is for users to build in
> 1+N behaviour. Django is supposed to abstract the database away (so much so
> that we avoid SQL related terms in our queryset methods), yet one of the
> more fundamental concepts such as joins we expect users to know about and
> optimise for.
>
>
> This is far from the only place where we expect users to be conscious of
> what has to go on at the database level. For example, we expect users to
> use `.count()` and `.exists()` where appropriate, and not use them where
> not appropriate - see https://docs.djangoproject.com
> /en/1.11/topics/db/optimization/#don-t-overuse-count-and-exists
>
> This is an example of doing it right. We could have 'intelligently' made
> `__len__()` do `count()`, but this introduces queries that the user did not
> ask for, and that's hard for developers to predict. That whole section of
> the docs on DB optimisation depends on the possibility of understanding
> things at a DB level, and understanding how QuerySets behave, and that they
> only do the queries that you ask them to do. The more magic we build in,
> the harder we make life for people trying to optimize database access.
>
> If we have an `auto_prefetch` method that has to be called explicitly,
> then we are allowing users who know less about databases to get something
> that works OK for many situations. But having it on by default makes
> optimization harder and adds unwelcome surprises.
>
>
> I'd be more in favour of throwing an error on
> non-select-related-foreign-key-access than what we're currently doing
> which is 

Re: Automatic prefetching in querysets

2017-08-15 Thread Tom Forbes
Exploding query counts are definitely a pain point in Django, anything to
improve that is definitely worth considering. They have been a problem in
all Django projects I have seen.

However I think the correct solution is for developers to correctly add
select/prefetch calls. There is no general solution for automatically
applying them that works for enough cases, and i think adding such a method
to querysets would be used incorrectly and too often.

Perhaps a better solution would be for Django to detect these O(n) query
cases and display intelligent warnings, with suggestions as to the correct
select/prefetch calls to add. When debug mode is enabled we could detect
repeated foreign key referencing from the same source.

On 15 Aug 2017 19:44, "Gordon Wrigley"  wrote:

Sorry maybe I wasn't clear enough about the proposed mechanism.

Currently when you dereference a foreign key field on an object (so
'choice.question' in the examples above) if it doesn't have the value
cached from an earlier access, prefetch_related or select_related then
Django will automatically perform a db query to fetch it. After that the
value will then be cached on the object for any future dereferences.

This automatic fetching is the source the N+1 query problems and in my
experience most gross performance problems in Django apps.

The proposal essentially is to add a new queryset function that says for
the group of objects fetched by this queryset, whenever one of these
automatic foreign key queries happens on one of them instead of fetching
the foreign key for just that one use the prefetch mechanism to fetch it
for all of them.
The presumption being that the vast majority of the time when you access a
field on one object from a queryset result, probably you are going to
access the same field on many of the others as well.

The implementation I've used in production does nest across foreign keys so
something (admittedly contrived) like:
for choice in Choice.objects.all():
print(choice.question.author)
Will produce 3 queries, one for all choices, one for the questions of those
choices and one for the authors of those questions.

It's worth noting that because these are foreign keys in their "to one"
direction each of those queryset results will be at most the same size (in
rows) as the proceeding one and often (due to nulls and duplicates) smaller.

I do not propose touching reverse foreign key or many2many fields as the
generated queries could request substantially more rows from the DB than
the original query and it's not at all clear how this mechanism would
sanely interact with filtering etc. So this is purely about the forward
direction of foreign keys.

I hope that clarifies my thinking some.

Regards
G

On Tue, Aug 15, 2017 at 7:02 PM, Marc Tamlyn  wrote:

> Hi Gordon,
>
> Thanks for the suggestion.
>
> I'm not a fan of adding a layer that tries to be this clever. How would
> possible prefetches be identified? What happens when an initial loop in a
> view requires one prefetch, but a subsequent loop in a template requires
> some other prefetch? What about nested loops resulting in nested
> prefetches? Code like this is almost guaranteed to break unexpectedly in
> multiple ways. Personally, I would argue that correctly setting up and
> maintaining appropriate prefetches and selects is a necessary part of
> working with an ORM.
>
> Do you know of any other ORMs which attempt similar magical optimisations?
> How do they go about identifying the cases where it is necessary?
>
> On 15 August 2017 at 10:44, Gordon Wrigley 
> wrote:
>
>> I'd like to discuss automatic prefetching in querysets. Specifically
>> automatically doing prefetch_related where needed without the user having
>> to request it.
>>
>> For context consider these three snippets using the Question & Choice
>> models from the tutorial
>>  
>> when
>> there are 100 questions each with 5 choices for a total of 500 choices.
>>
>> Default
>> for choice in Choice.objects.all():
>> print(choice.question.question_text, ':', choice.choice_text)
>> 501 db queries, fetches 500 choice rows and 500 question rows from the DB
>>
>> Prefetch_related
>> for choice in Choice.objects.prefetch_related('question'):
>> print(choice.question.question_text, ':', choice.choice_text)
>> 2 db queries, fetches 500 choice rows and 100 question rows from the DB
>>
>> Select_related
>> for choice in Choice.objects.select_related('question'):
>> print(choice.question.question_text, ':', choice.choice_text)
>> 1 db query, fetches 500 choice rows and 500 question rows from the DB
>>
>> I've included select_related for completeness, I'm not going to propose
>> changing anything about it's use. There are places where it is the best
>> choice and in those places it will still be up to the user to request it. I
>> will note that anywhere 

Re: Django 2.0 Python version support (Python 3.6+ only?)

2017-08-08 Thread Tom Forbes
One of the biggest gains would be allowing third party packages to begin to
add type hints, if we support 3.4 this won't happen for a while at least.

Other gains, for Django and third party  packages include:
- code improvements using unpacking generalizations
- speed improvements with OrderedDict and lru_cache
- support for the Http status enumeration in stdlib
- much faster directory iteration function with scandir
- other general speed improvements (
https://docs.python.org/3/whatsnew/3.5.html#optimizations)

Apart from type hinting (which is a contentious issue) there are not any
big gains we get from 3.5 over 3.4. lots of small ones though.

On 8 Aug 2017 09:16, "Curtis Maloney"  wrote:

Is there any list of things we gain from dropping / adding any particular
version?

The older discussion mentions a tracking ticket, but it is empty.

--
C


On 8 August 2017 9:45:54 AM AEST, Tim Graham  wrote:
>
> With a little more than a month to go until the Django 2.0 alpha (targeted
> for September 18), I'd like to make a final decision about whether or not
> to keep Python 3.4 support for Django 2.0. Jenkins is currently running the
> tests on pull requests with Python 3.4 and 3.6. I've seen a few times where
> contributors first used Python 3.5+ syntax and then had to make adjustments
> for 3.4 compatibility so while it's not a large burden, it's not a
> non-trivial one.
>
> Has anyone changed their thinking in the last few months? If not, I guess
> we'll keep Python 3.4 support for Django 2.0 and drop it for 2.1.
>
> On Friday, February 17, 2017 at 9:32:20 PM UTC-5, Tim Graham wrote:
>>
>> Ok, I created a ticket to track cleanups and new Python features we can
>> use when Python 3.4 support is removed: https://code.djangoproject.com
>> /ticket/27857
>>
>> We can evaluate that a bit later in the Django 2.0 release cycle and
>> decide whether or not to keep Python 3.4 support for 1.11.
>>
>> On Wednesday, January 18, 2017 at 12:20:13 PM UTC-5, Rotund wrote:
>>>
>>> I agree that allowing more people to be able to do development against
>>> Django 2.0 is important. That stated, please be very explicit in the
>>> release notes and documentation that "Versions below Python 3.6 are
>>> expected to be dropped before the next Django LTS will be released, so
>>> please keep that in your project planning." (Language too informal, but I
>>> think the idea is correct.)
>>>
>>>
>>>
>>> On Wed, Jan 18, 2017 at 2:28 AM, Claude Paroz 
>>> wrote:
>>>
 Le mardi 17 janvier 2017 15:48:46 UTC+1, Tim Graham a écrit :
>
> I propose to tentatively target Python 3.5+ for Django 2.0 but not to
> remove the current workarounds for Python 3.4 at this time. Shortly before
> the alpha for Django 2.0, an interested person can look into how much work
> is required to fix any test failures on Python 3.4 and we'll make a
> decision then.
>

 I'm strongly advocating for keeping 3.4 support for now, as I would
 have difficulty to continue contributing to Django.
 My main system is still using 3.4 and will be for some months. Even if
 I could rather easily installing manually a more recent Python, I very much
 like relying on my stable distro packages. Sorry for my dumbness!

 Claude

 --
 You received this message because you are subscribed to the Google
 Groups "Django developers (Contributions to Django itself)" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to django-develop...@googlegroups.com.
 To post to this group, send email to django-d...@googlegroups.com.
 Visit this group at https://groups.google.com/group/django-developers.
 To view this discussion on the web visit https://groups.google.com/d/ms
 gid/django-developers/09570028-4eea-41ac-b364-93ae2c946b21%
 40googlegroups.com
 
 .

 For more options, visit https://groups.google.com/d/optout.

>>>
>>>
>>>
>>> --
>>> Joe Tennies
>>> ten...@gmail.com
>>>
>>
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/6160B82D-223A-401F-BB3C-B87965C83823%40tinbrain.net

.

For 

Re: Admin list view counting

2017-08-07 Thread Tom Forbes
Mysql, Oracle and Postgres expose estimated rows in a pretty similar manner
(selecting the table name from a database-specific table). Perhaps a more
generic 'estimate_table_rows' function could be added for all backends? I
guess for sqlite it would have to use an actual count() as a fallback. This
could cause issues if people develop on sqlite and find the function
returns accurate row counts and this changes in production.

The problem is these methods only return estimates for the entire table,
not including any filters, which narrows down the usefulness a lot. The
postgres wiki suggests 'scraping' the output of `EXPLAIN ANALYZE` which
sounds rather too hacky to add?

On Mon, Aug 7, 2017 at 9:53 PM, Shai Berger <s...@platonix.com> wrote:

> On PG we can and should do much better -- there is a way to get a very
> fast,
> though not accurate, count of records in a table:
>
> https://wiki.postgresql.org/wiki/Count_estimate
>
> I think we should expose an API along the lines of
>
> queryset.table_count(estimate_above=2000)
>
> which actually performs the COUNT(*) only if the estimate is under the
> threshold, at least where such facilities are available. We could use that
> by
> default for the admin count queries, if the queryset has no filtering or
> limiting joins. IMO, this alone would solve a whole lot of the problem; and
> allowing it to be also used explicitly (e.g. by adding Tom's idea to allow
> the
> user to specify the number of objects) would solve a whole lot of the rest
> of
> the problem.
>
> On Monday 07 August 2017 15:03:15 Tom Forbes wrote:
> > This is a great idea. A related issue I've come across with the paginator
> > is not being able to pass an explicit count into it.
> >
> > If you have a query set with expensive annotations that don't effect the
> > count in any way it's very wasteful to include them in the count SQL
> > statement (on top of general PG count slowness), which Django does. Being
> > able to pass in 'num_objects=x' would be great.
> >
> > On 7 Aug 2017 08:03, "Adam Johnson" <m...@adamj.eu> wrote:
> >
> > +1 from me, I've spent long enough on those COUNT(*) queries at work.
> >
> > Is it worth putting this logic in Paginator so applications can reuse it?
> >
> > On 7 August 2017 at 07:45, Josh Smeaton <josh.smea...@gmail.com> wrote:
> > > The admin list page performs a count(*) (twice!) when viewing the list
> > > page of a model. One of these counts can be disabled, but the
> pagination
> > > count can not be. On huge tables, this can be a massive performance
> > > issue. https://code.djangoproject.com/ticket/8408 was created to
> disable
> > > the count, but it only allowed disabling one of them. I've reopened the
> > > ticket with Matthew who has implemented the suggestion to disable
> > > pagination features that require count.
> > >
> > > PR: https://github.com/django/django/pull/8858
> > >
> > > Does anyone have strong views one way or another here? Would like to
> hear
> > > if so.
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups
> > > "Django developers (Contributions to Django itself)" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> an
> > > email to django-developers+unsubscr...@googlegroups.com.
> > > To post to this group, send email to django-developers@
> googlegroups.com.
> > > Visit this group at https://groups.google.com/group/django-developers.
> > > To view this discussion on the web visit
> https://groups.google.com/d/ms
> > > gid/django-developers/d093b4dd-4bc4-4c16-910d-53c4740ec5ea%
> > > 40googlegroups.com
> > > <https://groups.google.com/d/msgid/django-developers/
> d093b4dd-4bc4-4c16-9
> > > 10d-53c4740ec5ea%40googlegroups.com?utm_medium=email_source=footer>
> .
> > > For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJMytwWk1FYbQze1rbFTr0EnyadLBZmcpxtya4Ufst32xg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Admin list view counting

2017-08-07 Thread Tom Forbes
This is a great idea. A related issue I've come across with the paginator
is not being able to pass an explicit count into it.

If you have a query set with expensive annotations that don't effect the
count in any way it's very wasteful to include them in the count SQL
statement (on top of general PG count slowness), which Django does. Being
able to pass in 'num_objects=x' would be great.

On 7 Aug 2017 08:03, "Adam Johnson"  wrote:

+1 from me, I've spent long enough on those COUNT(*) queries at work.

Is it worth putting this logic in Paginator so applications can reuse it?

On 7 August 2017 at 07:45, Josh Smeaton  wrote:

> The admin list page performs a count(*) (twice!) when viewing the list
> page of a model. One of these counts can be disabled, but the pagination
> count can not be. On huge tables, this can be a massive performance issue.
> https://code.djangoproject.com/ticket/8408 was created to disable the
> count, but it only allowed disabling one of them. I've reopened the ticket
> with Matthew who has implemented the suggestion to disable pagination
> features that require count.
>
> PR: https://github.com/django/django/pull/8858
>
> Does anyone have strong views one way or another here? Would like to hear
> if so.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/ms
> gid/django-developers/d093b4dd-4bc4-4c16-910d-53c4740ec5ea%
> 40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/CAMyDDM1Gs0vDhgGmMNFd29sftmvAd
53qibRo7DPUpLh%3DkwkZvw%40mail.gmail.com

.

For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAFNZOJN-pAX%3DDMvzYs6e2_Kc%3Dg7JW%2BBkOTPz-eACV6AWNGph%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Future of the development server's auto-reloading

2017-07-24 Thread Tom Forbes
I was looking into refactoring the auto-reloading code and I wanted to 
gather some feedback. There is some potential to squash several bugs in one 
go[1][2][3].

To begin with we can get rid of the Jython specific code, Jython is 2.7 
only at the moment and I don't think that will change soon.

We currently have an implicit dependency on `pyinotify`, if it's installed 
a more efficient inotify-based watcher is used. I've never seen this 
advertised anywhere, and pyinotify has not received any updates in over 3 
years. There is also this[4] blog post which says perhaps pyinotify isn't 
the best. I think the best way forward is to use the 'watchdog' package if 
it's installed, as it contains platform-specific implementations for 
observing file modifications[6] and even their fallback polling observer[7] 
is better than any we could include in Django. If we want to add the 
ability to use the watchman service with Django (something I fully agree 
with) perhaps adding support for it directly to that library would be more 
preferable than bundling it in Django itself.

The current autoreloading code contains some translations specific code 
that would be good to split out. It would be nice to have the autoreload 
module be as focused as possible on just detecting and handling file 
changes in a pretty neutral way, other parts of Django could register 
subscriptions to be notified when files change and can handle it themselves 
(perhaps via signals?). For example the cached template loader could 
register a callback to fire whenever one if it's source files changes and 
delete that entry from the cache, the translations framework could handle 
.mo files changing itself or third party modules can use these hooks to do 
whatever crazy things they want.

I've got a first-draft branch with a cleanup of the autoreload code 
here[8], any feedback would be welcome. It currently doesn't handle syntax 
error reloading or the i18n resetting. Could the syntax error reloading be 
improved with a clean re-implementation of the autoreloader? I can't think 
of any nice ways to handle it.

1. https://code.djangoproject.com/ticket/27685#ticket
2. https://code.djangoproject.com/ticket/25624
3. https://code.djangoproject.com/ticket/25791
4. 
http://www.serpentine.com/blog/2008/01/04/why-you-should-not-use-pyinotify/
5. 
https://pythonhosted.org/watchdog/_modules/watchdog/observers/polling.html
6. https://pythonhosted.org/watchdog/api.html#module-watchdog.observers
7. 
https://pythonhosted.org/watchdog/_modules/watchdog/observers/polling.html
8. 
https://github.com/orf/django/blob/27685-autoreloader-refactor/django/utils/autoreload.py

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b33d73e3-6ff5-431d-b9ff-7db1219d11cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: #28398: Allow management command invocation to suggest for incomplete commands?

2017-07-17 Thread Tom Forbes
Vlada:
I think this is a great idea for improving the usability of manage.py,
especially for newcomers. When I looked your current implementation used a
simple 'in' to find suggestions, but this is not great for the most
obvious/common use case: typos.

I would strongly advocate for using the levenshtein distance algorithm if
this does get merged, there are some simple and succinct python
implementations we can use, and the algorithm itself is what git uses to
suggest commands (and I've always found that quite good).

Brice:
You are right in that this is a case that could happen in part due to this
feature, but it's a long shot and IMO the benefits outweigh the risks. This
can happen without this feature anyway - if someone is typing in random
manage.py commands without thinking then issues can and will arise. The
only preventative measure we could take would be to add an "are you sure?"
prompt after every manage.py invocation which is not a great idea overall.

For newcomers though this could be great, seeing the 'command not found'
message can be confusing and unhelpful, any help we can add to that is a
good thing IMO. Adding stars to the command invocation can be even more
confusing as users have to escape them (and if they don't they could end up
running random commands) and it's not terribly nice UI wise.

Built in shell autocompletion would be the best way forward in this case,
at least for non-windows users. Kubectl has a nice way of doing this:
source <(kubectl completion bash|zsh). Maybe something like this could be
adapted for manage.py?


On 17 Jul 2017 08:49, "Brice PARENT"  wrote:

Hi!

I'm not sure how I feel about that. It feels like a good idea at first, but
it might lead to dangerous behaviours.

Let me explain my thought: having such a feature would encourage people to
use it (of course). Doing so can lead so side effects. For example,  if, in
a project you're working on, you want to use a custom management command
named "migrate_data_to_other_server", you might end up typing "./manage.oy
migrate" in hope for the system to display the exact name that you probably
have forgotten. But it won't, it will migrate your database instead. What I
mean is that executing commands that shouldn't work on purpose might lead
to executing the wrong command instead. And management commands might be
dangerous if not used at the right time (I've seen management commands
being used to push code to production for example. Executing them by error
in a dev environment might be a real issue!).

I'd prefer to encourage the use of "./manage.py help", which lists all
available commands, or use masks when searching for a command ("./manage.py
migrate*"). When you want to look for the name of a command, you'd know
that adding a star somewhere won't execute anything other than listing
available commands matching the value you just gave. And every developer
knows (I think) how to use wildcards.

-Brice

Le 15/07/17 à 18:36, Vláďa Macek a écrit :

Hi everyone,

I had an idea that would save me time working with Django:

The manage.py wouldn't only print "Unknown command" when incomplete
subcommand name given, but also print those available by substring search.

https://code.djangoproject.com/ticket/28398

I added a screenshot and a patch there.

In my opinion, such first implementation could be as simple as that.
Smarter versions may come later.

As suggested by Tim Graham (thanks, Tim), I resort here for opinions.

Thank you,

Vlada Macek

-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/5abdab10-e9e2-4dbd-81c7-ffd492781977%40googlegroups.
com

.
For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups
"Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/
msgid/django-developers/d67e85bc-c117-d527-d4f3-791782c15acd%40brice.xyz

.

For more options, visit 

  1   2   >