Re: Easy Pickings: support robust on_commit handlers

[Josh Smeaton]

Apologies for some reason I didn't get any replies to my inbox! If you'd 
like to tag me on any WIP PRs I'm willing to lend a hand and test.

On Thursday, 7 April 2022 at 22:19:44 UTC+10 revan...@almabase.com wrote:

>
> Hi Josh,
>
> I am also interested to work on this ticket. I am also aware of how can I 
> execute this. It would be great if I could contribute as I am eagerly 
> waiting to have my first contribution to Django. However, I would also be 
> happy to help anyone who is willing to contribute.
>
> Thanks,
> Revanth
> On Wednesday, April 6, 2022 at 11:08:04 PM UTC+5:30 omanir...@gmail.com 
> wrote:
>
>> Hi Josh, all 
>>
>> I am interested to work on this ticket, this would be my first time 
>> contributing. 
>> I have read and worked through the article "Writing your first patch for 
>> Django".   I should be fine with the getting started part. 
>>
>> I am looking for someone  reach out to personally to get my thoughts 
>> validated/ get help. 
>>
>> thanks
>> On Monday, 4 April, 2022 at 10:20:45 am UTC+5:30 josh.s...@gmail.com 
>> wrote:
>>
>>> I've just created a ticket for supporting robust on_commit handlers that 
>>> I think might be a good first ticket for somebody looking to get into 
>>> contributing to Django.
>>>
>>> https://code.djangoproject.com/ticket/33616
>>>
>>> I'd be more than happy to review any patch and provide guidance if 
>>> required. It's been quite a while since I've contributed directly so I may 
>>> not be too helpful in the "getting started" phase, but the existing 
>>> documentation would be a good place to begin 
>>> https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ee292903-3e00-4734-8594-ab08444dc336n%40googlegroups.com.

Easy Pickings: support robust on_commit handlers

[Josh Smeaton]

I've just created a ticket for supporting robust on_commit handlers that I 
think might be a good first ticket for somebody looking to get into 
contributing to Django.

https://code.djangoproject.com/ticket/33616

I'd be more than happy to review any patch and provide guidance if 
required. It's been quite a while since I've contributed directly so I may 
not be too helpful in the "getting started" phase, but the existing 
documentation would be a good place to begin 
https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d30f6430-245b-4b97-8191-9ada3ee6e2aan%40googlegroups.com.

Re: [TB] Proposal to add Florian Apolloner to Releasers team.

[Josh Smeaton]

Sorry to hijack, but maybe something to think about next time - but 
considering the 2 fellows cover the release process quite well already, 
would it be worth considering a "junior" releaser as a third? Someone to be 
mentored via pairing that would help encourage either new contributors OR 
keep existing ones engaged? 

On Thursday, 22 July 2021 at 23:42:21 UTC+10 charettes wrote:

> I'm also +1
>
> Le mercredi 21 juillet 2021 à 17:21:03 UTC-4, Adam Johnson a écrit :
>
>> +1 I’m in favour of giving Florian the access and necessary training, 
>> should he want it.
>>
>> On Wed, 21 Jul 2021 at 10:18, Carlton Gibson  
>> wrote:
>>
>>> Hello Technical Board. 
>>>
>>> Mariusz is working on updating the Organisation docs to reflect the DEP 
>>> 10 governance changes. 
>>>
>>> https://github.com/django/django/pull/14670
>>>
>>> Whilst looking at that we realise that we're short an active official 
>>> Releaser. 
>>>
>>> > The technical board select Releasers as necessary to maintain their 
>>> number
>>> > at a minimum of three, in order to spread the workload and avoid 
>>> over-burdening
>>> > or burning out any individual Releaser. There is no upper limit to the 
>>> number
>>> > of Releasers.
>>>
>>> Likely a third-releaser is not needed, with Mariusz and I alternating, 
>>> and covering each other, but it's a backup — we should have a third active 
>>> person in the role. 
>>>
>>> James Bennet has served as the third releaser since the DEP 10 changes, 
>>> but hasn't been active in the last period. Whether he wishes to continue or 
>>> step-back, I don't feel it's in line with the spirit of the above paragraph 
>>> to have the responsibility fall on James alone at the current time. 
>>>
>>> The three most plausible candidates for releaser are James Bennet, Tim 
>>> Graham, and Florian Apolloner. Of these Florian is currently most active, 
>>> and in most regular communication with us. 
>>>
>>> As such, I would like to nominate Florian as a new Releaser, and ask the 
>>> TB to discuss/vote on that in accordance with DEP 10. 
>>>
>>> (I didn't discuss this with him but, I hope that's OK with you Florian?) 
>>>
>>> As I say, it's unlikely to be needed, but came up in reviewing Mariusz 
>>> changes. 
>>>
>>> Kind Regards,
>>>
>>> Carlton
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/3c78de95-2d7d-4976-a64e-daf07acc1689n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/18f83227-9487-4b77-9ab4-484a1d6a3711n%40googlegroups.com.

Re: Proposal to add attribute 'step' to FloatField and DecimalField

[Josh Smeaton]

Just to clarify - you're talking about form fields rather than model 
fields, right? I can see the argument for form fields but not for model 
fields.

Is there a better API we can think of for customising widgets from the 
field constructor that could then be passed through? As a rough example:

my_step_field = forms.FloatField(widget_attributes={"step": 0.5})

Rather than encode each kind of attribute in the form field constructor it 
could take a dictionary that is passed along to customise the widget. Often 
I've found it 
annoying to track down the exact widget required, find the import, and then 
customise the widget. The specific arguments to fill out the attributes are 
less onerous.

Thoughts?

On Wednesday, 17 March 2021 at 18:54:29 UTC+11 jacob...@gmail.com wrote:

> On Wednesday, March 17, 2021 at 12:49:48 AM UTC+1 
> in...@markusholtermann.eu wrote:
>
>> That sounds like a sensible feature. Do you want to open a ticket and 
>> maybe implement it?
>>
>
> Hi Markus,
> ticket #32559  has been 
> issued to propose this feature.
> If accepted I will assign myself to implement it.
>
> – Jacob
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/67a32579-a119-4d54-b765-0e2dad850771n%40googlegroups.com.

Re: Idea for Rollback Migration Rules

[Josh Smeaton]

I run into the problem you're trying to solve often and it's quite 
frustrating. I don't think storing the migration text in the database is 
the right solution - I don't know why, but it just *feels* wrong.

Such a rollback would only support SQL based migrations - any RunPython 
migration operations would not be safe to run as the code has changed 
underneath.

On Friday, 19 June 2020 23:15:31 UTC+10, Alexander Lyabah wrote:
>
> I just have a very small idea, maybe I can get a feedback about it here.
>
> What if db will store not only migration name but also migration itself, 
> so the system can rollback the migration even if migration file is 
> no-longer exists.
>
>
> *Cases when it can be useful:*
>
> when I switch between branched in my git-repo I want to roll-back 
> migrations that were removed and apply new migrations.
>
> When I change migration itself, I want to roll-back its previous version 
> and apply a new changed version.
>
>
> Let me know what you think about it, I hope it is a right place to share 
> ideas like that.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/babfcb07-6a7f-4566-9d43-88551b947b0bo%40googlegroups.com.

Re: Overriding template blocks without copy/pasting entire template files

[Josh Smeaton]

Upon reflection, would you agree that this should be documented a little 
better? The release notes at the time had a vague deprecation note about `
supports_recursion` being removed, and the main docs 
https://docs.djangoproject.com/en/3.0/ref/templates/builtins/#std:templatetag-extends
 have 
no such explanation about recursive inheritance either.

I pointed out this thread to a few people in my team, and none of them were 
aware of this fix/feature, and they were all using 3rd party packages for 
support.

Funnily enough, 
https://docs.djangoproject.com/en/3.0/ref/contrib/admin/#overriding-vs-replacing-an-admin-template
 seems 
to suggest this was always possible, so I'm quite confused.
 

On Tuesday, 16 June 2020 15:54:11 UTC+10, Josh Smeaton wrote:
>
> Changed 5 years ago 
> <https://code.djangoproject.com/timeline?from=2015-04-22T17%3A28%3A19-05%3A00=second>
>  by Preston Timmons
>
> 臘‍♀️
>
> Thanks Carlton - I guess it has *only* been 5 years though :)
>
> On Tuesday, 16 June 2020 15:41:29 UTC+10, Carlton Gibson wrote:
>>
>> Hi Josh. 
>>
>> This was added in Django 1.9 here 
>> https://github.com/django/django/commit/fc2147152637e21bc73f991b50fa06254af02739
>>
>> It leverages the extends tag, re-using the same template name: 
>>
>> # myapp/templates/admin/base.html
>> {% extends "admin/base.html" %}
>> {% block footer %}this site is restricted, blah blah legal text blah
>> {% endblock %}
>>
>> So as long as myapp/templates are loaded before contrib.admin's then it 
>> you should see the result you're after. 
>>
>> #15053 <https://code.djangoproject.com/ticket/15053> references 
>> django-overextends as the influence. 
>>
>>
>> Kind Regards,
>>
>> Carlton
>>
>>
>>
>> On Tuesday, 16 June 2020 07:03:53 UTC+2, Josh Smeaton wrote:
>>>
>>> Something that has bugged me for awhile is the requirement to copy and 
>>> paste an entire template when you would just like to override a single 
>>> block. This arises mostly when overriding admin templates, like 
>>> `admin/base.html`.
>>>
>>> In my ideal world, I'd be able to do something like this:
>>>
>>> # myapp/templates/admin/base.html
>>> {% override "admin/base.html" %}
>>> {% block footer %}this site is restricted, blah blah legal text blah
>>> {% endblock %}
>>>
>>>
>>> And then the template loading system would find the next 
>>> `admin/base.html` in the chain and use my overrides.
>>>
>>> There is prior art too. https://pypi.org/project/django-apptemplates/ 
>>> allows you to override a template from a specific app using this syntax:
>>>
>>> # myapp/templates/admin/base.html
>>> {% extends "admin:admin/base.html" %}
>>> {% block footer %}this site is restricted, blah blah legal text blah
>>> {% endblock %}
>>>
>>>
>>> I think this kind of functionality should be included within Django 
>>> itself. If others agree, should there be a new name such as override, or 
>>> would overloading extends be good enough?
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/95e2e0b8-f781-4274-b202-ce28f3571ee0o%40googlegroups.com.

Re: Overriding template blocks without copy/pasting entire template files

[Josh Smeaton]

Changed 5 years ago 
<https://code.djangoproject.com/timeline?from=2015-04-22T17%3A28%3A19-05%3A00=second>
 by Preston Timmons

臘‍♀️

Thanks Carlton - I guess it has *only* been 5 years though :)

On Tuesday, 16 June 2020 15:41:29 UTC+10, Carlton Gibson wrote:
>
> Hi Josh. 
>
> This was added in Django 1.9 here 
> https://github.com/django/django/commit/fc2147152637e21bc73f991b50fa06254af02739
>
> It leverages the extends tag, re-using the same template name: 
>
> # myapp/templates/admin/base.html
> {% extends "admin/base.html" %}
> {% block footer %}this site is restricted, blah blah legal text blah{% 
> endblock %}
>
> So as long as myapp/templates are loaded before contrib.admin's then it 
> you should see the result you're after. 
>
> #15053 <https://code.djangoproject.com/ticket/15053> references 
> django-overextends as the influence. 
>
>
> Kind Regards,
>
> Carlton
>
>
>
> On Tuesday, 16 June 2020 07:03:53 UTC+2, Josh Smeaton wrote:
>>
>> Something that has bugged me for awhile is the requirement to copy and 
>> paste an entire template when you would just like to override a single 
>> block. This arises mostly when overriding admin templates, like 
>> `admin/base.html`.
>>
>> In my ideal world, I'd be able to do something like this:
>>
>> # myapp/templates/admin/base.html
>> {% override "admin/base.html" %}
>> {% block footer %}this site is restricted, blah blah legal text blah
>> {% endblock %}
>>
>>
>> And then the template loading system would find the next 
>> `admin/base.html` in the chain and use my overrides.
>>
>> There is prior art too. https://pypi.org/project/django-apptemplates/ 
>> allows you to override a template from a specific app using this syntax:
>>
>> # myapp/templates/admin/base.html
>> {% extends "admin:admin/base.html" %}
>> {% block footer %}this site is restricted, blah blah legal text blah
>> {% endblock %}
>>
>>
>> I think this kind of functionality should be included within Django 
>> itself. If others agree, should there be a new name such as override, or 
>> would overloading extends be good enough?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3ed867b6-0584-4e37-9099-1b60c1b412edo%40googlegroups.com.

Overriding template blocks without copy/pasting entire template files

[Josh Smeaton]

Something that has bugged me for awhile is the requirement to copy and 
paste an entire template when you would just like to override a single 
block. This arises mostly when overriding admin templates, like 
`admin/base.html`.

In my ideal world, I'd be able to do something like this:

# myapp/templates/admin/base.html
{% override "admin/base.html" %}
{% block footer %}this site is restricted, blah blah legal text blah{% 
endblock %}


And then the template loading system would find the next `admin/base.html` 
in the chain and use my overrides.

There is prior art too. https://pypi.org/project/django-apptemplates/ 
allows you to override a template from a specific app using this syntax:

# myapp/templates/admin/base.html
{% extends "admin:admin/base.html" %}
{% block footer %}this site is restricted, blah blah legal text blah{%
 endblock %}


I think this kind of functionality should be included within Django itself. 
If others agree, should there be a new name such as override, or would 
overloading extends be good enough?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6cf43971-1ea9-4b93-9a35-aaab5908327co%40googlegroups.com.

Re: Consider renaming `mark_safe` to `dangerously_trust_html` (etc)

[Josh Smeaton]

I agree that "dangerously" doesn't and shouldn't be in the name - it's 
unnecessarily verbose.

I think a name like "no_escape" ,or similar, conveys the meaning properly 
and is no more verbose than mark_safe.


On Thursday, 20 February 2020 10:16:12 UTC+11, Kye Russell wrote:
>
> Personally I think that the current name does a fairly good job of 
> conveying the effect, and this is coming from someone who doesn’t 
> explicitly understand the internal process and is merely inferring it. 
>
> However my anecdote doesn’t negate what we all see around the internet, so 
> I suppose it can be improved. 
>
> I am a strong -1 on adding words like “dangerously” to the method name. I 
> feel that method names are not the place for warnings like that. I do not 
> like the precedent that it sets for other potentially unsafe methods within 
> the Django codebase, and i believe that it’d result in unnecessarily 
> verbose project code. 
>
> Kye Russell 
> Sent from my iPhone 
>
> > On 19 Feb 2020, at 5:11 pm, Adam Johnson > 
> wrote: 
> > 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e50b75f6-f606-42ed-a2bf-9fd1c1d6afb9%40googlegroups.com.

Re: Consider renaming `mark_safe` to `dangerously_trust_html` (etc)

[Josh Smeaton]

My thoughts:

Use and promote a different name that does not make it sound like Django is
making the string safe for you.

Alias the new name to the old names to prevent churn.

Remove all references of the old name from the docs.

What the name should be, meh.

On Thu, 20 Feb 2020 at 03:56, Adam Johnson  wrote:

> Sorry I was a bit keen after a reminder of the thread by Josh Smeaton on
> reddit. I skimmed again and it felt like there was consensus on some kind
> of rename, and it had moved to decision on the name. I guess I'm also sick
> of fixing this in client projects :)
>
> I think your concerns are fair and well-reasoned as always Carlton. You've
> convinced me this isn't worth it.
>
> I'd love to see a bleach "battery" too. Was not aware of html5lib going
> unmaintained - it does highlight that there's a lot to do elsewhere in the
> ecosystem.
>
> On Wed, 19 Feb 2020 at 14:20, Carlton Gibson 
> wrote:
>
>> I've just closed the ticket as wontfix, because I'm not seeing a
>> consensus for the change here. I'm seeing a few Yeahs and a few Mehs.
>> That doesn't mean we can't have it, but the procedure is generally agree
>> here before a ticket.
>> So can I ask, those wanting this, to make the case here, then we can
>> re-open the ticket if there is a general agreement.
>>
>> 
>> FWIW I'm not convinced. The warnings on the mark_safe() docs are pretty
>> clear cut.
>> I think users will just use whatever gets their content on the screen,
>> probably knowing it's not safe, but telling themselves
>> that they'll come back later. Change the names and that will still be the
>> same.
>> There'll just have been a massive load of busywork in between.
>>
>> Ideally we'd just ship Bleach  That's a battery we could include.
>> But we can't really do that, because html5lib is unmaintained, and
>> there's no alternative.
>> Whatever effort there is to be spent on this, I'd rather see it spent
>> there.
>> 
>>
>> Kind Regards,
>>
>> Carlton
>>
>>
>> On Wednesday, 19 February 2020 10:11:11 UTC+1, Adam Johnson wrote:
>>>
>>> I made a ticket for this: https://code.djangoproject.com/ticket/31287
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/aa60f3b6-0ace-45fc-9698-6b54e317e091%40googlegroups.com
>> <https://groups.google.com/d/msgid/django-developers/aa60f3b6-0ace-45fc-9698-6b54e317e091%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>
>
> --
> Adam
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/AvgxWR-0VrE/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAMyDDM091w46Pt%3DkiAbU5qgTxOj-_BE5hqz7qZeNNBGkG-RX_A%40mail.gmail.com
> <https://groups.google.com/d/msgid/django-developers/CAMyDDM091w46Pt%3DkiAbU5qgTxOj-_BE5hqz7qZeNNBGkG-RX_A%40mail.gmail.com?utm_medium=email_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAPbDM0f3NKg2YybMu8jjFLeW4KWXcN1ptGqXeSrCSCqH7jr-0g%40mail.gmail.com.

Re: Django 3.0 Release Notes - ASGI

[Josh Smeaton]

Sorry to resurrect this thread, and to ask and not do, but there have been 
a few questions on this ML about async support, and I imagine there'd be a 
few over on django-users too.

I've created https://code.djangoproject.com/ticket/31182#ticket to track 
this. 

On Tuesday, 15 October 2019 15:15:46 UTC+11, Andrew Godwin wrote:
>
> I agree - we need to communicate that ASGI support does *not *mean you 
> can start writing async def views. I think we should put a big disclaimer 
> to that effect next to it in the release notes and say it should be coming 
> next release.
>
> Andrew
>
> On Mon, Oct 14, 2019 at 5:45 PM Josh Smeaton  > wrote:
>
>> A co-worker just linked me to 
>> https://docs.djangoproject.com/en/dev/releases/3.0/#asgi-support and 
>> asked me (basically) if we can start doing all kinds of async work in one 
>> of our projects. Unfortunately, I didn't really know how to answer.
>>
>> Preface: I haven't followed the ASGI plan very closely (I read the DEP 
>> and have a vague understanding of the vision).
>>
>> It's my understanding that there's only very limited support for ASGI, 
>> and most features of Django won't work properly when running under ASGI. 
>> But that's not clear from reading the release notes or the deploy on ASGI 
>> section of the docs. Should we have a section in the docs that show what is 
>> and is not supported, along with examples?
>>
>> It'd be good to have a spot in the docs to point to that shows what is in 
>> and out of scope for each milestone.
>>
>> Thoughts?
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-d...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/9b11ecd8-997f-4edc-a627-5523da611a55%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/django-developers/9b11ecd8-997f-4edc-a627-5523da611a55%40googlegroups.com?utm_medium=email_source=footer>
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a79be729-972d-4511-949b-64b3e774d5fc%40googlegroups.com.

Re: django-admin startproject settings.py has some security holes

[Josh Smeaton]

A quick idea from the top of my head, is to change the assignment of 
SECRET_KEY in the generated settings to something like:

SECRET_KEY = os.environ.get("DJANGO_SECRET_KEY", 
"insecure-")

It signals that secrets in the environment are a good idea, that the 
default generated value is insecure, and it still has a random part so that 
default sites aren't automatically hackable when deployed. There's no 
impact to people just getting started.

We could go a small step forward and use `check --deploy` to check for the 
substring `insecure` (even though I believe the KEY is technically bytes?).

Just throwing something out there.


On Monday, 21 October 2019 23:48:59 UTC+11, Taymon A. Beal wrote:
>
> Is the requirement here to avoid introduce additional barriers to getting 
> up and running in local development, or to deploying a site so that it's 
> accessible from the public internet?
>
> Both of these are important goals, but trading off security against the 
> latter worries me. I don't think we're doing beginners any favors if we 
> make it easier for them to deploy sites with security issues, especially 
> since they won't be in a good position to appreciate the consequences. 
> Ideally we'd make it easy for beginners to deploy sites without security 
> issues, but that's a hard problem given the diversity of production 
> environments; in the meantime, I think we need to accept the reality that 
> figuring out how to store secrets *is* a prerequisite to deploying Django 
> in production, notwithstanding how much we wish it weren't.
>
> I'd be interested in trying to contribute a solution more secure than the 
> status quo without introducing more barriers to local development, if it 
> would have a chance of being accepted.
>
> Taymon
>
> On Friday, October 11, 2019 at 8:00:59 AM UTC-7, Carlton Gibson wrote:
>>
>> It's just scope:
>>
>>* Not clear we need to _replace_ the space for books, and blog posts, 
>> and so on, in the main docs. 
>>
>> and bandwidth:
>>
>>* These things are difficult to get right, and it needs someone to do 
>> them. (PRs always warmly received!)
>>
>> On balance, I have to say, I think the default project template does very 
>> well. 
>> Taking a beginner, say, and adding, "As well as the million things you're 
>> already dealing with, there are these things called environment variable 
>> and..." is a step I'd be very cautious about taking. 
>>
>> Yes, granted, for professional deployment, you might want different — but 
>> we have to serve everyone. 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3443dbe6-1a6a-45af-b5ec-08cf78426869%40googlegroups.com.

Django 3.0 Release Notes - ASGI

[Josh Smeaton]

A co-worker just linked me to 
https://docs.djangoproject.com/en/dev/releases/3.0/#asgi-support and asked 
me (basically) if we can start doing all kinds of async work in one of our 
projects. Unfortunately, I didn't really know how to answer.

Preface: I haven't followed the ASGI plan very closely (I read the DEP and 
have a vague understanding of the vision).

It's my understanding that there's only very limited support for ASGI, and 
most features of Django won't work properly when running under ASGI. But 
that's not clear from reading the release notes or the deploy on ASGI 
section of the docs. Should we have a section in the docs that show what is 
and is not supported, along with examples?

It'd be good to have a spot in the docs to point to that shows what is in 
and out of scope for each milestone.

Thoughts?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/9b11ecd8-997f-4edc-a627-5523da611a55%40googlegroups.com.

Re: Explore integrating django-docker-box in some way?

[Josh Smeaton]

I was going to archive the repo, but it seems I don't have the necessary 
permissions. Carlton, do you?

On Tuesday, 8 October 2019 21:45:22 UTC+11, Adam Johnson wrote:
>
> +1 to archiving django-box
>
> On Tue, 8 Oct 2019 at 11:01, Tom Forbes > 
> wrote:
>
>> Thank you for the kind words Bruno! I'm glad it's helped you, if you have 
>> any suggestions for improvements then please open an issue on the repo or 
>> post a message here, I know it's not perfect. I would have replied earlier 
>> but this message didn't get delivered to me.
>>
>> I'm biased, but I'd be +1 on archiving the old django-box. It's served us 
>> well, but unless someone is willing to spend some time updating it then 
>> it's going to confuse new users.
>>
>> On Friday, 4 October 2019 12:40:02 UTC+1, Bruno Alla wrote:
>>>
>>> Just a note that as a newish contributor to Django, this 
>>> django-docker-box is fantastic, it makes things much easier to setup. 
>>>
>>> Thank you to everyone involved!
>>>
>>> On Wednesday, 5 December 2018 00:02:27 UTC, Tom Forbes wrote:
>>>>
>>>> To have this completely working at sprints without having everyone 
>>>> building their own local images we would need to have the Jenkins server 
>>>> use docker in some capacity. This would also require an official django 
>>>> account on Docker hub.
>>>>
>>>> The pattern I’m using right now is that on every build we pull the 
>>>> django-ci:latest image (from my personal account). Docker uses this 
>>>> image as a cache automatically (preventing rebuilds). On any successful 
>>>> master build we push the new image to docker-hub, so subsequent builds can 
>>>> utilise it.
>>>>
>>>> Then anyone wanting to speed up their bootstrapping can do docker-compose 
>>>> pull and automatically have the latest image available for running 
>>>> right away. We can make this smaller, for sure, but we can also suggest 
>>>> people download this beforehand (i.e at their hotel).
>>>>
>>>> I don’t know how feasible this is but it’s also very easy to run a 
>>>> caching docker mirror (docker run -p 5000:5000 registry). Organizers 
>>>> could run this at large events and configuring docker to use a local 
>>>> mirror 
>>>> on the network is a one-line change for atendees.
>>>>
>>>>
>>>>
>>>>
>>>> On 4 December 2018 at 23:52:42, Josh Smeaton (josh@gmail.com) 
>>>> wrote:
>>>>
>>>> Size of the image could definitely be a concern, especially at sprints 
>>>> where wifi speeds aren't always optimal. The django-box image is 
>>>> significantly larger though so it'd still be a net win. There are also 
>>>> optimisations that can be made to the image for reducing size over time, 
>>>> so 
>>>> I'd fully expect it to come down. I've spent a little bit of time trying 
>>>> to 
>>>> optimise a $work$ python docker file, I'll provide what I've got as an 
>>>> issue to possibly look at. 
>>>>
>>>> I see that the ticket has been accepted and I think that's a great step 
>>>> forward. I'd also like to hear from the infrastructure team what their 
>>>> thoughts on using docker over customised build environments would be.
>>>>
>>>> Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, 
>>>> this list of names is from memory).
>>>>
>>>> On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote: 
>>>>>
>>>>> Thank you for the reply Josh. I didn’t anticipate any suggestions for 
>>>>> including this inside core but off the back of your suggestion I’ve made 
>>>>> a 
>>>>> ticket here: https://code.djangoproject.com/ticket/30010.
>>>>>
>>>>> I don’t think it should be complex at all to include this inside 
>>>>> Django - it’s four or five new files at most. Hopefully this should 
>>>>> improve 
>>>>> the experience at sprints, however the current Dockerfile weighs in at 
>>>>> 650+mb so the problem may switch from ‘it is hard to set up an 
>>>>> environment’ 
>>>>> to ‘it is hard to download one’!
>>>>>
>>>>>
>>>>>
>>>>> On 5 November 2018 at 23:02:30, Josh Smeaton (josh.s...@gmail.com) 
>>>>> wrote:
>>>>

Re: Explore integrating django-docker-box in some way?

[Josh Smeaton]

+1

On that note, how do people feel about archiving/deprecating 
https://github.com/django/django-box and updating the readme to point to 
the docker box project?

The last official version it supports is Django 2.1, but it was already on 
life support at that time, since half of the people who tried to use it 
couldn't.

On Friday, 4 October 2019 21:40:02 UTC+10, Bruno Alla wrote:
>
> Just a note that as a newish contributor to Django, this django-docker-box 
> is fantastic, it makes things much easier to setup. 
>
> Thank you to everyone involved!
>
> On Wednesday, 5 December 2018 00:02:27 UTC, Tom Forbes wrote:
>>
>> To have this completely working at sprints without having everyone 
>> building their own local images we would need to have the Jenkins server 
>> use docker in some capacity. This would also require an official django 
>> account on Docker hub.
>>
>> The pattern I’m using right now is that on every build we pull the 
>> django-ci:latest image (from my personal account). Docker uses this 
>> image as a cache automatically (preventing rebuilds). On any successful 
>> master build we push the new image to docker-hub, so subsequent builds can 
>> utilise it.
>>
>> Then anyone wanting to speed up their bootstrapping can do docker-compose 
>> pull and automatically have the latest image available for running right 
>> away. We can make this smaller, for sure, but we can also suggest people 
>> download this beforehand (i.e at their hotel).
>>
>> I don’t know how feasible this is but it’s also very easy to run a 
>> caching docker mirror (docker run -p 5000:5000 registry). Organizers 
>> could run this at large events and configuring docker to use a local mirror 
>> on the network is a one-line change for atendees.
>>
>>
>>
>>
>> On 4 December 2018 at 23:52:42, Josh Smeaton (josh@gmail.com) wrote:
>>
>> Size of the image could definitely be a concern, especially at sprints 
>> where wifi speeds aren't always optimal. The django-box image is 
>> significantly larger though so it'd still be a net win. There are also 
>> optimisations that can be made to the image for reducing size over time, so 
>> I'd fully expect it to come down. I've spent a little bit of time trying to 
>> optimise a $work$ python docker file, I'll provide what I've got as an 
>> issue to possibly look at. 
>>
>> I see that the ticket has been accepted and I think that's a great step 
>> forward. I'd also like to hear from the infrastructure team what their 
>> thoughts on using docker over customised build environments would be.
>>
>> Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this 
>> list of names is from memory).
>>
>> On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote: 
>>>
>>> Thank you for the reply Josh. I didn’t anticipate any suggestions for 
>>> including this inside core but off the back of your suggestion I’ve made a 
>>> ticket here: https://code.djangoproject.com/ticket/30010.
>>>
>>> I don’t think it should be complex at all to include this inside Django 
>>> - it’s four or five new files at most. Hopefully this should improve the 
>>> experience at sprints, however the current Dockerfile weighs in at 650+mb 
>>> so the problem may switch from ‘it is hard to set up an environment’ to ‘it 
>>> is hard to download one’!
>>>
>>>
>>>
>>> On 5 November 2018 at 23:02:30, Josh Smeaton (josh.s...@gmail.com) 
>>> wrote:
>>>
>>> I'm sorry I haven't had the time to review or contribute yet, but I 
>>> think it'll be a very useful project - especially for new contributors that 
>>> might have a little docker experience. The current vagrant solution is 
>>> heavy, does not work properly on windows and some linuxes, and isn't that 
>>> easy to maintain or deploy. I'd be in favour of adding the docker files 
>>> directly to django/django to minimise setup burden (DJANGO_PATH), and 
>>> improving the contributing docs to show users how to test using docker. 
>>>
>>> One of the hardest things I found at sprints was getting development 
>>> environments setup to effectively contribute - even using the docker-box 
>>> project which I understand quite well. Anything we can do to improve that 
>>> situation will be very beneficial.
>>>
>>> I have fewer opinions about the official CI story, hopefully some of the 
>>> infrastructure team can comment more on that. I think that replacing the 
>>> ansible roles with a docker setup ca

Re: Django LTS support time

[Josh Smeaton]

I don't think the DSF has the capacity or the will to run a business 
offering paid support contracts.

But nothing is stopping an enterprising individual or company from doing 
so. All security patches are made public (eventually) and backporting fixes 
would be fairly low effort. https://railslts.com/ is prior art in this 
space.

I'm incredibly surprised no one has started up a business to do this for 
Django.

On Tuesday, 13 August 2019 10:32:21 UTC+10, 1337 Shadow Hacker wrote:
>
> Actually I'm pretty sure it could be done even if DSF kept a profit, to 
> re-inject it into other developments for exemple. AFAIK the major 
> difference between non-profit and company is that you don't own it and as 
> such you cannot take dividends out of it personally. IMHO everybody would 
> benefit if DSF did more commerce and was stronger by that mean. I could 
> sing an ode to commerce but that would be much off-topic.
> *אורי* if you're looking for commercial extension you can certainly hire 
> someone for that for the time being, Patryk's company seems to have willing 
> developers, or hire someone to face the debt in your codebase, which 
> Patryk's company will probably recommend for the same budget. Sorry for 
> being so binary but unless another idea emerges from this topic...  it 
> seems pretty cornered at this point, sorry for not being more helpful.
>
> Best of luck
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ce923775-e684-43ad-ae16-e6db236e04d7%40googlegroups.com.

Re: Creating a new "Triage & Review Team"

[Josh Smeaton]

I think that's sensible, including making the team public.

> This struck me as a little absurd: if we weren't going to dissolve it, 
Nick should definitely be part of the old "Django Core".

It's not done yet, and while we're in this limbo stage, we're potentially 
missing out on some good maintainers before a decision has even been made. 
You can still put forward names for the core team if you think they'd make 
good members,  though I'd understand your hesitation.

On Tuesday, 13 August 2019 05:12:51 UTC+10, Carlton Gibson wrote:
>
> Oh, one more thing. I'd like to make this a "Public" team. I think there 
> should be some visibility and recognition for the folks who work so hard on 
> keeping Django strong. 
>
> As part of that I'd add it to the Teams list on djangoproject.com, and 
> maintain a list of previous members, for those who had stepped down. 
>
> C. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8b5b205e-56ca-44dc-bfd8-8a22de90be60%40googlegroups.com.

Re: Make Development More Accessible

[Josh Smeaton]

Mariatta has put together a some PEPs for migrating CPython issues over to 
GitHub.

https://www.python.org/dev/peps/pep-0581/ proposing the migration.
https://www.python.org/dev/peps/pep-0588/ migration plan.

Django and Cpython are not the same, so there'll be substantial 
differences. But it's worth familiarising oneself with prior art.

For what it's worth I'd strongly support such a move just for the increase 
in engagement.

Carlton, Mariusz, how would you gauge the level of triage activity in Trac 
from non-core members? High/Medium/Low?

https://github.blog/changelog/2019-05-23-triage-and-maintain-roles-beta/ 
describes 
the new triage and maintain roles, but they're still to be granted to 
trusted individuals (which would be an excellent gateway into full core 
membership if that is the direction Django is going to continue in).

On Wednesday, 7 August 2019 17:46:18 UTC+10, John Gooding wrote:
>
> Hi Aymeric,
>
> You bring up a lot of good points. There will undoubtedly be challenges 
> and huge amount of work in moving to a new system, or implementing any big 
> sweeping changes, however, I truly honestly believe that it would be worth 
> it in the long run, and the payoff would far outweigh the cost.
>
> As far as Microsoft owning github, etc I think it is almost moot. Any 
> process will have some amount of vendor lock in, whether github, atlassian 
> (jira & bitbucket parent company), or even gitlab. I think what is 
> important is to pick one system as a community that we are happy with. Any 
> one of those three could do what is ultimately needed, which is a 
> centralized and consistent development platform.
>
> On Wednesday, August 7, 2019 at 12:33:59 AM UTC-7, Aymeric Augustin wrote:
>>
>> Hello John,
>>
>> This was discussed before, when we moved from self-hosted svn to 
>> GitHub-hosted git, but I'm not sure there are public archives of all 
>> discussions.
>>
>> As far as I remember, the main points to tackle are:
>>
>> 1. Does GitHub allow "anonymous triage" i.e. labelling, closing, and 
>> reopening issues by non-committers? I think there was a recent announcement 
>> in this area. I didn't check the details. Previously, bot-powered 
>> workarounds were suggested, but they wouldn't provide a good user 
>> experience. You want discoverable buttons, not a cheat sheet of magic 
>> comments.
>>
>> 2. Does the GitHub UI scale to thousands of issues? In theory, any 
>> classification system can be reproduced with namespaced labels e.g. 
>> "component:ORM", "status:ready-for-checkin", etc. In practice, it's 
>> unlikely to be as convenient as what currently exists on Trac.
>>
>> Perhaps it's just me, but I always found GitHub issues hard to use when I 
>> had more than on page of issues. Indeed, at that point, I need a labelling 
>> system to filter issues. Then I need to keep all the rules of that system 
>> in my head instead of having the UI guide me — and prevent me from 
>> infringing the system...
>>
>> 3. How do we migrate issues history from Trac to GitHub? Preserving 
>> comment authorship doesn't seem obvious, especially for authors who don't 
>> have the same username on Trac and GitHub or authors who don't have a 
>> GitHub account.
>>
>> Initially an effort was made to sync usernames of core devs between Trac 
>> and GitHub to prevent security problems but that's a small subset of 
>> contributors.
>>
>> 4. Are we still able to export everything from GitHub and move on to the 
>> next thing? Perhaps there's an obvious answer. I didn't look. Usually 
>> Django takes a pragmatic position: we won't reject GitHub outright because 
>> it isn't open source. However, we wouldn't want to lock ourselves into a 
>> platform we don't control.
>>
>> Who would have bet, three years ago, that GitHub would be the property of 
>> Microsoft today? What if Microsoft sells it to Oracle in three years? It's 
>> nice to keep our options open :-)
>>
>> We put the code there because we were confident that we could pull the 
>> git history. Then everyone started using pull requests, which was likely a 
>> good thing, but wasn't really planned or thought through, and I don't think 
>> we can export PR comments meaningfully. GitHub did some good vendor lock in 
>> there.
>>
>> 5. How do we preserve links to SVN commits? Currently, they're redirected 
>> on https://code.djangoproject.com/ with this nginx rule:
>>
>> rewrite ^/changeset/(\d+)/?$ 
>> https://www.djangoproject.com/svntogit/$1/ permanent;
>>
>> and then redirected again by this application:
>>
>> https://github.com/django/djangoproject.com/tree/master/svntogit
>>
>> It would be nice to preserve these links in issues copied from Trac to 
>> GitHub, which probably means pre-processing comments to rewrite links.
>>
>> There may be more, but that's what comes to mind!
>>
>> A process DEP 
>> 
>>  is 
>> the way to go to propose this change.
>>
>> Best 

Re: Redis cache support in core

[Josh Smeaton]

Markus, I'd prefer that approach over vendoring a more incomplete solution. 
Bonus points if we can add packages to extra_requires, so that `pip install 
Django[redis]` would work nicely.

On Friday, 21 June 2019 23:17:25 UTC+10, Markus Holtermann wrote:
>
> Hi all, 
>
> may I suggest that django-redis may be "promoted" to an official Django 
> package under the Django GitHub organization? This would follow 
> https://github.com/django/deps/blob/master/final/0007-official-projects.rst 
> . The package would be pointed out explicitly in the Django docs but would 
> be shipped outside of Django. 
>
> The benefit with 3rd party packets is their shorter release cycle. Which, 
> in the context of django-redis could be beneficial . 
>
> /Markus 
>
> On Fri, Jun 21, 2019, at 2:43 PM, 'Ivan Anishchuk' via Django developers 
>  (Contributions to Django itself) wrote: 
> > I wouldn't say it's that complicated a setup. It would require a single 
> > settings snippet -- just like the ones for other backends -- and, I 
> > guess, a link to django-redis docs for more details (if django-redis is 
> > what we recommend), maybe a quick explanation of what is CLIENT_CLASS 
> > and other options. While it would add some maintenance burden 
> > (occasionally checking whether any breaking changes were introduced in 
> > the 3rd party package that require updating settings) it's still way 
> > easier than adding a backend to django core. 
> > 
> > While I agree with others about redis being popular and adding such a 
> > backend in django being a good idea (I would love if that happened) I 
> > understand the reasons for not doing it. A recommendation of a 3rd 
> > party package + setup documentation, on the other hand, is pretty 
> > simple thing to do. 
> > 
> > If we want, it's also not very hard to provide 
> > `django.core.cache.backend.redis.Redis Cache` that depends on 
> > django-redis and is an alias for `django_redis.cache.RedisCache` -- 
> > it's basically the way it works with DB backends, I don't see why it 
> > wouldn't be a good idea for cache as well. 
> > 
> > Ivan. 
> > 
> > On Thu, Jun 20, 2019, 04:02 Josh Smeaton  > wrote: 
> > > Celery explicitly document their integration with Redis though. I 
> don't think we want to take over documenting the setup of a 3rd party 
> package in Django. 
> > > 
> > > On Thursday, 20 June 2019 11:00:27 UTC+10, Ivan Anishchuk wrote: 
> > >> How about making one of the third-party packages an optional 
> dependency? Celery, for example, does that: you can just install 
> celery[redis] without having to figure out what other packages you need to 
> enable redis support. 
> > >> 
> > >> Ivan. 
> > >> 
> > >> On Wed, Jun 19, 2019 at 6:44 AM Josh Smeaton  
> wrote: 
> > >>> There are already several 3rd party packages that implement redis as 
> a django cache backend, for example https://github.com/niwinz/django-redis 
> > >>> 
> > >>> We already have a base class for cache backends - and several 
> implementing it (such as memcache). I don't think there's much benefit 
> taking on another backend when it's already got very good support as an 
> external package. 
> > >>> 
> > >>> 
> > >>> On Tuesday, 18 June 2019 01:14:25 UTC+10, Dulmandakh Sukhbaatar 
> wrote: 
> > >>>> Hello, 
> > >>>> 
> > >>>> I would like to work on Redis support in core, and I would like to 
> discuss proper solution for that. 
> > >>>> 
> > >>>> Redis is getting so popular and almost every modern backend stack 
> uses it someway, therefore I think that supporting it as a cache backend in 
> core would make Django more appealing. A solution I'm proposing is to 
> extract base KV backend from current Memcached and extend it for both 
> Memcached and Redis, and this won't add many new code to the core. Also 
> we'll have base class for KV storage backends. 
> > >>>> 
> > >>>> Thanks. 
> > 
> > >>>  -- 
> > >>>  You received this message because you are subscribed to the Google 
> Groups "Django developers (Contributions to Django itself)" group. 
> > >>>  To unsubscribe from this group and stop receiving emails from it, 
> send an email to django-d...@googlegroups.com. 
> > >>>  To post to this group, send email to django-d...@googlegroups.com. 
> > >>>  Visit this group at 
> https://groups.google.com/group/django-developers. 
> > >>>  To vi

Re: Redis cache support in core

[Josh Smeaton]

Celery explicitly document their integration with Redis though. I don't 
think we want to take over documenting the setup of a 3rd party package in 
Django.

On Thursday, 20 June 2019 11:00:27 UTC+10, Ivan Anishchuk wrote:
>
> How about making one of the third-party packages an optional dependency? 
> Celery, for example, does that: you can just install celery[redis] without 
> having to figure out what other packages you need to enable redis support.
>
> Ivan.
>
> On Wed, Jun 19, 2019 at 6:44 AM Josh Smeaton  > wrote:
>
>> There are already several 3rd party packages that implement redis as a 
>> django cache backend, for example https://github.com/niwinz/django-redis
>>
>> We already have a base class for cache backends - and several 
>> implementing it (such as memcache). I don't think there's much benefit 
>> taking on another backend when it's already got very good support as an 
>> external package.
>>
>>
>> On Tuesday, 18 June 2019 01:14:25 UTC+10, Dulmandakh Sukhbaatar wrote:
>>>
>>> Hello,
>>>
>>> I would like to work on Redis support in core, and I would like to 
>>> discuss proper solution for that.
>>>
>>> Redis is getting so popular and almost every modern backend stack uses 
>>> it someway, therefore I think that supporting it as a cache backend in core 
>>> would make Django more appealing. A solution I'm proposing is to extract 
>>> base KV backend from current Memcached and extend it for both Memcached and 
>>> Redis, and this won't add many new code to the core. Also we'll have base 
>>> class for KV storage backends.
>>>
>>> Thanks.
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-d...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/bdb84d20-0489-4ecd-b198-fa5878f5c617%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/django-developers/bdb84d20-0489-4ecd-b198-fa5878f5c617%40googlegroups.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/335b087c-801a-452b-a5b3-a9711e4a00b8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Redis cache support in core

[Josh Smeaton]

There are already several 3rd party packages that implement redis as a 
django cache backend, for example https://github.com/niwinz/django-redis

We already have a base class for cache backends - and several implementing 
it (such as memcache). I don't think there's much benefit taking on another 
backend when it's already got very good support as an external package.


On Tuesday, 18 June 2019 01:14:25 UTC+10, Dulmandakh Sukhbaatar wrote:
>
> Hello,
>
> I would like to work on Redis support in core, and I would like to discuss 
> proper solution for that.
>
> Redis is getting so popular and almost every modern backend stack uses it 
> someway, therefore I think that supporting it as a cache backend in core 
> would make Django more appealing. A solution I'm proposing is to extract 
> base KV backend from current Memcached and extend it for both Memcached and 
> Redis, and this won't add many new code to the core. Also we'll have base 
> class for KV storage backends.
>
> Thanks.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/bdb84d20-0489-4ecd-b198-fa5878f5c617%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Faster Migrations! But at what cost?

[Josh Smeaton]

Just to add some stats to the conversation, our largest project has ~90 
apps (+3rd party), and 240 migration files. This is after we reset our 
migration history when migrating from 1.8 -> 1.11 (just over a year ago). 
We would have had well in excess of 800 migration files at that point.

To avoid running migrations in unit tests, we define:

class DisableMigrations(object):
def __contains__(self, item):
return True

def __getitem__(self, item):
return None

MIGRATION_MODULES = DisableMigrations()

Which effectively syncdb's rather than applying migrations. Caveat - you 
lose all data/runpython/runsql operations in your test environment.

Our dev/staging setup relies on exporting a sanitised version of 
production, so we only need to run incremental migrations since the last 
backup.



On Tuesday, 21 May 2019 13:26:39 UTC+10, Raphael Gaschignard wrote:
>
> Hi Markus, Simon, 
>
>   Both of you, thank you for the detailed replies and status report on 
> this sort of stuff. 
>
> > Did you look into squashing these 500 migrations by any chance? 
>
> Yeah so I'll go in and squash things, though (partly because effective 
> squashing requires moving models around, though we're still at Django 
> 1.11 so it might become easier, partly because of cross-app references) 
> it's a decent amount of work. 
>
> I do like the idea of reusing a "effectively production DB" for things, 
> not least because it's actually acurate. 
>
> OK so the Operation API is effectively documented. I am tempted to try 
> modifying the `database_forwards` and `database_backwards` to not 
> require `from_state` (probably through adding a hook that gets called 
> _before_ `state_forwards` to allow capturing the small details of the 
> old state). It might actually be possible to make this backwards 
> compatible by porting Django migrations to use an `apps`-free workflow, 
> but falling back to the old "re-render the universe" mechanism for 
> operations that don't apply it. 
>
> This is very "that xkcd comic about time spent versus time saved" 
>
>  > will cause trouble with RunSQL and other operations that use related 
> field or model attributes 
>
> So one thing I felt like was an invariant in this code was that field 
> sharing was expected? From the docstring of ModelState: 
>
>  Note that while you are allowed to mutate .fields, you are not 
> allowed 
>  to mutate the Field instances inside there themselves - you must 
> instead 
>  assign new ones, as these are not detached during a clone. 
>
> Also maybe you meant to refer to RunPython instead of RunSQL. But I get 
> your point in general here. Related models can be a problem 
>
> One throwaway idea would be to not allow related model/related field 
> access in these models? There's already a lot of documentation related 
> to not allowing general model methods (effectively establishing that 
> "migration models are _not_ normal models"), so there's a bit of 
> precedent. But beyond the general backwards incompatability, it might 
> not actually even be obvious how one would implement this. And you kinda 
> need this info for foreign keys and the like anyways. 
>
>
> Working directly off of `ModelState` is interesting, and I think there 
> might be a backwards-compatible way forward there, where we still allow 
> for rendering on certain operations but hold off on it on the basic 
> ones. Even in our large project, most of our migration operatiosn are 
> dirt-simple, so if the core django migrations could work off of 
> `ModelState` then we could get a fast path through there. 
>
> Thanks for your input, both of you. I have a couple ideas now that I'm 
> pretty tempted to try out, mainly around the "fast path and slow path" 
> strategies that should offer backwards compatibility. 
>
>   Raphael 
>
> Markus Holtermann wrote on 2019/05/21 2:26: 
> > Thanks Raphael for bringing this topic up and Simon for your input 
> already. 
> > 
> > I just left a note on your PR: 
> https://github.com/django/django/pull/11388#issuecomment-494076750 . I'll 
> quote it here for ease of readability: 
> > 
> > As far as I can see right now, a similar caching happened as a first 
> approach to the Django 1.8 release but cause significant problems, 
> specifically with regards to relational fields. Relational fields 
> (ForeignKey, OneToOneField, ManyToManyField) keep an instance reference to 
> the related model in `.related_model` or the related fields in 
> `.related_fields`. The problem now is, if you reuse a field (and you do 
> because you're only calling `list()` on `self.fields` to copy but not 
> deepcopy the list), you're screwing up references between the models that 
> _will_ cause trouble with `RunSQL` and other operations that use related 
> field or model attributes. 
> > 
> > 
> https://github.com/django/django/blob/1d0bab0bfd77edcf1228d45bf654457a8ff1890d/django/db/models/fields/__init__.py#L495-L499
>  
> > 
> >  From my work on migrations, 

Re: Proposal to format Django using black

[Josh Smeaton]

To answer the question about decision making...

Usually a decision is made if there’s reasonable consensus in the
discussion or after a vote. If no clear consensus can be reached with the
core team, then a decision can be escalated to the technical board for a
vote.

In the future, if the core team is dissolved in its primary capacity,
consensus in the discussion rather than consensus among the core team would
be the first step.

I figure all reasonable arguments have probably been made at this stage.
But there are really two questions we should be proposing.

1. Should Django adopt an autoformatter?
2. If it should, should that autoformatter be black or something else?

Considering the rather strong views in this thread, I’d probably recommend
that a DEP is drafted for each of the above, where opinions and comments
can be gathered more formally.

On Thu, 25 Apr 2019 at 20:15, James Bennett  wrote:

> I like Django's style guide.
>
> I like the *idea* of an autoformatter.
>
> I dislike the particular mostly-unconfigurable style Black enforces, and I
> find that several of its rules negatively impact code readability.
>
> So I would be -1 on applying it to Django.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/wK2PzdGNOpQ/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/CAL13Cg-MJqE%2BxmxpBhymxaUz__6V_htE1XA-XRxZH5Uxa3WY_A%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAPbDM0f24%3D62%3DBXD-OxgonjGpiJ13W4koaHg%2BbnwPdefzM2Baw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal to format Django using black

[Josh Smeaton]

Whoops, you are correct.

It did not at the time I added black to my projects, my information was
quite out of date. Sorry for the wrong correction!

On Wed, 24 Apr 2019 at 22:51, Florian Apolloner 
wrote:

>
> On Wednesday, April 24, 2019 at 1:25:55 PM UTC+2, Josh Smeaton wrote:
>>
>> Black does not support disabling formatting by block with a comment. It
>> removes all choice except for the upfront choices of length and string
>> normalisation.
>>
>
> It does, "# fmt: off" and "# fmt: on" can be used
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/wK2PzdGNOpQ/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/237eb4ea-72b5-4ba0-a484-ad545d46479c%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/237eb4ea-72b5-4ba0-a484-ad545d46479c%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAPbDM0dC4iQvbRD_bk7U9bh1C4PG5a8s6f3npAtwkkSQfd_BAQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal to format Django using black

[Josh Smeaton]

Black does not support disabling formatting by block with a comment. It
removes all choice except for the upfront choices of length and string
normalisation.

On Wed, 24 Apr 2019 at 20:22, Rene Fleschenberg 
wrote:

> Hi
>
> > The arguments against a particular code style will never stop. In this
> > case it's a big advantage to choose a tool developed by others which
> > does not allow any bikeshedding.
> >
> Will black really stop the arguments (which are not bikeshedding, IMO --
> readability counts)? Or will they just be replaced by arguments over
> when to use ``# fmt: off``?
>
> --
> Rene
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers  (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/django-developers/wK2PzdGNOpQ/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/954d583a-8dd4-bc83-dad0-a95b165d5546%40fleschenberg.net
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAPbDM0ehKKHiznK%2BsGzH%3DddrLDTcrHZUJChpYVHHE-K32RRgNA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal to format Django using black

[Josh Smeaton]

FWIW, I'd be opposed to YAPF at all, but especially over Black. The 
community (that is, large projects that aren't Django) are standardising 
around Black - a single unambiguous format style. Having a slightly 
different format to everybody else I see as backwards. Contributors would 
also likely need to maintain two installations - one for Django, and one 
for (nearly) everything else. I'm probably being a little dramatic here 
with regards to the reach Black currently has, but there are many projects 
using it successfully.

There are also the problems with YAPF as described by Łukasz (black author) 
himself: https://news.ycombinator.com/item?id=17155205

Copied from link above:

- YAPF would at times not produce deterministic formatting (formatting the 
same file the second time with no changes in between would create a 
different formatting); Black treats this as a bug;

- YAPF would not format all files that use the latest Python 3.6 features 
(we have a lot of f-strings, there's cases of async generators, complex 
unpacking in collections and function calls, and so on); Black solves that;

- YAPF is based on a sophisticated algorithm that unwinds the line and 
applies "penalty points" for things that the user configured they don't 
like to see. With a bit of dynamic programming magic it arrives at a 
formatting with the minimal penalty value. This works fine most of the 
time. When it doesn't, and surprised people ask you to explain, you don't 
really know why. You might be able to suggest changing the penalty point 
value of a particular decision from, say, 47 to 48. It might help with this 
particular situation... but break five others in different places of the 
codebase.

I'm unsure how relevant all of these points still are. For me, I was never 
able to get YAPF to play nice, or get widespread agreement on a set of 
rules that all devs were OK with (lots of bikeshedding).

On Wednesday, 24 April 2019 16:35:57 UTC+10, Carlton Gibson wrote:
>
> Thanks for the YAPF suggestion (and sample config!) I'll have a go with 
> this this week. 
> (If we can get auto-formatting, just on diffs(?), that matches the 
> existing style...)
>
> On Monday, 22 April 2019 20:10:41 UTC+2, thinkwel...@gmail.com wrote:
>>
>> I wonder if there's a middle ground between minimizing code churn and 
>> having a standardized formatter. Our team recently switched to yapf after 
>> carefully configuring a .style.yapf file that's included in the root 
>> directory of every new repo. Once that config file is done, the workflow 
>> for yapf vs an unconfigurable formatter is identical.
>>
>>
>> I experimented a bit with the following .style.yapf settings on the 
>> django codebase and think the output is as good as black's without some of 
>> the arbitrariness of string quotes:
>>
>> [style]
>> based_on_style = pep8
>> column_limit=100
>> i18n_function_call=['_']
>> blank_line_before_nested_class_or_def=True
>> join_multiple_lines=False
>> indent_dictionary_value=False
>>
>> coalesce_brackets=True
>> dedent_closing_brackets=True
>> align_closing_bracket_with_visual_indent=False
>> space_between_ending_comma_and_closing_bracket=False
>>
>> split_complex_comprehension=True
>> split_before_first_argument=True
>> split_before_logical_operator=False
>> split_before_bitwise_operator=False
>> split_arguments_when_comma_terminated=True
>> split_before_expression_after_opening_paren=True
>> split_before_named_assigns=True
>>
>>
>> Yapf currently has more stars than black, but whether black has more 
>> momentum or not, who can say.
>>
>>
>> On Monday, April 22, 2019 at 10:14:44 AM UTC-4, Nick Sarbicki wrote:
>>>
>>>
 I'm just saying that if "As contributor, I can haz automatic code 
 formatter to lower the barrier" is precisely the story you want to solve, 
 then black may not be the only solution you want to consider deeply ;)


>>> Jamie, sure, I wasn't responding directly to you about this, more to the 
>>> general people arguing against blacks style choices. I would happily 
>>> consider alternatives to black - although (without any formal research to 
>>> back this claim) it does feel like black has the most community support.
>>>
>>> My point is mostly that if there is a growing community consistency 
>>> through black then I'd be hesitant to choose another tool that goes against 
>>> this.
>>>
>>>  
>>>
 > Consistency in the end is the most important thing (even PEP8 agrees 
 > there). 

 Not sure where you got that impression: 
 https://pep8.org/#a-foolish-consistency-is-the-hobgoblin-of-little-minds 
 
  

 Pep8 clearly states consistency is less important then readability 
 (it's the 
 first thing mentioned and mentioned repeatedly that you can use as an 
 argument 
 to break consistency). And this is the 

Re: Proposal to format Django using black

[Josh Smeaton]

We aren't talking about code minifiers though, are we? We're talking about 
a very specific tool with very specific rules. No one will ever agree on 
one specific code style, which is why subjectivity is anti-productive. 
Black chooses a specific set of rules and removes ambiguity. Some choices 
will be agreeable, others will not be. And the ones that are agreeable 
aren't agreeable to every person.

> and code passed through autoformatters (especially if author _didn't 
think_ about style) is harder to understand then written by thinking humans

I find this statement to be mostly incorrect with regard to Black. There 
are edge cases, but they are few.

On Wednesday, 17 April 2019 09:45:11 UTC+10, Ivan Anishchuk wrote:
>
> Yeah, it's very common to confuse style with formatting. Not having to 
> think is a productivity win when and only when you don't care about your 
> product's quality (in which case I'd suggest using generators and stop 
> thinking about code altogether). Language is a communication tool, a 
> programming language is a tool programmers use to talk to each other, and 
> code passed through autoformatters (especially if author _didn't think_ 
> about style) is harder to understand then written by thinking humans. It's 
> pretty much like trying to stop thinking about style in English -- by using 
> some processor to put your commas and whitespace in place you can generate 
> something that is usually possible to understand, at least for the most 
> part, but that and _good style_ would still be worlds apart (yes, natural 
> languages are different, it's much easier to make things worse there, but 
> it's not any easier to make things better in programming languages).
>
> There are situations where compromising style for formatting is acceptable 
> and even a good idea, I don't think here is one of them. Removing a 
> significant expressive tool from a language just to ensure your whitespace 
> is arranged according to some arbitrary rules is about as 
> counter-productive and pointless as it can get. Not because sometimes it 
> produces some bad results but exactly because it prevents humans from 
> thinking and expressing themselves properly. What would you think about 
> some processor renaming all your variables so that you don't have to think 
> about naming them? Must be even better for productivity.
>
> Ivan.
>
> On Wed, Apr 17, 2019 at 12:35 AM Josh Smeaton  > wrote:
>
>> Ivan, what you’re talking about is subjective code formatting, and lends 
>> itself to extreme bikeshedding and little consensus. Django already has a 
>> formatting standard that mergers try to enforce at review and commit time. 
>> But it’s time consuming, prone to missing things, and requires lots of back 
>> and forth. 
>>
>> Not having to think of formatting at all, either as a developer or 
>> reviewer, is a major productivity win. Even if some of the formatting 
>> choices are not agreeable. 
>>
>> On Wed, 17 Apr 2019 at 03:38, 'Ivan Anishchuk' via Django developers 
>> (Contributions to Django itself) > > wrote:
>>
>>> My two cents: black's mostly ok but I'm against any auto-formatting. 
>>> Code style is much more than whitespace arrangement, it requires looking, 
>>> creative thinking, judging, and sometimes actually compromising rules and 
>>> looks for readability while the usual attitude with such tools is "I have 
>>> this autoformatter tool so I don't have to think about style anymore, see, 
>>> all my whitespace is nice and shining". Using some helpers in your favorite 
>>> editor is great but relying on automatics without looking is almost 
>>> completely counter-productive.
>>>
>>> I find that PR autocommenter is a great way to keep issues detectable by 
>>> flake8/pylint to a minimum.
>>>
>>> Ivan.
>>>
>>> On Sat, Apr 13, 2019 at 6:35 PM Herman S >> > wrote:
>>>
>>>> Hi.
>>>>
>>>> I propose that Django starts using 'black' [0] to auto-format all 
>>>> Python code.
>>>> For those unfamiliar with 'black' I recommend reading the the projects 
>>>> README.
>>>> The short version: it aims to reduce bike-shedding and non value-adding
>>>> discussions; saving time reviewing code; and making the barrier to 
>>>> entry lower
>>>> by taking some uncompromissing choices with regards to formatting.  
>>>> This is
>>>> similar to tools such as 'gofmt' for Go and 'prettier' for Javascript.
>>>>
>>>> Personally I first got involved contributing to Django couple of weeks 
>>>> back,
>&g

Re: Proposal to format Django using black

[Josh Smeaton]

Ivan, what you’re talking about is subjective code formatting, and lends
itself to extreme bikeshedding and little consensus. Django already has a
formatting standard that mergers try to enforce at review and commit time.
But it’s time consuming, prone to missing things, and requires lots of back
and forth.

Not having to think of formatting at all, either as a developer or
reviewer, is a major productivity win. Even if some of the formatting
choices are not agreeable.

On Wed, 17 Apr 2019 at 03:38, 'Ivan Anishchuk' via Django developers
(Contributions to Django itself)  wrote:

> My two cents: black's mostly ok but I'm against any auto-formatting. Code
> style is much more than whitespace arrangement, it requires looking,
> creative thinking, judging, and sometimes actually compromising rules and
> looks for readability while the usual attitude with such tools is "I have
> this autoformatter tool so I don't have to think about style anymore, see,
> all my whitespace is nice and shining". Using some helpers in your favorite
> editor is great but relying on automatics without looking is almost
> completely counter-productive.
>
> I find that PR autocommenter is a great way to keep issues detectable by
> flake8/pylint to a minimum.
>
> Ivan.
>
> On Sat, Apr 13, 2019 at 6:35 PM Herman S 
> wrote:
>
>> Hi.
>>
>> I propose that Django starts using 'black' [0] to auto-format all Python
>> code.
>> For those unfamiliar with 'black' I recommend reading the the projects
>> README.
>> The short version: it aims to reduce bike-shedding and non value-adding
>> discussions; saving time reviewing code; and making the barrier to entry
>> lower
>> by taking some uncompromissing choices with regards to formatting.  This
>> is
>> similar to tools such as 'gofmt' for Go and 'prettier' for Javascript.
>>
>> Personally I first got involved contributing to Django couple of weeks
>> back,
>> and from anecdotal experience I can testify to how 'formatting of code'
>> creates
>> a huge barrier for entry. My PR at the time went multiple times back and
>> forth
>> tweaking formatting. Before this, I had to research the style used by
>> exploring
>> the docs at length and reading at least 10-20 different source – and even
>> those
>> were not always consistent. At the end of the day I felt like almost 50%
>> of the
>> time I used on the patch was not used on actually solving the issue at
>> hand.
>> Thinking about code formatting in 2019 is a mental energy better used for
>> other
>> things, and it feels unnecessary that core developers on Django spend
>> their time
>> "nit-picking" on these things.
>>
>> I recently led the efforts to make this change where I work. We have a
>> 200K+
>> LOC Django code-base with more than 30K commits. Some key take-aways: it
>> has
>> drastically changed the way we work with code across teams, new engineers
>> are
>> easier on-boarded, PR are more focused on architectural choices and
>> "naming
>> things", existing PRs before migration had surprisingly few conflicts and
>> were
>> easy to fix, hot code paths are already "blameable" and it's easy to
>> blame a
>> line of code and go past the "black-commit", and lastly the migration went
>> without any issues or down-time.
>>
>> I had some really fruitful discussions at DjangoCon Europe this week on
>> this
>> very topic, and it seems we are not alone in these experiences. I would
>> love to
>> hear from all of you and hope that we can land on something that will
>> enable
>> *more* people to easier contribute back to this project.
>>
>> I've set up how this _could_ look depending on some configurables in
>> Black:
>>
>> * Default config: https://github.com/hermansc/django/pull/1
>> * Line length kept at 119: https://github.com/hermansc/django/pull/3
>> * Line length kept at 119, no string normalization:
>> https://github.com/hermansc/django/pull/2
>>
>> Please have a look at the Black documentation. It explains the benefits
>> better
>> than I possibly could do here.
>>
>> With kind regards,
>> Herman Schistad
>>
>> [0]: https://github.com/ambv/black
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Django developers  (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to django-developers+unsubscr...@googlegroups.com.
>> To post to this group, send email to django-developers@googlegroups.com.
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/django-developers/CAN%3DnMTx0EE5WfXuccv_e3MBuCxp9u_pAV_ow5MxNST6MptTDBw%40mail.gmail.com
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this topic, visit
> 

Re: Proposal to format Django using black

[Josh Smeaton]

Yes, let’s not discuss the tools choice in quotes, that’s for another
place. To provide some information though, black will not convert single
quote strings if the string itself contains a double quote. It won’t escape
the character, it’ll use single quotes.

On Mon, 15 Apr 2019 at 18:48, Kye Russell  wrote:

> This is something discussed af length on the black issue tracker and not
> something the author wishes to change.
>
> On Mon, 15 Apr 2019 at 4:47 pm, Dmitriy Sintsov 
> wrote:
>
>> Why can't it use mixed quotes, single quotes for all strings except the
>> strings that contain single quote characters? I think mixed syntax of
>> strings is made in various programming languages so both single and double
>> quotes can be used inside strings not having to use ugly backslash
>> escaping. Maybe you or someone can create such feature request in black
>> repo? Why the people do not like mixed quotes strings and try to eliminate
>> such great feature? Aesthetical issues are very much biased and enforcing
>> these automatically is not always a good idea. Does the switching to double
>> quote string makes the code error prone and safer? Probably not. Just some
>> voluntary personal choice.
>>
>>
>> On Monday, April 15, 2019 at 2:02:49 AM UTC+3, charettes wrote:
>>>
>>> I was and and I'm still bugged by how black decided to go for double
>>> quotes instead of single ones. Even if it's backed some valid arguments it
>>> left all the projects following PEP 8's recommendations over the years with
>>> this git blaming loss trade off. From past experimentation with large'ish
>>> projects following PEP8's guideline string quotes changes represented well
>>> over 50% of the diff and there's no way to turn off only this form of
>>> string normalization, it's all or nothing. At $WORK we even went as far as
>>> forking the project to switch to use single quotes instead[0]. I understand
>>> requiring the use of a fork is not a feasible solution for Django but I
>>> just wished there was an option to stick to single quotes to reduce the
>>> cost of adoption for such projects without loosing all the other nifty
>>> string formatting goodies[1] /rant
>>>
>>> Even if I don't completely agree with black's formatting choices and the
>>> sometimes quite unnatural code it generates my past month's usage at work
>>> and the adoption of the project in the Python ecosystem that made it's way
>>> in most IDE makes me mostly share the same feeling as Aymeric; it's not
>>> always pretty but not having to think about it makes it worth it. I suggest
>>> we stick to disabling string normalization to reduce the noise generated by
>>> the migration though.
>>>
>>> Cheers,
>>> Simon
>>>
>>> [0] https://github.com/zapier/black/pull/1
>>> [1] https://github.com/ambv/black#strings
>>>
>>> Le dimanche 14 avril 2019 09:40:10 UTC-4, Aymeric Augustin a écrit :

 Hello,

 I'm strongly in favor of adopting black with the default options.

 In my opinion, it's the first Python code formatter that passes the bar
 of "on average, does better than any single programmer". Trying to enforce
 something else manually is a waste of energy that produces worse results.

 I like nicely formatted code and I hate some of what black produces,
 typically on Django model definitions. However, I've seen the benefits of
 an automated code formatter, even on projects where I write almost all of
 the code.

 I'm positive the benefits will be great on Django, where you can tell
 when a module was (re)written just by looking at the style. Yes, there are
 some downsides, but it's clearly a good tradeoff.

 Regarding the migration strategy, converting one package at a time
 sounds good, because then we can proof-read the Big Diff and perhaps tweak
 the code where the result really hurts the eyes.

 Most of Django's style guide will still applies, as it's mostly
 conventions about naming things, ordering declarations, etc.

 Best regards,

 --
 Aymeric.



 On 13 Apr 2019, at 13:52, Herman S  wrote:

 Hi.

 I propose that Django starts using 'black' [0] to auto-format all
 Python code.
 For those unfamiliar with 'black' I recommend reading the the projects
 README.
 The short version: it aims to reduce bike-shedding and non value-adding
 discussions; saving time reviewing code; and making the barrier to
 entry lower
 by taking some uncompromissing choices with regards to formatting.
 This is
 similar to tools such as 'gofmt' for Go and 'prettier' for Javascript.

 Personally I first got involved contributing to Django couple of weeks
 back,
 and from anecdotal experience I can testify to how 'formatting of code'
 creates
 a huge barrier for entry. My PR at the time went multiple times back
 and forth
 tweaking formatting. Before this, I had to research the style 

Re: Proposal to format Django using black

[Josh Smeaton]

Agree with Florian that the progressive rollout is more trouble than it's 
worth. Tangling up feature changes with whole file formatting will make it 
harder to review changes, but will also be more difficult to use tools like 
git blame.

As for disagreeing with some of Blacks choices - you learn very quickly to 
live with those choices, and forget those choices. Any set of configuration 
we could come up with for an existing tool would likely be bikeshedded to 
death and nothing would eventuate.

For line lengths, projects I work on set Black to 100 columns, but flake8 
line length is capped to 120. This lets some long strings (black doesn't 
break long strings over multiple lines) get past flake8 without being over 
the top.

I'm in favour of using blacks double quotes for strings. I **hated** this 
decision when it was first made, but have seriously come around to it, and 
prefer it aesthetically too. Consistency with other projects (most big 
projects are adopting black) I feel is a good goal. The diff is already 
going to be huge, there's not much value in protecting a few strings.

Finally, there are some tricks you pick up if black fights you on some 
decisions. To use Berkers example:

TIME_INPUT_FORMATS = [ 
'%H:%M:%S', # '14:30:59' 
'%H:%M:%S.%f', # '14:30:59.000200' 
'%H:%M', # '14:30' 
] 

TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M'] # '14:30:59' 
# '14:30:59.000200' # '14:30' 

Break each individual format into its own variable, with appropriate 
comment, and add the variables to the list.

HMS = "%H:%M:%S"  # 14:30:59
HMSF = ".." 
HM = ".."
TIME_INPUT_FORMATS  = [HMS, HMSF, HM]

Obviously just an example, but something to keep in mind.


On Sunday, 14 April 2019 20:20:31 UTC+10, Florian Apolloner wrote:
>
> Hi,
>
> On Sunday, April 14, 2019 at 10:22:46 AM UTC+2, Curtis Maloney wrote:
>>
>> Can such a tool be automated into, say, github in a way that doesn't 
>> create extra commit noise? 
>>
>
> Probably, but after blacking (is that even a word ;)) the codebase once 
> there shouldn't be much commit noise.
>
> I side with those who favour a progressive solution, whereby new code 
>> only has the new tool applied. 
>>
>
> I think that would make it hard on tools like git-hyper-blame which allow 
> one to skip revisions. Also you cannot just run black over changes, the 
> minimal unit is one file as far as I am aware. Which would make PR very 
> very hard to review because you'd have a bunch of syntax changes next to 
> real changes.
>
> In my experience with using black [we use it at work], there are 
>> numerous choices (including those demonstrated in this list already) 
>> where it can significantly _increase_ the cognitive load in simply 
>> parsing the code. 
>>
>
> Generally I am okay with the way black formats code. But I have to admit 
> that for some very tricky codepaths I tend to turn it off from time to 
> time. That does not happen very often though.
>
> As simple as black can make the job of code formatting, I feel I'd 
>> rather see a different tool that retained the benefits of "trivial code 
>> reformatting", but still allowed us to retain some of Django's existing 
>> code formatting rules. 
>>
>
> Open to suggestions, did you just offer to write one :)
>
> (An interesting [and defensible] choice, we had a module with a lot of 
>> strings wrapped across lines. black opted to push them onto the same 
>> line, but NOT to merge them.  This is because in Python prior to 3.7, it 
>> would have altered the generated AST - one of the guides black uses) 
>>
>
> We can and should fix those occurrences in our codebase then I guess.
>  
> Cheers,
> Florian
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8b81a047-babe-42d0-97f5-3af5f6b264fa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Filterable subqueries ...

[Josh Smeaton]

With regard to the stunning silence you're witnessing, it's my guess that 
you haven't made yourself clear enough in a concise way. The stackoverflow 
post is large, and doesn't point out what's missing very clearly.

What is the SQL you want?

What is the SQL you're getting, and what is the queryset you're 
constructing?

I **think** what you're trying to get to is this:

SELECT * FROM ( 
   SELECT
   t.field,
   lag(t.other, 1) over ( .. ) as wlag
   FROM T t
   WHERE t.field = 1
) inn
WHERE wlag = 3;


That is, you want to be able to filter on annotations without copying the 
annotation into the WHERE clause by wrapping with an outer query that does 
the filtering. Is that correct?

On Thursday, 28 February 2019 23:27:07 UTC+11, Bernd Wechner wrote:
>
> I have a problem I've only been able to solvewhich bugs me. I've posted on 
> the Django users list (to stunning silence) and on stackoverflow:
>
>
> https://stackoverflow.com/questions/54388936/using-django-window-functions-on-a-filtered-queryset
>
> to comparable silence.  I'm rather convinced this can't be done in Django 
> without raw SQL and that it is an integral part of Window function utility 
> so I'd like to propose a native ORM based solution to the problem.
>
> To understand the context it will be necessary to read the post on Stack 
> Overflow above, there seems little point in copying the text here. 
>
> The proposal though is simple enough. It is that django.db.models.Subquery 
> support the methods that django.db.models.QuerySet does, specifically SQL 
> constructing methods like annotate() and filter() - I'm sure there are 
> more. 
>
> The idea is to make easily available a way of selecting from a subquery 
> such that something akin to:
>
> SQ = Subquery(model.objects.filter(...))   
>
> produces SQL in the form:
>
> SELECT ... FROM model WHERE ...
>
> and now:
>
> Q = SQ.filter(---)
>
> would produce SQL in the form:
>
> SELECT * FROM (SELECT ... FROM model WHERE ...) AS SQ WHERE ---
>
> Essentially permitting us to filter on the results of a Subquery. 
>
> Again, this is crucial when using Window functions like LAG and LEAD over 
> a filtered list of objects. The reasons for this are explained on Stack 
> Overflow with samples of code and SQL.
>
> Of course, in spite of the stunning silence on the Django users list and 
> Stackoverflow, there's a chance we can already do this in Django in a way 
> that does not involve Raw SQL that I have not found yet, in which case 
> apologies, this really is a support question for the Django users list. But 
> I have looked hard and asked hard and searched hard and experimented hard 
> and delved into the Django code quite hard and I'm fairly convinced it's 
> not possible at present ... without constructing Raw SQL.
>
> Regards,
>
> Bernd.
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6fff5aee-cd4d-467c-8e6d-4c59bf734547%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add an defer=True option for model fields

[Josh Smeaton]

There is a ticket for this one already, filed 4 years ago by me :)

https://code.djangoproject.com/ticket/24096

There are a few options described, but I think `defer=True` was winning 
out. I don't think we considered an `undefer`, but a `defer(None)` would 
fix that. Once that API is built, then we can consider getting LOB fields 
to defer themselves in specific situations.

On Wednesday, 20 February 2019 05:02:50 UTC+11, Dan Davis wrote:
>
>
> What I mean by the below:
> > I was not sure whether to tell him to implement a ModelManager with a 
> get_queryset() method that defers the field,
>
> Of course this works, but I'm not going to maintain this code, and that 
> sort of sophistication creates a need for more sophisticated maintenance.
>
>
> On Tue, Feb 19, 2019 at 12:43 PM Dan Davis  > wrote:
>
>> I have a developer who stores the binary copy of a file in his table.  In 
>> ColdFusion, this was acceptable, because he was writing every query by 
>> hand, and could simply exclude that field.  However, with the Django ORM it 
>> is a bit of a problem.   The primary table he uses is just for the file, 
>> and has a file_name, file_type, file_size, and BinaryField.
>>
>> The problem is that he has a database-level view that incorporates this 
>> field, and it may be that he needs to keep this because other schemas in 
>> our big-office Oracle use the view as an exported synonym.
>>
>> What I advised him to do was to take the BinaryField out of the 
>> database-level view, to protect the ORM from reading these large files into 
>> memory, as in:
>>
>>  [obj for obj in LicensesDBView.objects.all()] 
>>
>> Or, if he cannot do that, to simply defer the field:
>>
>>  [obj for obj in 
>> LicensesDBView.objects.defer('scanned_license').all()] 
>>
>> I was not sure whether to tell him to implement a ModelManager with a 
>> get_queryset() method that defers the field, but it made me wonder whether 
>> we should have a concept of an "initially deferred" field.
>> That is, this is a field that starts deferred, and can be pulled into the 
>> select using a values iterator or a call to only() or defer(), e.g. the one 
>> that cancels prior defers.   The concept of "initially deferred" fields 
>> would certainly require a new queryset method, such as "nodefer" which is 
>> sort of like only but doesn't cause only those fields to load, or rather 
>> defer could accept a syntax like defer('-scanned_license') to cancel that 
>> previous deferred loading field.
>>
>> I'm afraid I probably don't understand all the implications of this 
>> feature, so I thought I'd bring it up on the list before filing any sort of 
>> issue. Its likely this has been discussed before; I cannot do a historical 
>> search all the time, especially when ancient history may not be today's 
>> read on this issue.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/ee5c04e5-69d6-42f9-95ff-c01d553b24c1%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/640bd321-2e8a-4958-bdff-a63ed92ad948%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Will Django ever CompositePrimaryKeys?

[Josh Smeaton]

To add - there's definitely appetite for this feature, but it's a difficult 
one, and no one has stepped up to do it.

There are DEP drafts that cover pieces:

https://github.com/django/deps/blob/master/draft/0191-composite-fields.rst
https://github.com/django/deps/blob/master/draft/0192-standalone-composite-fields.rst

Citus Data was looking to sponsor development for this feature, and still, 
no one with the technical capabilities was willing (or able) to step up: 
https://groups.google.com/forum/#!searchin/django-developers/composite|sort:date/django-developers/wakEPFMPiyQ/ke5OwgOPAQAJ

I would strongly advise not just running off and attempting something on 
your own. Look over the previous attempts and DEPs, and then document your 
way forward either by editing the DEP, and by initiating a thread on this 
list.

Cheers

On Wednesday, 20 February 2019 04:29:41 UTC+11, Dan Davis wrote:
>
> James,
>
> As a Django user I've had this problem often. My best practice ways to 
> handle this is as follows:
>
>
>- If the table is read-only, then create a database-level view that 
>manufactures a primary key by concatenating the primary key columns 
>together.   Lie to Django and say this combined column is the primary key, 
>and keep the Database-level view's model as managed = False at the Django 
>level.   However, keep the SQL for the view in your git repository, and 
>when it changes, build a manual migration - manage.py makemigration fooapp 
>--empty -n create_replace_bar_view.
>
>
>- If the table is read-write, then either (a) create a view with 
>instead of update/instead of insert triggers that manage an underlying 
>table, or (b) just add a unique ID and make the existing primary key a 
>unique together constraint and index.  With the trigger to add a new ID, 
>other users of that table shouldn't notice any issues.
>
>
>- Maybe my statement that Django should manage migrations to the 
>schema seems unworkable. However, one of the biggest gains you can get 
> from 
>Django with old schemas like this is to get their DDL into git. Even if 
> git 
>doesn't do it, make sure some code does it, and it isn't left unmanaged. 
>That's been a key challenge and opportunity for me.
>
>
> As a Django developer, I'm a bit green to volunteer to handle this issue, 
> but this is exactly the kind of issue that leads me to be a developer - I 
> see the ORM and the all-bells included nature of Django as the killer 
> combo.  I don't have to go outside the farm to have database migrations, 
> url routing, etc.  It is almost all there.  So, please ping back in 3 
> months to see whether I'm up to it.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f2a3663c-3d0a-4f97-9699-f016c6bc00cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Validation of m2m

[Josh Smeaton]

Considering https://code.djangoproject.com/ticket/12938 is still marked as 
"open" and https://code.djangoproject.com/ticket/24731 is more or less a 
duplicate (is there a reason it isn't closed as a dupe specifically?), I 
don't think any more consensus is really required. The problem seems 
difficult without diving in and getting your hands dirty, which is probably 
why you're seeing such light discussion on this thread.

What I'd suggest is: put together a proposal and let discussion occur 
around that. Propose an API, and speak to how it won't impact existing 
users, and I think you'll find lots of people will have something to say.

FWIW, I'd be interested in seeing some kind of solution. Our ($work$) 
current response to feature requests that involve validation of m2m fields 
is "too hard for the effort".

Cheers

On Friday, 15 February 2019 22:44:22 UTC+11, Bernd Wechner wrote:
>
> I'm curious what consensus looks like. In what forum among which 
> stakeholders. Clearly among developers who have some knowledge of Djangos 
> innards, and so I suspect here. But I find conversation here on this thread 
> so neatly short I can digest it in a short read, and 
>
> https://code.djangoproject.com/ticket/24731
> https://code.djangoproject.com/ticket/12938
>
> both touch on the same issue with similarly delectably short comment 
> streams that I could read and digest them in a jiffy. Read another way, not 
> a lot of consensus generation discussion or activity visible.
>
> So I'll play the newb (because I am I guess, well been coding with Django 
> for almost 4 years very very casually part time, and have had this 
> particular issue on my "to solve" list for a long time already. But for me 
> to make progress on it I need to invest some time into learning and some of 
> that empirically through experimentation but I'll try and stimulate a 
> little conversation here by doing some of it through the posing of a 
> possibly naive question or two.
>
> Before that let me state that in my application (and I suspect this is not 
> unusual, I am trying to validate m2m relations and one2m as well in what 
> I'd describe as a rich model object, I use the term rich to describe a 
> handful of models related to one another, all of which are updated with one 
> form submission. In my simplest example I am modelling a game session which 
> has a one2many relationship to ranks, which have a one2one relationship to 
> teams which have an m2m relationship to players. A single game session is 
> logged as the ranked list of teams and their players ... but I wouldn't get 
> lost in the one example, the point is such relationships can't be wildly 
> uncommon and we would all like to keep the validation logic in the 
> models. In my example a session would like to check that there are the 
> right number of ranks, not too few, not too many, as the game allows. 
> Validating this in the form is frustrating as it's a) somewhat more 
> complicated and b) not as secure (allows erroneous saves through means 
> other than this form). And the logic belongs in the model. The session 
> knows how many ranks it's expecting. I've seen other examples just as clear.
>
> And so onto the learning through possibly naive questioning.
>
> As I see it there are two versions of this rich object (objects of several 
> models all interrelated:
>
> 1) The database version 
> 2) the ORM version 
>
> The problem I see is that the ORM versions lack primary keys and the 
> relationships they create until they are saved. So aside from inviting 
> correction of any misunderstandings I may have tabled I will ask the 
> salient question: Is this not a classic application for transactions. 
> Namely we save all of the objects without committing then do the validation 
> of the relationships in the ORM. This seems naive to me as it presupposes a 
> few things that may or may not be true (and I fear are not):
>
> 1) The an uncommitted transaction delivers primary keys
> 2) That an uncommitted transaction can easily be reflected back in the ORM
>
> If these are possible, is it not a good chance then to validate the 
> relations int he respective model's Clean() methods and through an 
> exception on failure, that results in a complete roll back and if it 
> succeeds results in a commit. 
>
> I invite commentary, and discussion in the hope of achieving the elusive 
> beast of consensus that Frederico alludes to.
>
> The main need I see is to create the relationships in the ORM, and it may 
> be possible to do this pre-save too, with place-holder PKs, I do exactly 
> this sort of thing at the form level, managing the relations between the 
> various form elements, with placeholder IDs on elements. And so I imagine 
> loosely if the strategy above is impossible that another might implement 
> something along these lines, a way to create all the objects and have them 
> as ORM objects but with placeholder PKs where needed managing the relations 
> and (a central 

Re: Google Summer of Code 2019

[Josh Smeaton]

If you really think you want to work on a deployment project, you should 
get the requirements together very quickly, so someone on this list can 
sanity check that it's something both feasible and useful.

I have done a **lot** of different deployments, and other than deploying to 
heroku, they have never been the same. I'm rather skeptical that some tool 
can be built for deploying a Django application that didn't already mimic 
an existing tool like Ansible or Salt (or Heroku CLI).

Examples: what application server (gunicorn, uwsgi, mod_wsgi)? What web 
server (nginx, apache, caddy)? What database server, and is it remote or 
local? How are environment variables securely provisioned and deployed? 
Where is static content served from, and do you need S3 keys? How are you 
managing TLS certificates?

Start a new thread on this list when you have an outline for what you want 
to do. At that stage we should be able to tell you if you should proceed 
with a detailed project scope or not. I don't want you to work on a 
detailed project scope and then get knocked back for the idea being 
infeasible. Remember, the project would need to be very useful for Django 
users.

I'd also just like to call out that projects like Django Channels would 
likely be in scope too, provided the current maintainers felt comfortable 
mentoring a student. A quick note about Channels specifically - the current 
maintainers have **just** taken up that mantle, so you should have a 
specific idea in mind if that's what you're interested in, not ask the team 
for ideas.




On Friday, 15 February 2019 20:53:34 UTC+11, Shashank shet wrote:
>
> That's a good idea. I'll start working on a draft for the deployment 
> support project.
>
> On Friday, February 15, 2019 at 2:58:45 PM UTC+5:30, Carlton Gibson wrote:
>>
>> The timeline for GSoC is here: 
>> https://summerofcode.withgoogle.com/how-it-works/#timeline
>>
>> Applications are ≈ a month away. 
>>
>> Here's an example from a previous year: 
>> https://gist.github.com/chrismedrela/82cbda8d2a78a280a129
>>
>> I'd suggest drafting things in more detail and starting a new thread to 
>> invite discussion. 
>>
>> I don't know what'll be suitable: it depends on the proposal: 
>>
>> - something not in core? Yeah, maybe. 
>> - Something for deployment? Well, seems a bit ambitious, but again 
>> maybe... 
>> - Bringing the two(?) community Redis cache backends into one in core? 
>> Reasonable idea perhaps, yes. 
>> - A cross-db JSONField etc. Yep, great... 
>>
>> ...and so on. What do you want to work on? 
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/350d7430-5644-4faa-8019-6e0968e554f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Use CDN for djangoproject.com

[Josh Smeaton]

Cloudflare have many SSL options, including fully encrypted and
authenticated comms all the way through (terminate and reconnect).
Typically done by having a “hidden” origin domain that also hosts a
certificate. I’m unsure if it’s possible to have both origin and front
hosting the same name so that DNS alone can decide to hit cdn or origin.

Anyway, it seems weird to me to dismiss a CDN offhand “because security”.
Especially considering the size of the providers and the expertise their
teams have.

Cloudflare (fastly, cloudfront, whatever) aren’t some “random TLS”
providers. I would probably go as far to say that putting a CDN in front of
both the docs and the release packages would likely be a net improvement in
security for users.

On Thu, 14 Feb 2019 at 21:58, Tom Forbes  wrote:

> That makes sense, but in this case we are only talking about potentially
> yielding control of the docs subdomain which is not used to serve sensitive
> build artefacts?
>
> Another option is fastly.com, who support other large open source
> projects for free. They essentially give you geographically distributed
> HAProxy instances and you have a lot more control over them. I believe
> several large Linux distributions use them to serve cached apt packages.
>
> Regarding TLS termination, unfortunately any CDN we use will likely need
> to do this for the whole domain to get any benefit. The Django docs are
> text/html heavy with very few, if any, images. So the real speed benefit
> will have to come from serving that, which requires TLS termination (and
> therefore interception) at their end.
>
> On Thu, 14 Feb 2019, 06:32 Markus Holtermann, 
> wrote:
>
>> Hi all
>>
>> to elaborate on what Tobias said: we deliberately have the infrastructure
>> spread across multiple service providers: DNS registry, nameservers,
>> hosting, TLS certificate authority, … None of them have access to
>> everything. The reason is that we offer the download of the release
>> artifacts from the djangoproject.com website. And we would like to
>> ensure that the TLS termination happens by us and not some random service
>> provider. After all, Django is used by enterprises that do have some
>> restrictions on where you're allowed to download software from.
>>
>> By handing over DNS to some CDN provider, we loose the ability to ensure
>> that happens.
>>
>> That said, if there's a CDN that works as a reverse proxy and doesn't
>> require us to hand over control of DNS, I guess we could be interested in
>> moving the docs behind that.
>>
>> /Markus
>>
>> On Thu, Feb 14, 2019, at 2:22 AM, Tobias McNulty wrote:
>> > For me it's the trust factor (allowing someone else to decrypt and
>> > re-encrypt all our data). This may be less of an issue for the docs
>> > site, *if* we don't have to assign DNS authority for the whole domain
>> > to the CDN provider.
>> >
>> > Tobias
>> >
>> >
>> > On Wed, Feb 13, 2019, 7:47 PM Kye Russell > > > I’ve been hearing that there are other CDN providers that offer a
>> very comparable service for a fraction of the cost of CloudFront.
>> > >
>> > > Anyways, at this stage let’s not get bogged down on provider
>> decisions. I’m curious if anyone has any general objections to a CDN of any
>> kind.
>> > >
>> > > It shouldn’t be that big a deal to automatically invalidate when the
>> docs are updated. But I’m sure there’s something I’m missing.
>> > >
>> > > On Thu, 14 Feb 2019 at 8:36 am, Cristiano Coelho <
>> cristianocc...@gmail.com> wrote:
>> > >> Consider AWS's cloudfront then :)
>> > >>
>> > >> El martes, 12 de febrero de 2019, 2:34:09 (UTC-5), Florian Apolloner
>> escribió:
>> > >>> Especially cloudflare is a service we do not want to use. as for
>> the docs only, does the mirror on rtd work better for you? They are
>> probably behind a CDN.
>> > >>>
>> > >>> Cheers,
>> > >>> Florian
>> > >>>
>> > >>> On Tuesday, February 12, 2019 at 6:43:41 AM UTC+1, Cheng C wrote:
>> >  Hi,
>> > 
>> >  Is it possible to utilize a CDN service for djangoproject.com, or
>> at least on docs.djangoproject.com? The site is actually quite fast for
>> me but I think there is still room for improvement. Cloudflare sponsored
>> dozens of open source projects <
>> https://developers.cloudflare.com/sponsorships/>, probably they can
>> provide free service for django as well.
>> > 
>> >  Tested from Melbourne, Australia:
>> > 
>> >  https://www.djangoproject.com/
>> >   Average Ping: 245ms
>> >   Browser: 21 requests, 211KB transferred, Finish: 2.52s,
>> DOMContentLoaded: 1.16s, Load: 1.48s
>> > 
>> >  https://git-scm.com/
>> >   Average Ping: 5ms
>> >   Browser: 42 requests, 351KB transferred, Finish: 717ms,
>> DOMContentLoaded: 564ms, Load: 699ms
>> > 
>> >  Tested on Chrome with "Disable cache" checked (but not the first
>> time visit, so DNS query time might not be included).
>> > 
>> >  Best regards and thanks for all your great work.
>> > >>
>> >
>> >
>> > >>  --
>> > >>  You 

Re: Use CDN for djangoproject.com

[Josh Smeaton]

Why do we not want to use Cloudflare?

FWIW I agree that the docs site performance is not great (also from 
melbourne) - but I'd still suffer the performance hit over going via RTD 
mirrors for familiarity.


On Tuesday, 12 February 2019 18:34:09 UTC+11, Florian Apolloner wrote:
>
> Especially cloudflare is a service we do not want to use. as for the docs 
> only, does the mirror on rtd work better for you? They are probably behind 
> a CDN.
>
> Cheers,
> Florian
>
> On Tuesday, February 12, 2019 at 6:43:41 AM UTC+1, Cheng C wrote:
>>
>> Hi,
>>
>> Is it possible to utilize a CDN service for djangoproject.com, or at 
>> least on docs.djangoproject.com? The site is actually quite fast for me 
>> but I think there is still room for improvement. Cloudflare sponsored 
>> dozens of open source projects 
>> , probably they can 
>> provide free service for django as well.
>>
>> Tested from Melbourne, Australia:
>>
>> https://www.djangoproject.com/
>>  Average Ping: 245ms
>>  Browser: 21 requests, 211KB transferred, Finish: 2.52s, 
>> DOMContentLoaded: 1.16s, Load: 1.48s
>>
>> https://git-scm.com/
>>  Average Ping: 5ms
>>  Browser: 42 requests, 351KB transferred, Finish: 717ms, 
>> DOMContentLoaded: 564ms, Load: 699ms
>>
>> Tested on Chrome with "Disable cache" checked (but not the first time 
>> visit, so DNS query time might not be included).
>>
>> Best regards and thanks for all your great work. 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/285bd588-44a7-4152-a270-7f1be0e5de29%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: include only columns from selected related models in select_related query

[Josh Smeaton]

Personally, I depend on this behaviour, but it's only because I'm aware of 
it. Others would be too though, so we can't just make that change (not that 
you're suggesting that approach).

One approach I think could be good would be:

Book.objects.select_related('author__hometown').defer("author").get(id=4)

Deferring a relation would defer all fields except the key field (which is 
necessary for lazy loading). What do you think?


On Tuesday, 12 February 2019 03:48:00 UTC+11, riccardo.magliocchetti wrote:
>
> Hello Collin, 
>
> Il 11/02/19 17:35, Collin Anderson ha scritto: 
> > So would you "defer" the other columns like "only()"? 
>
> Yeah, something like that 
>
> > If nothing else, you could try using .annotate(F('author__hometown')) 
> (not 
> > sure if that works) or .values('author__hometown') to just get the 
> values 
> > you need. 
>
> Sure but the query would be everything but readable :) 
> The point of a parameter to select_related would be to avoid to type 
> explicitly 
> fields not in select_related and the one in select_related twice. 
>
> > On Mon, Feb 11, 2019 at 5:50 AM Riccardo Magliocchetti < 
> > riccardo.ma...@gmail.com > wrote: 
> > 
> >> Hello, 
> >> 
> >> I'm debugging views leaking lots of memory in django 1.11. It looks 
> like 
> >> there 
> >> is some connections with my usage of select_related(). But that's mail 
> is 
> >> not 
> >> about that, not sure about my findings yet :) 
> >> 
> >> So I have looked again at the select_related documentation here: 
> >> 
> https://docs.djangoproject.com/en/2.1/ref/models/querysets/#select-related 
> >> 
> >> and found this: 
> >> Book.objects.select_related('author__hometown').get(id=4) will cache 
> the 
> >> related 
> >> Person and the related City 
> >> 
> >> Up until now i thought that only the related model i've specified would 
> >> get 
> >> added to selected columns e.g. only the City because of hometown. But 
> it 
> >> looks 
> >> that's not how it is :) 
> >> 
> >> Would it make sense to add a parameter to change select_related 
> behaviour 
> >> to 
> >> include only the columns of the related models specified? That could 
> save 
> >> quite 
> >> a lot of bandwitdh for some use cases. 
> >> 
> >> What do you think? 
> >> 
> >> Thanks 
> >> 
> >> -- 
> >> Riccardo Magliocchetti 
> >> @rmistaken 
> >> 
> >> http://menodizero.it 
> >> 
> >> -- 
> >> You received this message because you are subscribed to the Google 
> Groups 
> >> "Django developers  (Contributions to Django itself)" group. 
> >> To unsubscribe from this group and stop receiving emails from it, send 
> an 
> >> email to django-develop...@googlegroups.com . 
> >> To post to this group, send email to django-d...@googlegroups.com 
> . 
> >> Visit this group at https://groups.google.com/group/django-developers. 
> >> To view this discussion on the web visit 
> >> 
> https://groups.google.com/d/msgid/django-developers/a42050d3-25a2-2f47-c841-918d7d085757%40gmail.com
>  
> >> . 
> >> For more options, visit https://groups.google.com/d/optout. 
> >> 
> > 
>
>
> -- 
> Riccardo Magliocchetti 
> @rmistaken 
>
> http://menodizero.it 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/8db815a1-91a8-4dac-bf60-eeca96a0d71e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Extend FAQ with "How do I get Django and my JS framework to work together?"

[Josh Smeaton]

For the vast majority of users, right now, Vue or React are the only real 
options. Let's not try predicting the future, and just focus on what the 
trends currently are. Users going for newer targets are likely more 
advanced, and can figure out the necessary changes.

Aymerics articles are fantastic, and our team has gotten a lot of value out 
of them for our newer projects. Especially the customisations to static 
files which removes the re-hashing of static content. I would be very keen 
for Django to provide a first class integration method with Webpack, in 
particular, but most popular bundler frameworks if the necessary interfaces 
exist.

- "Importing" static content that has already been bundled and hashed
- Automatic reloading of static content as it is being changed
- Visual debugging in editors

I think the idea of a how-to that considers authentication and 
communication between frontend and backend is also what we need.

So while I don't think Django needs to bless a particular frontend 
framework, I think we have to provide a really nice integration story so 
that the most popular JS frameworks do not have to jump through a lot of 
hoops, including copy and pasting large sections of code, to get their 
existing tooling working with Django. I no longer think it's enough to 
require 3rd party django libraries to make the frontend integration 
possible.

On Sunday, 10 February 2019 04:50:54 UTC+11, Dmitriy Sintsov wrote:
>
> Very nice tuturials! Although there is web components standard gradually 
> being adapted by major browsers, which should bring custom tags / 
> components not having to use the third party Javascript libraries like Vue 
> or React. So, while in Python world the leadership of Django is quite 
> stable and obvious one, in Javascript world it's not so easy to say whether 
> Vue and React would dominate in a long run. Maybe Marko or Stencil.js are 
> closer to the future vision of Javascript.
>
> On Sat, Feb 9, 2019 at 8:32 PM Aymeric Augustin <
> aymeric@polytechnique.org > wrote:
>
>> Hello,
>>
>> I wrote a three-part essay on this question last year:
>> 1. 
>> https://fractalideas.com/blog/making-react-and-django-play-well-together/
>> 2. 
>> https://fractalideas.com/blog/making-react-and-django-play-well-together-hybrid-app-model/
>> 3. 
>> https://fractalideas.com/blog/making-react-and-django-play-well-together-single-page-app-model/
>>
>> Even though I took a narrower view — I only considered React — I found 
>> enough decisions factors to write over 2000 words in the first post, which 
>> is too long for a FAQ :-)
>>
>> On one hand, I'm not sure the Django docs should go into this level of 
>> detail and provide specific information about a particular JS framework. On 
>> the other hand, it's rather useless to talk about integrating Django with a 
>> JS frontend without discussing authentication and it's hard to discuss 
>> authentication in less than 2000 words — which were just for justifying my 
>> favorite solution, not for investigating every option!
>>
>> I think we need a how-to guide rather than a FAQ entry. I would find it 
>> nice:
>>
>> *A. To describe the Singe Page App model — where Django only serves the 
>> API*
>>
>> We need to explain CORS and CSRF in the clearest terms possible. Many 
>> devs end up shotgun-debugging CORS or CSRF errors, which always results in 
>> insecure deployments.
>>
>> My consulting experience suggests that we have a problem there: I never 
>> did an audit where the client got that right, even though they're all smart 
>> people trying to get things right.
>>
>> Perhaps a condensed version of my third post could do the job?
>>
>> Some may disagree with my recommendation against JWT, which may too 
>> opinionated for the Django docs. Again, in my experience, people tend to 
>> get security more wrong with JWT, which is why I prefer discouraging it and 
>> letting those who know what they're doing ignore my advice.
>>
>> *B. To say something about integrating a modern JS framework with 
>> django.contrib.staticfiles *
>>
>> It's perfectly doable and provides all the benefits of 
>> django.contrib.staticfiles. However, it requires a bit of duct tape, as 
>> shown in my second post.
>>
>> I'm a huge fan of this technique for simple website but I'm afraid I'm 
>> biased by my experience with Django. This is unlikely to be a popular 
>> option for those who are more familiar with a modern frontend framework 
>> than with django.contrib.staticfiles.
>>
>> The docs should at least give the general idea of "compile your frontend 
>> to somewhere Django can find the files, then run collectstatic".
>>
>> If someone starts writing documentation about this, I'm interested in 
>> reviewing it.
>>
>> Best regards,
>>
>> -- 
>> Aymeric.
>>
>>
>>
>> On 5 Feb 2019, at 11:17, Carlton Gibson > > wrote:
>>
>> I think this topic is very interesting. 
>>
>> Two sides of it: 
>>
>> * Static files handling
>> * APIs
>>
>> Curtis is right, there are 

Re: Proposal to re-open #27017 (updating only dirty fields in save())

[Josh Smeaton]

That's a +1 from me. I've certainly hit the bugs you've mentioned before, 
and I can't think of a good reason not to do dirty field tracking, 
especially if it were to be opt in.

Bikeshedding a little bit, rather than having a global setting, I'd rather 
see an option on Options/Meta so that opt in is per-model. On a related 
note, I'd like to see some real world testing of any solution with packages 
that implement polymorphic models.

On Tuesday, 29 January 2019 01:00:21 UTC+11, Daniel Tao wrote:
>
> Hi!
>
> This is my first post on this list. I recently left a comment on #27017 
>  requesting to 
> re-open the topic of only saving dirty fields in save(). Tim Graham 
> helpfully directed to #4102  and 
> advised that I make the proposal on the dev mailing list, so that's what 
> I'm doing :)
>
> I've gone through the history of #4102 and taken notes on the challenges 
> that arose when this was first attempted 12 years ago (!). The TL;DR is 
> that, while there were indeed quite a few complications, I don't see 
> anything that came up that should be considered a flat-out showstopper. 
> Rather, what happened was that at about the 69th comment 
> , back in 2012, 
> the conversation shifted as Matt Long made this observation:
>
> what started as a simple opt-in feature request of adding a field 
>> white-list to the Model's save function morphed into a complicated dirty 
>> flag approach that obviously has many edge cases and performance 
>> implications given that this ticket has been open for 5 years now
>
>
> From here it seems things progressed towards the current solution of 
> supporting the update_fields argument, and that's where things ended. I 
> would like to point out that Matt did *not* advocate for completely 
> abandoning all efforts to support dirty field tracking; to the contrary, in 
> the same comment he said this (emphasis mine):
>
> Clearly some people feel differently and favor the dirty flag approach for 
>> a more hands-off approach. As such, I propose adding support for *both 
>> methods*
>
>
> With that in mind, I believe it's worth re-opening this discussion. For a 
> fairly lengthy justification, see my aforementioned comment on #27017 
> . I'll copy the 
> effective TL;DR of the proposal here for convenience:
>
> In my opinion Django could make most code bases inherently more resilient 
>> against latent race conditions by implementing some form of dirty field 
>> tracking and effectively providing the functionality of update_fields 
>> automatically. I would like to propose a new setting, something like 
>> SAVE_UPDATE_DIRTY_FIELDS_ONLY, to change the ORM's default behavior so 
>> that calls to Model.save() only update the fields that have been set on 
>> the model instance. Naturally for backwards compatibility this setting 
>> would be False by default.
>
>
> As for the concerns that were raised when this was first attempted, I will 
> now attempt to summarize what I found along with, in most cases, a bit of 
> editorializing from me.
>
> Performance
>
> The performance angle was first explored in a comment 
>  that said it 
> "doesn't look good" and provided some benchmarks showing a performance hit 
> from 0.17s to 2.64s for setting an attribute using the timeit 
>  package. I didn't see 
> anyone point out that the timeit method defaults to executing code *a 
> million times*; so the throughput of the operation went from about 6 
> million to closer to 400 thousand times per second. (The percentage change 
> is indeed significant, but this doesn't *smell* like a potential 
> bottleneck.)
>
> It was noted in a couple 
>  places 
>  that it seems 
> potentially shortsighted to focus so much on the performance of getting and 
> setting attributes without taking into account the potential performance 
> *benefits* of executing smaller UPDATE statements that write fewer 
> columns. As far as I can tell, no one in the thread on #4102 actively 
> investigated the latter.
>
> Based on the unlikelihood of attribute setting representing a performance 
> bottleneck, and the lack of data for the performance impact of executing 
> smaller updates, I would consider the performance discussion largely a 
> distraction. Though I do think it's worth measuring the latter.
>
> Compatibility
>
> It was observed that the two approaches considered on the ticket 
> (overriding __setattr__ or defining custom property setters) would not 
> work with obj.__dict__.update 
> , which is 
> apparently an optimization you can find 

Re: Google Summer of Code 2019

[Josh Smeaton]

Other ideas can be found in the DEPS repo 
too: https://github.com/django/deps/pulls

The only one of those 3 that would be GSOC doable (IMO) would be the query 
expression language one.

On Friday, 25 January 2019 01:43:13 UTC+11, Carlton Gibson wrote:
>
> Perhaps it's partly the GSoC doesn't cross the radar until just a few 
> weeks before the deadline... 
>
> I'm happy to help mentor but also Django Core Mentorship is there...
> https://groups.google.com/forum/#!forum/django-core-mentorship
>
>
> One idea for a good project might be adding a cross DB JSONField as per 
> this thread
>
>
> https://groups.google.com/d/topic/django-developers/zfred27yVPg/discussion
>
> * All the supported DBs have native JSON support. 
> * SQLite is the only one where we don't have a Django model field already 
> to work on. 
>* That would be first step as PoC I'd guess. 
> * Then, how can we unify? 
> * And, a migration path (from contrib.postgres) 
>
> I don't know if that's perfect but it strikes me as eminently do-able. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5f9a5198-fd0d-470d-a6a9-81be3c8de87f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: revisiting the Python version support policy

[Josh Smeaton]

Don't discount being able to use features from newer versions of python 
within Django itself.

https://docs.python.org/3/whatsnew/3.6.html

- dicts are more performant
- dicts/kwargs/class attributes are ordered (cpython implementation detail 
for 3.6 - allowing us to consider removing descriptor counters)
- fstrings
- type annotations (something some people are quite in favour of)
- async comprehensions and generators (less important for Django right now 
- may be more important for Channels)
- secrets module
- pathlib
- descriptor improvements (set_name, __init_subclass__)

I'm more in favour of maintaining the existing policy than playing 
favourites with distro support, but not strongly so. The LTS Django is 
already covering Python 3.5 for 18 months **longer** than the EOL. I don't 
think the newest versions of Django need to be so concerned with distro 
compatibility.

On Wednesday, 23 January 2019 02:03:54 UTC+11, Federico Capoano wrote:
>
> I would ask: what are the pros and cons of dropping support for python 3.5?
>
> I think allowing users to easily use and install django based applications 
> is more important than strictly follow a python version support policy.
>
> I think that if we drop support for python 3.5, which is the default 
> python version on many linux platforms right now, we will make the life of 
> our users and developers harder.
> I don't understand the reason for doing so, if we have to do it for a good 
> reason, like a security issue, or because django has to take advantage of 
> features that are available only from python 3.6 onwards, I would be in 
> favour, but if we have to do it only because the policy says so, without 
> any other advantage, I would amend the policy.
>
> My 2 cents.
>
> Thanks for your hard work maintaining django
> Federico 
>
>
> On Monday, January 21, 2019 at 10:56:40 AM UTC-5, Tim Graham wrote:
>>
>> When deciding when to drop support for Python 2 in Django, there was 
>> consensus to adopt this Python version support policy [0]: "Typically, we 
>> will support a Python version up to and including the first Django LTS 
>> release whose security support ends after security support for that version 
>> of Python ends. For example, Python 3.3 security support ends September 
>> 2017 and Django 1.8 LTS security support ends April 2018. Therefore Django 
>> 1.8 is the last version to support Python 3.3."
>>
>> Since then, we didn't abide by this policy when dropping Python 3.4, 
>> mainly because Debian stable still used Python 3.4 at the time and Claude 
>> argued that some people like him would have difficulty contributing to 
>> Django if they had to install another version of Python [1].
>>
>> Based on the policy, it's time to drop support for Python 3.5 in the 
>> master branch (Django 3.0) -- with Django 2.2 LTS supported until April 
>> 2022 and Python 3.5 supported until September 2020). I created a ticket [2] 
>> and PR [3] for dropping support for Python 3.5 [2], however, Claude 
>> commented, "I'm not so enthusiast to drop Python 3.5 now (it is still the 
>> default version in Debian stable). Couldn't this be done in Django 3.1 
>> instead?"
>>
>> Are you in favor of amending the Python support version policy to account 
>> for the Python version in Debian stable?
>>
>> [0] 
>> https://docs.djangoproject.com/en/dev/faq/install/#what-python-version-can-i-use-with-django
>> [1] 
>> https://groups.google.com/d/msg/django-developers/4rbVKJYm8DI/TTh3i04pBQAJ
>> [2] https://code.djangoproject.com/ticket/30116
>> [3] https://github.com/django/django/pull/10864
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a8e5bc63-15b1-4c77-92ae-59e61daa30fd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Looking for feedback] Make Admin raw_id_fields the default choice for FKs, or project-wide configurable

[Josh Smeaton]

If we were happy with that particular implementation, then I'd prefer 
adding it as an official subclass thats importable for users rather than 
just dumping the code to the docs. But I guess the issue is a slippery 
slope - how many subclasses do we add for various ModelAdmin use cases. 
It's definitely an issue that bites many people, and I'd like to see some 
way forward.

On Friday, 18 January 2019 03:52:52 UTC+11, Santiago Basulto wrote:
>
> Ok, sorry for the Fake News, seems like it's not so complicated to make 
> one ModelAdmin parent class that provides this behavior. Here's a working 
> example:
>
>
> from django.contrib import admin
>
> from django.contrib.admin import widgets
>
>
> class RawFieldModelAdmin(admin.ModelAdmin):
> def formfield_for_foreignkey(self, db_field, request, **kwargs):
> db = kwargs.get('using')
> if 'widget' not in kwargs:
> if db_field.name not in (self.get_autocomplete_fields(request
> ), self.radio_fields):
> kwargs['widget'] = widgets.ForeignKeyRawIdWidget(db_field.
> remote_field, self.admin_site, using=db)
>
>
> return super().formfield_for_foreignkey(db_field, request, **
> kwargs)
>
> Do you folks think we should add this to the docs? I still think that 
> having a one-off setting for the "default foreign key" widget would be 
> valuable, at least for me as a user.
>
> What do you think?
>
> On Thursday, January 17, 2019 at 11:27:12 AM UTC-5, Santiago Basulto wrote:
>>
>> I think the proposed solution of "you can just extend/subclass 
>> ModelAdmin" doesn't work, because the fields on different models can have 
>> different names. I can't just write one global ModelAdmin and then use it 
>> for all my models, because they'll have different names for their fields. 
>> Or if it works, it'll need A LOT of introspection (to dynamically check 
>> which fields are FKs and making them part of raw_id_fields).
>>
>> Maybe I'm wrong and I'm missing the point, do you folks have an 
>> implementation of that ModelAdmin superclass to show?
>>
>> On Thursday, January 17, 2019 at 11:00:42 AM UTC-5, Carlton Gibson wrote:
>>>
>>>
>>>
>>> On Thursday, 17 January 2019 16:14:31 UTC+1, Collin Anderson wrote:

 One problem with any of the alternatives (besides making it readonly by 
 default) is that it requires the other model to be registered in the admin

>>>
>>> Off-hand I don't follow you here. Can you explain. 
>>>
>>>  
>>>
 I hope there's _something_ we can do to somehow improve the situation. 
 Maybe we could at least improve the examples in the documentation? Maybe 
 give an example in the docs of a ModelAdmin subclass that defaults to 
 using 
 raw_id?

>>>
>>> An example definitely.
>>>
>>> Maybe we could add an attribute to ModelAdmin with a number: More than 
>>> this use raw_id — but what would that look like? 
>>> (Easy subclass rather than a setting...)
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ec47fd5a-5c34-47ad-805d-d2036d430aa3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: What do you think about unify templates feature?

[Josh Smeaton]

Are you 
using 
https://docs.djangoproject.com/en/2.1/ref/templates/api/#django.template.loaders.cached.Loader
 
in your tests? And you should find a way to test rendering from the command 
line, which will allow you to get much better timing information.

On Friday, 18 January 2019 20:40:29 UTC+11, Goinnn wrote:
>
> I answer between lines
>
>
> El vie., 18 ene. 2019 a las 5:03, shiva kumar ( >) escribió:
>
>> Could you please explain about it. I hadn't understood ur idea.
>>
>
>
>1. Currently: with you render a view, this render a template. But 
>usually, this template extends of another and include others. So Django 
> has 
>to get these templates to create an HTML. This process get time.
>2. My proposal: Preprocess templates for some (or all) them. Creating 
>a template without extends or includes. So, in my previous email you can 
>see, Django render less templates, so Django win time.
>
> We can get a backend a  little faster with very little effort.
>
> Best!
>
>  
>
>> On Thu 17 Jan, 2019 2:32 pm J. Pablo Martín Cobos >  wrote:
>>
>>> Hi,
>>>
>>> From one year ago, I am using an own command for Django templates that 
>>> unify them. With an example it is easy to see. If I am to render for 
>>> example a template call news.html like it:
>>>
>>> 1. news.html
>>>
>>> {% extends "base.html" %}
>>>
>>> {% block title %}
>>> {% include "inc.news.title.html" %}
>>> {% endblock %}
>>>
>>> {% block content %}
>>> {% for news_item in news %}
>>> {{ news_item.title }}
>>> {{ news_item.subtitle }}
>>> {% endfor %}
>>> {% endblock %}
>>>
>>> 2. base.html
>>>
>>> >> http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd;>
>>> http://www.w3.org/1999/xhtml; lang="{{ 
>>> LANGUAGE_CODE|default:"en-us" }}" {% if LANGUAGE_BIDI %}dir="rtl"{% endif 
>>> %}>
>>> 
>>> {% block title %}{% endblock %}
>>> 
>>> 
>>> {% block content %}{% endblock %}
>>> 
>>> 
>>>
>>> 3. inc.news.title.html
>>> News
>>>
>>> With this command I preproces every template of a settings variable and 
>>> I get something like this:
>>>
>>> news.unify.html
>>>
>>> >> http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd;>
>>> http://www.w3.org/1999/xhtml; lang="{{ 
>>> LANGUAGE_CODE|default:"en-us" }}" {% if LANGUAGE_BIDI %}dir="rtl"{% endif 
>>> %}>
>>> 
>>> News
>>> 
>>> 
>>> {% for news_item in news %}
>>> {{ news_item.title }}
>>> {{ news_item.subtitle }}
>>> {% endfor %}
>>> 
>>> 
>>>
>>> So I have a two improves:
>>>
>>>1. It is more fast. And in a real project a view can render easyly 
>>>50 templates
>>>2. I use news.html to develop and news.unify.html to production. So 
>>>I don't lose legilibility.
>>>
>>>
>>> What do you think about "unify templates feature"? Do you know if exists 
>>> a similar public project in github/gitlab/bitbucket etc?
>>>
>>>
>>> Best,
>>>
>>> -- 
>>> Pablo Martín Cobos
>>> Computer engineer
>>> Python/Django developer
>>> 652 53 37 36
>>> goi...@gmail.com 
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com .
>>> To post to this group, send email to django-d...@googlegroups.com 
>>> .
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/CALNyWLGNcuK8DTnU9w9fyGFhFfT3dAz7vfj3B%2BnDHWTfneLNFw%40mail.gmail.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/CAMsYeuFV3EV_3H%2BR0q9xi5Q6EQq6L2cLkatcvrTtSYW%3D2zRk7A%40mail.gmail.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> -- 
> Juan Pablo Martín Cobos
> Ingeniero informático
> Desarrollador Python/Django
> 652 53 37 36all 
> 

Re: Widening participation (Thoughts from DjangoCon)

[Josh Smeaton]

For what it's worth, I agree. I think we should consider using GitHub 
issues. I don't think there's anything in Trac, from a user perspective, 
that we couldn't really do with Issues. The main issue, I think, would be 
allowing non-committers (organisational members) to triage tickets and 
change the ticket status. 

But I don't have the time to plan or champion a change like that through 
these days, so while I'd like to see it happen, I'm not the person to push 
for it to happen.

Jamesie: please see "Why not Gitlab?" [0]. While it might be better from a 
technical standpoint, it would not be better than GH Issues from an 
onboarding perspective.

[0]https://www.python.org/dev/peps/pep-0581/#why-not-gitlab 

On Tuesday, 11 December 2018 23:20:21 UTC+11, Tom Forbes wrote:
>
> > The only reason Github issues would be a consideration is if the group 
> thought the onboarding experience (being where users already are with a 
> tool they're already familiar with) would have more value than sticking 
> with with the status quo which is strictly better from a feature 
> perspective than GH Issues
>
> I really think it's worth considering - I'm not convinced Trac is strictly 
> better from a feature perspective. Things like the dashboards are 
> interesting, sure, but what does it do that Github issues don't? And by 
> this I don't mean features that exist but we don't use, I mean our general 
> ticket workflow that 99% of contributors are exposed to. There are far 
> larger projects than Django currently using Github issues successfully and 
> they have a far more complex workflow including CLA's, high-granularity 
> tags, automatic reviewer selection, automatic closing, etc etc.
>
> The cost of a migration would not be cheap, but is it worth doing a formal 
> evaluation and consider it? The "why Github section" (
> https://www.python.org/dev/peps/pep-0581/#why-github) in that PEP lists 
> some good reasons that apply to us as well. The key one for me is lowering 
> the barrier to entry which I think we should be aiming towards. If we are 
> looking to expand the development group and get more people onboard we may 
> need to shake things up, as the status quo is problematic in that area.
>
> On 11 December 2018 at 12:04:32, Josh Smeaton (josh.s...@gmail.com 
> ) wrote:
>
> I don't think something like Jira would even be a consideration.  
>
> The only reason Github issues would be a consideration is if the group 
> thought the onboarding experience (being where users already are with a 
> tool they're already familiar with) would have more value than sticking 
> with with the status quo which is strictly better from a feature 
> perspective than GH Issues. Incrementally improving Trac can improve the 
> value prop for not changing, but I don't think anyone that actually uses 
> Trac would say it's worse from a feature perspective. Even if that value 
> judgement did land on GH issues side, there would still be a large 
> operational undertaking to make that transition. 
>
> Again, the PEP for doing this for CPython would be very similar for 
> Django, so please refer to https://www.python.org/dev/peps/pep-0581/ for 
> some examples of why and what operational concerns there can be (migration, 
> permissions, etc).
>
> On Tuesday, 11 December 2018 07:26:58 UTC+11, Ira Abbott wrote: 
>>
>> Apologies for the double post - my last one was not clear.  "just that" 
>> means that the payment is intended to indicate that it is worth a JIRA 
>> license to me to not use JIRA.
>>
>> This group does great things.  I am sure that the group can come up with 
>> some interesting ways to scale that will ultimately benefit the framework 
>> as a whole.
>>
>> On Monday, December 10, 2018 at 3:21:53 PM UTC-5, Ira Abbott wrote: 
>>>
>>> FWIW: Please consider my contribution of $84 (one bulk JIRA license for 
>>> one year) to be just that.
>>>
>>> On Monday, December 10, 2018 at 2:14:07 PM UTC-5, Zachary Garwood wrote: 
>>>>
>>>> I'd pay money to NOT use Jira.
>>>>
>>>> On Mon, Dec 10, 2018, 11:09 AM Dan Davis >>>
>>>>> Tom, you are right about the UX issues, but full-text and inverted 
>>>>> indexing would help with the responsiveness as well.  Technically, I was 
>>>>> talking about djangoproject's TracSearch 
>>>>> <https://code.djangoproject.com/search>.  That's closer to good UX as 
>>>>> well, because you can get there by progressive enhancement rather than 
>>>>> taking stuff away.
>>>>>
>>>>> You are also right that switching to another system such as github 
>>>&g

Re: Widening participation (Thoughts from DjangoCon)

[Josh Smeaton]

I don't think something like Jira would even be a consideration. 

The only reason Github issues would be a consideration is if the group 
thought the onboarding experience (being where users already are with a 
tool they're already familiar with) would have more value than sticking 
with with the status quo which is strictly better from a feature 
perspective than GH Issues. Incrementally improving Trac can improve the 
value prop for not changing, but I don't think anyone that actually uses 
Trac would say it's worse from a feature perspective. Even if that value 
judgement did land on GH issues side, there would still be a large 
operational undertaking to make that transition. 

Again, the PEP for doing this for CPython would be very similar for Django, 
so please refer to https://www.python.org/dev/peps/pep-0581/ for some 
examples of why and what operational concerns there can be (migration, 
permissions, etc).

On Tuesday, 11 December 2018 07:26:58 UTC+11, Ira Abbott wrote:
>
> Apologies for the double post - my last one was not clear.  "just that" 
> means that the payment is intended to indicate that it is worth a JIRA 
> license to me to not use JIRA.
>
> This group does great things.  I am sure that the group can come up with 
> some interesting ways to scale that will ultimately benefit the framework 
> as a whole.
>
> On Monday, December 10, 2018 at 3:21:53 PM UTC-5, Ira Abbott wrote:
>>
>> FWIW: Please consider my contribution of $84 (one bulk JIRA license for 
>> one year) to be just that.
>>
>> On Monday, December 10, 2018 at 2:14:07 PM UTC-5, Zachary Garwood wrote:
>>>
>>> I'd pay money to NOT use Jira.
>>>
>>> On Mon, Dec 10, 2018, 11:09 AM Dan Davis >>
 Tom, you are right about the UX issues, but full-text and inverted 
 indexing would help with the responsiveness as well.  Technically, I was 
 talking about djangoproject's TracSearch 
 .  That's closer to good UX as 
 well, because you can get there by progressive enhancement rather than 
 taking stuff away.

 You are also right that switching to another system such as github 
 issues could be a better fix than customizing Trac.   But I find it 
 difficult to believe that Github issues grows up to cover as much workflow 
 management as Trac issues or JIRA.   Maybe we need money to get JIRA.


 -- 
 You received this message because you are subscribed to the Google 
 Groups "Django developers (Contributions to Django itself)" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to django-develop...@googlegroups.com.
 To post to this group, send email to django-d...@googlegroups.com.
 Visit this group at https://groups.google.com/group/django-developers.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/django-developers/CAFzonYZ%3DwRCX3HpfYCj0G2i3UrcQmVCDj%3Da%2BOb7zeqtvQm-pxg%40mail.gmail.com
  
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/127550d4-b2dd-4ad7-8e6e-b56a66080194%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: A faster paginator for django

[Josh Smeaton]

I think most people would typically sort/paginate on the primary key field, 
which still exhibits a linear scan, where 
the first few pages are fast and the last few pages take significantly 
longer. Just wanted to call that out in case there
were listeners thinking an index solves the problem. Sorting by an 
unindexed field just makes a bad query a horrible one.

On Thursday, 6 December 2018 07:32:47 UTC+11, Curtis Maloney wrote:
>
> On 12/5/18 8:30 PM, Saleem Jaffer wrote: 
> > Hi all, 
> > 
> > The default paginator that comes with Django is inefficient when dealing 
> > with large tables. This is because the final query for fetching pages 
> > uses "OFFSET" which is basically a linear scan till the last index of 
> > the current page. Does it make sense to have a better paginator which 
> > does not use "OFFSET". 
> > 
> > If this sounds like a good idea, I have some ideas on how to do it and 
> > with some help from you guys I can implement it. 
>
> There are a number of alternatives to this, as well as low-cost 
> solutions to improve OFFSET / LIMIT pagination. 
>
> By adding an index to the sorting field(s), it can drastically improve 
> the "simple" case. 
>
> Beyond that, you start getting into cases with more significant 
> trade-offs.  I know Matthew Schinckel was recently working on a drop-in 
> replacement paginator that used "keyset pagination" 
> https://bitbucket.org/schinckel/django-keyset-pagination 
>
> There's a lot of published work on this topic, and I'd be very 
> interested to see, at least, a library implementing some of these 
> alternatives. 
>
> At the very least, we could want to ensure the documentation recommends 
> indexing on the ORDERing fields, where possible. 
>
> -- 
> Curtis 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ea4f87d6-5b9a-4f82-9486-7a7f4b094607%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Explore integrating django-docker-box in some way?

[Josh Smeaton]

Size of the image could definitely be a concern, especially at sprints 
where wifi speeds aren't always optimal. The django-box image is 
significantly larger though so it'd still be a net win. There are also 
optimisations that can be made to the image for reducing size over time, so 
I'd fully expect it to come down. I've spent a little bit of time trying to 
optimise a $work$ python docker file, I'll provide what I've got as an 
issue to possibly look at.

I see that the ticket has been accepted and I think that's a great step 
forward. I'd also like to hear from the infrastructure team what their 
thoughts on using docker over customised build environments would be.

Florian, Tim, Markus .. any thoughts? (Apologies, I've missed some, this 
list of names is from memory).

On Wednesday, 5 December 2018 10:39:16 UTC+11, Tom Forbes wrote:
>
> Thank you for the reply Josh. I didn’t anticipate any suggestions for 
> including this inside core but off the back of your suggestion I’ve made a 
> ticket here: https://code.djangoproject.com/ticket/30010.
>
> I don’t think it should be complex at all to include this inside Django - 
> it’s four or five new files at most. Hopefully this should improve the 
> experience at sprints, however the current Dockerfile weighs in at 650+mb 
> so the problem may switch from ‘it is hard to set up an environment’ to ‘it 
> is hard to download one’!
>
>
>
>
> On 5 November 2018 at 23:02:30, Josh Smeaton (josh.s...@gmail.com 
> ) wrote:
>
> I'm sorry I haven't had the time to review or contribute yet, but I think 
> it'll be a very useful project - especially for new contributors that might 
> have a little docker experience. The current vagrant solution is heavy, 
> does not work properly on windows and some linuxes, and isn't that easy to 
> maintain or deploy. I'd be in favour of adding the docker files directly to 
> django/django to minimise setup burden (DJANGO_PATH), and improving the 
> contributing docs to show users how to test using docker. 
>
> One of the hardest things I found at sprints was getting development 
> environments setup to effectively contribute - even using the docker-box 
> project which I understand quite well. Anything we can do to improve that 
> situation will be very beneficial.
>
> I have fewer opinions about the official CI story, hopefully some of the 
> infrastructure team can comment more on that. I think that replacing the 
> ansible roles with a docker setup can have some definite improvements and 
> open up CI tasks to a larger pool of people (anyone that can edit docker 
> files), but it'd come with maintaining the host that runs docker (cleaning 
> up images, dealing with disk space issues, etc).
>
>
> On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote: 
>>
>> Hello all,
>>
>> I’ve been working on a docker-compose based alternative to django-box 
>> (imaginatively named django-docker-box) over the last month and it 
>> finally appears to be mostly complete.
>>
>> For reference the tool is just a Dockerfile and a docker-compose 
>> definition that is able to run a complete test matrix of every supported 
>> Python and DB version. It’s as simple as docker-compose run sqlite. You 
>> can see a full test run (excluding oracle) here: 
>> https://travis-ci.com/orf/django-docker-box/builds/90167436
>>
>> Florian suggested I create a thread here to gather feedback and discuss 
>> any potential future directions for the project, so here goes:
>>
>> Firstly I’d like to know if there is any support for moving this under 
>> the Django project itself, maybe even as a replacement for django-box? I 
>> think the setup is pretty quick compared to django-box and is more flexible 
>> in terms of database version support as well as working with Oracle. I’d 
>> also really like some help improving Oracle support if anyone has the time!
>>
>> Secondly is there any support for integrating this with our current 
>> Jenkins setup? I think it would be pretty neat to have parity between what 
>> runs on the CI and what we can run locally and have any improvements shared 
>> between both. Perhaps a full matrix run (which right now is 66 different 
>> environments) is out of the question but a smaller subset could be good?
>>
>> Thirdly, and this is a bit wild, but what about using this to reduce the 
>> burden of running Jenkins by running the tests on a managed CI service like 
>> Travis CI? We would likely still need Jenkins due to issues with Oracle and 
>> running tests on Windows (unless 
>> https://github.com/django/django/pull/10259 works with Docker!), but we 
>> could offload some of the environments onto a third party serv

Re: TestCase.setUpTestData in-memory data isolation.

[Josh Smeaton]

Our project also suffers extensively with mutating objects assigned from 
setUp, preventing us from moving most of our tests to setUpTestData. I'll 
likely begin using your pypi package right away, thanks Simon!

Backward compat issues are probably likely - but they'd be in test cases 
exclusively, making them extremely easy to find during an upgrade. That 
said, a deprecation warning is probably the most sensible path forward to 
prevent the need for immediate action.

Is there anyway to determine the pickle-ability of something without just 
trying to pickle it? I wouldn't be keen on that overhead. Could you just 
capture any copy exceptions, raise a deprecation warning, and abandon the 
copy for that attribute?

On Saturday, 24 November 2018 14:29:33 UTC+11, charettes wrote:
>
> Dear developers,
>
> Django 1.8 introduced the `TestCase.setUpTestData()` class method as a 
> mean to
> speed up test fixtures initialization as compared to using `setUp()`[0].
>
> As I've come to use this feature and review changes from peers using it in
> different projects the fact that test data assigned during its execution
> couldn't be safely altered by test methods without compromising test 
> isolation
> has often be the source of confusion and frustration.
>
> While the `setUpTestData` documentation mentions this limitation[1] and 
> ways to
> work around it by using `refresh_from_db()` in `setUp()` I believe it 
> defeats
> the whole purpose of the feature; avoiding unnecessary roundtrips to the
> database to speed up execution. Given `TestCase` goes through great 
> lengths to
> ensure database level data isolation I believe it should do the same with 
> class
> level in-memory data assigned during `setUpTestData`.
>
> In order to get rid of this caveat of the feature I'd like to propose an
> adjustment to ensure such in-memory test data isolation.
>
> What I suggest doing is wrapping all attributes assigned during 
> `setUpTestData`
> in descriptors that lazily return `copy.deepcopy()`ed values on instance
> attribute accesses. By attaching the `deepcopy()`'s memo on test instances 
> we
> can ensure that the reference graph between objects is preserved and thus
> backward compatible.
>
> In other words, the following test would pass even if `self.book` is a deep
> copy of `cls.book`.
>
> class BookTests(TestCase):
> @classmethod
> def setUpTestData(cls):
> cls.author = Author.objects.create()
> cls.book = cls.author.books.create()
>
> def test_relationship_preserved(self):
> self.assertIs(self.book.author, self.author)
>
> Lazily returning `deepcopy'ies and caching returned values in `__dict__` à 
> la
> `cached_property` should also make sure the slight performance overhead 
> this
> incurs is minimized.
>
> From a check against a few projects and Django's test suite[2] I have only
> identified a single issue which is that attributes assigned during
> `setUpTestData` would now have to be `deepcopy()`able but it shouldn't be
> a blocker given `Model` instance are.
>
> In order to allow other possible issues from being identified against 
> existing
> projects I packaged the proposed feature[3] and made it available on 
> pypi[4]. It
> requires decorating `setUpTestData` methods but it shouldn't be too hard to
> apply to your projects if you want to give it a try.
>
> Given this reaches consensus that this could be a great addition I'd file
> a ticket and finalize what I have so far[2].
>
> Thank your for your time,
> Simon
>
> [0] 
> https://docs.djangoproject.com/en/1.8/releases/1.8/#testcase-data-setup
> [1] 
> https://docs.djangoproject.com/en/2.1/topics/testing/tools/#django.test.TestCase.setUpTestData
> [2] 
> https://github.com/charettes/django/compare/setuptestdata...charettes:testdata
> [3] https://github.com/charettes/django-testdata
> [4] https://pypi.org/project/django-testdata/
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/df0991aa-18d3-4fcd-999d-7226a64faf75%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add Python 3.7 support for Django 1.11?

[Josh Smeaton]

Good merge!

Should this also be a policy change, or is it better to maintain a position 
of "if it's relatively easy and unobtrusive"?

On Sunday, 18 November 2018 10:08:43 UTC+11, Tim Graham wrote:
>
> Okay, I've merged the PR, https://github.com/django/django/pull/10654.
>
> On Saturday, November 17, 2018 at 12:11:17 PM UTC-5, Markus Holtermann 
> wrote:
>>
>> Agreed, let's add official 3.7 support. 
>>
>> /Markus 
>>
>> On Sat, Nov 17, 2018, at 1:15 PM, Adam Johnson wrote: 
>> > Since it's about 3 lines in django itself, I think it's a good idea to 
>> > backport and save users the pain. 
>> > 
>> > On Fri, 16 Nov 2018 at 15:37, Ramiro Morales  wrote: 
>> > 
>> > > On Fri, Nov 16, 2018 at 12:32 PM Tom Forbes  wrote: 
>> > > 
>> > >> Do we have an idea of how many fixes would need to be backported? 
>> > >> 
>> > > 
>> > > 
>> > > 
>> https://github.com/django/django/compare/stable/1.11.x...moneymeets:moneymeets/1.11.16-py37
>>  
>> > > 
>> > > 
>> > >> Visit this group at 
>> https://groups.google.com/group/django-developers. 
>> > >> To view this discussion on the web visit 
>> > >> 
>> https://groups.google.com/d/msgid/django-developers/CAFNZOJNqLQtq03ee-Sfc5v5z1YzETbxu%3D-bWN9FQk0%3D5Yd1Whg%40mail.gmail.com
>>  
>> > >> <
>> https://groups.google.com/d/msgid/django-developers/CAFNZOJNqLQtq03ee-Sfc5v5z1YzETbxu%3D-bWN9FQk0%3D5Yd1Whg%40mail.gmail.com?utm_medium=email_source=footer>
>>  
>>
>> > >> . 
>> > >> For more options, visit https://groups.google.com/d/optout. 
>> > >> 
>> > > 
>> > > 
>> > > -- 
>> > > Ramiro Morales 
>> > > @ramiromorales 
>> > > 
>> > > -- 
>> > > You received this message because you are subscribed to the Google 
>> Groups 
>> > > "Django developers (Contributions to Django itself)" group. 
>> > > To unsubscribe from this group and stop receiving emails from it, 
>> send an 
>> > > email to django-develop...@googlegroups.com. 
>> > > To post to this group, send email to django-d...@googlegroups.com. 
>> > > Visit this group at https://groups.google.com/group/django-developers. 
>>
>> > > To view this discussion on the web visit 
>> > > 
>> https://groups.google.com/d/msgid/django-developers/CAO7PdF_VpdxxCPkLSh3GHCA6svsoCKa7zb0WiLnxuZFSPQe4%2Bw%40mail.gmail.com
>>  
>> > > <
>> https://groups.google.com/d/msgid/django-developers/CAO7PdF_VpdxxCPkLSh3GHCA6svsoCKa7zb0WiLnxuZFSPQe4%2Bw%40mail.gmail.com?utm_medium=email_source=footer>
>>  
>>
>> > > . 
>> > > For more options, visit https://groups.google.com/d/optout. 
>> > > 
>> > 
>> > 
>> > -- 
>> > Adam 
>> > 
>> > -- 
>> > You received this message because you are subscribed to the Google 
>> > Groups "Django developers  (Contributions to Django itself)" group. 
>> > To unsubscribe from this group and stop receiving emails from it, send 
>> > an email to django-develop...@googlegroups.com. 
>> > To post to this group, send email to django-d...@googlegroups.com. 
>> > Visit this group at https://groups.google.com/group/django-developers. 
>> > To view this discussion on the web visit 
>> > 
>> https://groups.google.com/d/msgid/django-developers/CAMyDDM1VANrdq0M70N4%3DPX1b2zzuCguz83u82DM6s%3DJS1FXWfQ%40mail.gmail.com.
>>  
>>
>> > For more options, visit https://groups.google.com/d/optout. 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e8be3ceb-8866-43c3-8588-4a77d0021718%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Allow querying JSONField with F objects #29769

[Josh Smeaton]

Hi Mani,

No response can mean either the readers here do not have the same problem 
and have no comment, or your implementation does not require much 
discussion. In any case, you could continue with the process by proposing a 
formal patch, and getting someone to review it once it meets the 
requirements for a finished patch.

See https://docs.djangoproject.com/en/dev/internals/contributing/ 
and https://docs.djangoproject.com/en/dev/intro/contributing/

The change should come with tests and documentation.

Regarding your implementation, I'm not thrilled with the requirement for 
users to provide a Cast. If casting needs to occur, it'd be preferable to 
have the interface of JSONF(lookup, output_field), and then wrap self with 
Cast(self, self.output_field).

Is there any particular reason why you chose a separate expression type 
rather than containing the functionality within the existing F expression? 
I'd be keen to hear the pros/cons of each.

On Sunday, 11 November 2018 13:51:57 UTC+11, Mani S wrote:
>
> Hi devs,
> Any update on this? Would like to know your thoughts
>
>
>
>
> Thanks,
> Mani
>
> On Sun, Oct 28, 2018 at 1:28 AM Mani S > 
> wrote:
>
>> Django's F objects does not perform JSON lookups. A ticket has been 
>> raised for the same https://code.djangoproject.com/ticket/29769
>>
>> I have written a customer expression for querying JSONField and
>> * it works!*
>>
>> class KeyTextTransformFactory:
>>
>> def __init__(self, key_name):
>> self.key_name = key_name
>>
>> def __call__(self, *args, **kwargs):
>> return KeyTextTransform(self.key_name, *args, **kwargs)
>>
>> class JSONF(F):
>>
>> def resolve_expression(self, query=None, allow_joins=True, reuse=None, 
>> summarize=False, for_save=False):
>> rhs = super().resolve_expression(query, allow_joins, reuse, 
>> summarize, for_save)
>>
>> field_list = self.name.split(LOOKUP_SEP)
>> for name in field_list[1:]:
>> rhs = KeyTextTransformFactory(name)(rhs)
>> return rhs
>>
>> It is necessary to include Cast in rhs,
>> Sample.objects.filter(jsonfield__lookup__value=Cast(JSONF('value'), 
>> IntegerField())) 
>>
>>
>> Comment ref: https://code.djangoproject.com/ticket/29769#comment:5
>>
>>
>> Note: This is my first contribution in Django source code, any 
>> comments/suggestions would help me learn the process better.
>>
>>
>>
>>
>>
>> Thank you,
>> Mani
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/88f52ca0-ebd9-4139-b88e-9f21292b7603%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b45f2c10-2285-4585-b303-4fb12de209b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Remove pyinotify autoreloader support

[Josh Smeaton]

I ran a slightly modified version of your test code since we have many 
modules at "top level". "k3" is one of the top level modules that houses a 
bunch of apps. In total there are about 1340 python files (including 
recently squashed migrations and tests). Here's the results:

Project: 1575
Total: 7381
[(u'django', 890),
 (u'k3', 675),
 (u'nltk', 426),
 (u'newrelic', 294),
 (u'google', 172),
 (u'braintree', 171),
 (u'requests', 144),
 (u'rest_framework', 108),
 (u'boto', 101),
 (u'suds', 95),
 (u'numpy', 94),
 (u'core', 90),
 (u'weasyprint', 87),
 (u'PIL', 87),
 (u'docutils', 73),
 (u'cssutils', 70),
 (u'hyper', 68),
 (u'Crypto', 68),
 (u'email', 62),
 (u'oauthlib', 56),
 (u'cryptography', 55),
 (u'pygments', 55),
 (u'celery', 54),
 (u'oauth2client', 53),
 (u'elasticsearch', 50),
 (u'coreapi', 48),
 (u'jinja2', 47),
 ]

On Monday, 5 November 2018 02:14:12 UTC+11, Tom Forbes wrote:
>
> I have done a lot more reading on this and I really feel pyinotify may not 
> be required, or we could at least switch to a watchman based service for 
> simplicity right away. Projects like uwsgi use a stat based approach, as 
> do most other projects I’ve seen and it appears to work ok for them. 
> watchman has the advantage of being easier to test, as we can mock the 
> actual watchman service itself, and it supplies a sane, simple API across 
> all supported platforms. However it suffers from some limitations I need to 
> work around.
>
> As an example of a fairly large project I took a look at Sentry. After 
> running the test suite (and excluding test modules) there are 1,000 loaded 
> Python modules under sentry.* and 4,000 total entries in sys.modules. 
> Django accounts for about 600, by far the biggest third party module.
>
> During development it seems likely that it’s not worth polling any builtin 
> python modules at all, and drastically reducing the polling time for site 
> packages including Django. That would reduce the number of files to poll by 
> 75%. Perhaps we could even just skip watching site-packages entirely - I 
> think it’s not very common to edit these in a usual development session, 
> and if the user finds themselves editing them we could add a flag to 
> include them in the watch list?
>
> If anyone here has a fairly large Django app they work on could I request 
> that they run this snippet at some point after it has booted and send me 
> the output, to see if Sentry is representative?
>
> from collections import Counter
> counter = Counter(m.split('.')[0] for m in sys.modules.keys())
> total = sum(counter.values())
> project_total = counter['PROJECT NAME HERE']
> print('Project:', project_total)
> print('Total:', total)
> print(counter.most_common(20))
>
> Thanks,
>
> Tom
>
>
>
>
> On 8 October 2018 at 12:04:08, Tom Forbes (t...@tomforb.es ) 
> wrote:
>
> Thanks for the feedback! In the pull request I have re-added support for 
> pyinotify with tests, it was not as hard to write them as I believed. They 
> still fail, but I’m working on that!
>
> I’ve found an interesting module confusingly called watchgod. The author 
> says[1] that with the new os.scandir() method added in python 3.5 a stat 
> based method can scan a project of 850 files in about 24ms.
>
> While this is watching a single directory tree and not the entirety of 
> sys.modules I believe we could get similar speedups by being a bit more 
> intelligent in our approach. For example we really don’t need to stat every 
> single django.* module every second. I’m guessing that those change very 
> rarely (unless you’re hacking on Django!), so could maybe stat them every 
> 2–3 seconds or even not at all. We could reduce the stating time on 
> site-packages modules as well for similar reasons.
>
> If we do this right we could potentially get the overhead down to an 
> acceptable level that we don’t necessarily need platform specific watchers.
>
>1. 
>
> https://github.com/samuelcolvin/watchgod#why-no-inotify–kqueue–fsevent–winapi-support
>  
>
>
>
> On 6 October 2018 at 20:56:17, charettes (chare...@gmail.com ) 
> wrote:
>
> While I understand the complexity of 1. I think shipping a version of 
> Django without
> an equivalent inotify replacement such as watchdog could be problematic.
>
> From my personal experience when using Django's development server in 
> Docker
> containers sharing local volumes installing pyinotify resulted a 
> significant performance
> CPU and I/O improvement.
>
> Simon
>
> Le samedi 6 octobre 2018 15:32:33 UTC-4, Tom Forbes a écrit : 
>>
>> What do we think about removing the pyinotify functionality in the 
>> autoreloader? For context, if pyinotify is installed (Linux only) the 
>> autoreloader will attempt to use it to detect file changes over the 
>> standard polling-based approach. It is generally much more efficient but is 
>> not cross platform, and is not well documented currently IMO.
>>
>> I’m hacking away at my attempt at refactoring the autoreloader (
>> 

Re: Explore integrating django-docker-box in some way?

[Josh Smeaton]

I'm sorry I haven't had the time to review or contribute yet, but I think 
it'll be a very useful project - especially for new contributors that might 
have a little docker experience. The current vagrant solution is heavy, 
does not work properly on windows and some linuxes, and isn't that easy to 
maintain or deploy. I'd be in favour of adding the docker files directly to 
django/django to minimise setup burden (DJANGO_PATH), and improving the 
contributing docs to show users how to test using docker.

One of the hardest things I found at sprints was getting development 
environments setup to effectively contribute - even using the docker-box 
project which I understand quite well. Anything we can do to improve that 
situation will be very beneficial.

I have fewer opinions about the official CI story, hopefully some of the 
infrastructure team can comment more on that. I think that replacing the 
ansible roles with a docker setup can have some definite improvements and 
open up CI tasks to a larger pool of people (anyone that can edit docker 
files), but it'd come with maintaining the host that runs docker (cleaning 
up images, dealing with disk space issues, etc).


On Monday, 5 November 2018 01:20:03 UTC+11, Tom Forbes wrote:
>
> Hello all,
>
> I’ve been working on a docker-compose based alternative to django-box 
> (imaginatively named django-docker-box) over the last month and it 
> finally appears to be mostly complete. 
>
> For reference the tool is just a Dockerfile and a docker-compose 
> definition that is able to run a complete test matrix of every supported 
> Python and DB version. It’s as simple as docker-compose run sqlite. You 
> can see a full test run (excluding oracle) here: 
> https://travis-ci.com/orf/django-docker-box/builds/90167436
>
> Florian suggested I create a thread here to gather feedback and discuss 
> any potential future directions for the project, so here goes:
>
> Firstly I’d like to know if there is any support for moving this under the 
> Django project itself, maybe even as a replacement for django-box? I think 
> the setup is pretty quick compared to django-box and is more flexible in 
> terms of database version support as well as working with Oracle. I’d also 
> really like some help improving Oracle support if anyone has the time!
>
> Secondly is there any support for integrating this with our current 
> Jenkins setup? I think it would be pretty neat to have parity between what 
> runs on the CI and what we can run locally and have any improvements shared 
> between both. Perhaps a full matrix run (which right now is 66 different 
> environments) is out of the question but a smaller subset could be good?
>
> Thirdly, and this is a bit wild, but what about using this to reduce the 
> burden of running Jenkins by running the tests on a managed CI service like 
> Travis CI? We would likely still need Jenkins due to issues with Oracle and 
> running tests on Windows (unless 
> https://github.com/django/django/pull/10259 works with Docker!), but we 
> could offload some of the environments onto a third party service. Travis 
> gives large OS projects like Django increased concurrency limits on their 
> accounts so we could end up with pretty speedy test runs. Also with 
> docker-compose switching between CI services (including Jenkins) would be 
> very simple.
>
> The repo is here: https://github.com/orf/django-docker-box.
>
> Any feedback on these points or the project itself would be greatly 
> appreciated,
>
> Tom 
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/074f68c9-2199-4128-a37a-bfc1852f4806%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: backend specific tests

[Josh Smeaton]

I don't think there's a full list of extensions the test suite uses, 
but https://docs.djangoproject.com/en/2.1/ref/contrib/postgres/operations/ 
would be close to a full set I'd imagine.

On Sunday, 4 November 2018 14:43:23 UTC+11, Dan Davis wrote:
>
> So, the contributor guidelines page about unit tests mentions running 
> database specific tests:
>
>
> https://docs.djangoproject.com/en/2.1/internals/contributing/writing-code/unit-tests/#testing-other-python-versions-and-database-backends
>
> I am working on ticket 29984, and it seems to me that since the TruncDay 
> is attempting to cast to the timezone on the database level, it is working, 
> and its job is done.  So, the fix should be at the backend level, and the 
> ticket provides one for Postgres.
>
> Following the unit test advice, I created a settings file.   But not using 
> my Postgresql admin user.   I get the following errors:
>
> psycopg2.ProgrammingError: permission denied to create extension 
> "btree_gin"
> HINT:  Must be superuser to create this extension.
>
> I'm not going to do that - do you have any list of required extensions?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d6b9f379-3a3d-4ea7-925c-ddd3f2422dab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Idea: Allow queryset.get() and queryset.filter() to accept a positional argument for implicit primary key filtering

[Josh Smeaton]

I'm in the same boat as Simon - I've wanted this many times in the last few 
months, but only when working at the shell. I'd be +1 for get and -1 for 
filter also.

On Thursday, 1 November 2018 05:12:53 UTC+11, charettes wrote:
>
> As I've mentioned on the ticket I've been wishing get(pk) could translate
> to get(pk=pk) for a while but I never got to suggest it. Probably because I
> didn't feel like adding (pk=...) was really that long. I'd note that most 
> of the
> times I wished this existed was when doing some object manipulations
> through a Django shell.
>
> I'm not convinced this would be as useful for `filter()` though as I don't
> recall wanting to retrieve a set of objects matching a value I know will
> be unique.
>
> Something to keep in mind as well is whether or not we want to allow
> this to be coupled with extra args and kwargs.
>
> e.g.
>
> Book.objects.get(isbn, author__name='Daniel Roy Greenfeld')
>
> I'd be in favor of preventing pk and kwarg or Q args mixing.
>
> Count me +1 for the get() case and -1 for the filter() one.
>
> Simon
>
> Le mercredi 31 octobre 2018 13:13:34 UTC-4, Antwan a écrit :
>>
>> Hi, 
>> I'm creating this topic to see if there is interest to implement 
>> positional arguments in queryset filtering.
>>
>> Current situation 
>>
>> Currently the only way to use positional arguments to filter can be 
>> either:
>>
>>- Passing a single or multiple Q objects: 
>>
>>MyClass.objects.filter(Q(key=value))
>>MyClass.objects.filter(Q(key=value), Q(other_key=value))
>>
>>
>>
>>- Passing a couple is also working (not sure if this is a happy 
>>accident, should it be removed?) 
>>
>>MyClass.objects.filter((key, value))
>>
>>
>>
>>- Combination of both is also proven to work 
>>
>>MyClass.objects.filter((key, value), Q(other_key=value))
>>
>>
>>
>> Suggestion 
>>
>> This feature suggestion is to leverage the case when a non-Q / non couple 
>> object is passed, so it implicitly interpreted as the value for the model's 
>> pk.
>>
>> This could ease/simplify code by omitting pk when this is the only 
>> filter used:
>>
>>
>> MyClass.objects.get(value)
>> # Translates into: MyClass.objects.get(pk=value)
>>
>>
>> or
>>
>>
>> MyClass.objects.filter(value)
>> # Translates into: MyClass.objects.filter(pk=value)
>>
>>
>> or 
>>
>>
>> MyClass.objects.filter(Q(value))
>> # Translates into: MyClass.objects.filter(Q(pk=value))
>>  
>>
>> Do you think it's worth it? It could be leveraged to simplify many 
>> situations.
>> I'd be happy to proceed to the development myself if this is something 
>> gathering interest.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/ba9128d7-9e49-4269-b3a3-9995a998b491%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Widening participation (Thoughts from DjangoCon)

[Josh Smeaton]

I strongly dislike Trac in nearly every way. It's hard to search and the 
filters are next to useless, and the categorisation features we use are not 
very useful. I believe the better way to search Trac is to use google and 
site:code.djangoproject.com which is a red flag itself.

On Saturday, 27 October 2018 11:09:38 UTC+11, Tom Forbes wrote:
>
> How much of this would you attribute to the current ticketing system 
> itself, rather than tickets being tagged appropriately?
>
> I know when I started contributing I found trac to be pretty intimidating 
> in terms of complexity, especially the search. I still prefer to use the 
> 'Search Trac' field in the root code.djangoproject.com page than fiddle 
> with the myriad of options and drop downs in the browse tickets section.
>
> If we think of getting new people onboard as a conversion funnel we need 
> to stop dropoff as much as possible, and that extends to the UI of the 
> ticket tracker as well I believe.
>
> Tom
>
> On Fri, 26 Oct 2018, 22:43 Ian Foote, > 
> wrote:
>
>> Hi Carlton,
>>
>> I've had similar thoughts sitting in the back of my mind for at least a 
>> couple of months, so thank you for sharing this. I agree that finding 
>> tickets is one of the big problems here, both for new contributors and for 
>> sprint leaders. At Pycon UK I took on the role of sprint leader along with 
>> Adam Johnson and directing people to appropriate tickets was a definite 
>> difficulty. I was also unaware of the django core mentorship list and will 
>> be joining that soon. I'm willing to spend some time mentoring a small 
>> number of people, life permitting.
>>
>> Ian
>>
>> On Fri, 26 Oct 2018 at 14:44, Carlton Gibson > > wrote:
>>
>>> Hi All. 
>>>
>>> OK, so last week I was at DjangoCon US in San Diego. (Thank you if you 
>>> organised that! Hi! if we met and chatted.) 
>>> I gave a talk ("Your web framework needs you!") inspired by the 
>>> discussion on the 
>>>  
>>> DSF list and the proposal to dissolve Django Core 
>>> . (Can’t see the DSF list? Join 
>>> the DSF 
>>> 
>>> .)
>>> I was asking for more participation in general, and participation that 
>>> is more representative of the wider Django community in particular.
>>>
>>> There was lots of good input from many people, including (but not, at 
>>> all, limited to) representatives of groups such Pyladies, DjangoGirls, and 
>>> so on. 
>>>
>>>
>>> The recurring themes seem to me to fit into three categories:
>>>
>>>1. The importance of *mentoring*. 
>>>2. The difficulty of *finding tickets*. 
>>>3. The importance of *sprints*. 
>>>
>>> The rest here is a summary of that. Hopefully it’s useful. 
>>>
>>> Mentoring
>>>
>>> For whatever reasons, the exiting *Contributing How-To* 
>>>  doesn’t 
>>> lead to contributions from a demographic that matches the wider Django 
>>> Community. 
>>>
>>> The point that came up again and again about this was that *mentoring* 
>>> is one of the best (perhaps the best?) tool in helping to change this. 
>>>
>>> Django Core Mentorship
>>>
>>> We don’t have an official mentoring programme but we do have the 
>>> django-core-mentorship 
>>> list . 
>>>
>>> This must be about the best-kept secret in the Django world: it’s gets 
>>> ≈0 traffic, but I told everybody at DjangoCon about it, and that they 
>>> should use it. 
>>>
>>> If you are not on django-core-mentorship, and you’re willing to help 
>>> prospective contributors, please sign-up. I’m hoping we can drive some 
>>> traffic to it. 
>>>
>>> Maybe there’s call for something more formal, but at least until DCM is 
>>> actually being used, that seems (to me) like something we can postpone. 
>>>
>>> Finding Tickets
>>>
>>> The next thing was that there’s not enough guidance on what to work on. 
>>>
>>> The guidance is to look for *Easy Pickings*. There are ≈1300 accepted 
>>> open tickets in TRAC. 13 of these are marked *Easy Pickings*. 
>>>
>>> That’s not enough. I think we’re too tight with it (or need another 
>>> grade). 
>>>
>>> There are *many* tickets which aren’t super hard: I put it that, most 
>>> of our community solve harder problems every day *using Django* than 
>>> most tickets require. 
>>>
>>> Yes, they still require time, love, energy, etc — and maybe some 
>>> mentoring — but it’s not primary research, in the main.
>>>
>>> I talked to people who had (at the conference) got the test suite 
>>> running and such, but been overawed by the (for want of a better phrase) 
>>> *sheer 
>>> face* of issue tracker. 
>>>
>>> We would do well to invite people better here. (I don’t have instant 
>>> solutions.) 
>>>
>>> Sprints
>>>
>>> I’m not historically a 

Re: QuerySet.iterator together with prefetch_related because of chunk_size

[Josh Smeaton]

I tend to agree with Tobi. Prefetching silently not working on iterator can 
be quite confusing, unless you have a good understanding of both APIs. It 
might be possible to do what you're asking, but it'd mean that django is 
now actually caching the result when it explicitly says it isn't - even if 
the result is a much smaller moving cache. Prefetching chunk_size results 
per chunk is unlikely to make a material difference to memory usage. Users 
are usually concerned about the entire result set of the primary table.

I don't know if you can change the API to make these suggested changes 
without also impacting how we iterate over result sets - but I'd be 
interested in seeing a proof of concept at the very least.



On Monday, 15 October 2018 20:41:13 UTC+11, tobias@truffls.com wrote:
>
> Thank you for your feedback. I would like to answer some statements to 
> either convince you or make it more clear, where my idea stems from:
>
> The fundamental reason why iterator() cannot be used with 
>> prefetch_related() is that the latter requires a set of model instance to 
>> be materialized to work appropriately which chunk_size doesn't control at 
>> all.
>> In other words chunk_size only controls how many rows should be fetched 
>> from the database cursor and kept into memory at a time. Even when this 
>> parameter is used, iterator() will only materialize a single model instance 
>> per yield.
>>
>  
> It should be easily possible to change the involved code of ModelIterable 
> to materialize the retrieved rows in batches. After materializing the batch 
> / chunk, it could do the prefetching.
>  
>
>> Given that iterator() always ignored prefetch related lookups instead of 
>> erroring out when they were specified make me feel like turning such a 
>> feature on by default could be problematic as it could balloon the memory 
>> usage which is the main reason why iterator is useful anyway.
>>
>
> I would argue, that users who thoughtlessly applied prefetching together 
> with iterator now actually get, what they thought of: less DB query round 
> trips traded against a little more memory usage.
>
> Best,
> Tobi
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e7a36092-0743-41dc-998e-eee11fa1180b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Jenkins resources

[Josh Smeaton]

Hi Matt,

The django-box project[0] uses very close to the same ansible configuration 
that is used to configure all of the Jenkins slaves Django uses with the 
notable omission of an Oracle builder.

Since the repos are separate, there has been some divergence on the Jenkins 
side of things, but nothing too major. If you find anything missing I'd be 
happy to update the box repo.

[0] https://github.com/django/django-box 

Regards,

On Thursday, 12 July 2018 22:37:29 UTC+10, Matt Cooper wrote:
>
> Hi Carlton, thanks for your quick reply!
>
> I'm setting things up here 
> .
>  
> Let me know if you can't access that link. It's configured as a public 
> project, meaning you shouldn't need a login to view it. (There are still 
> some known issues with public projects -- we're in preview right now.) The 
> build definition is actually specified here 
>  in my clone 
> of the Django repo as a YAML file. We do offer grids (we call it "matrix") 
> as well -- for testing purposes I'm only running against Python 3.5 on 
> Linux but it's straightforward to add ranges. I would very much appreciate 
> your guidance when and if you have the time.
>
> I'm confident we can put something together in terms of resources for the 
> DSF. I want to first convince myself that we're credible technically -- if 
> we have big gaps, no amount of dollars or build minutes from Microsoft 
> would make it worth moving. Is it alright with you if we postpone further 
> discussion on this topic?
>
> Sorry for the length -- what follows are just some specifics of what I've 
> done so far. We can take this off-list if it gets too noisy.
>
> My build steps are essentially what you suggested. First, I make sure pip, 
> setuptools, and wheel are up to date. Then I grab unittest-xml-reporting, 
> install the rest of the py3.txt requirements, cd into tests, then 
> ./runtests.py against a custom settings file (see below).
>
> One of the things I'm unclear on is where you get your settings files for 
> configurations other than SQLite. The Jenkins build hints at copying over 
> some directories for the other DBs, which I'm assuming have the settings 
> and maybe fixtures. The non-SQLite databases are an area I need to think 
> more about, so I'm not blocking the rest of my work on that.
>
> Other minor issues I've hacked around for now:
>
>- pylibmc wouldn't build for me (but I just got some advice from a 
>colleague on troubleshooting that) so I've temporarily removed it.
>- I need to add a custom test runner (
>xmlrunner.extra.djangotestrunner.XMLTestRunner), so I copied 
>test_sqlite.py and make a 1-line addition.
>- tests.timezones.ForcedTimeZoneDatabase generates output with an 
>invalid timestamp, so I have to delete that test's results or else the 
>whole test reporting step fails. I'm not clear if the bug is on our side 
> in 
>VSTS or someplace in one of the XUnit implementations.
>
>
> I'm curious what issues you faced building GitHub repos. We've had support 
> for several years, but greatly improved it over the last year. From an open 
> source perspective, the worst problem the "black box" nature -- the 
> community couldn't see built results, test failures, etc since VSTS 
> projects were always private. Now with public projects, we're hoping to 
> close that gap.
>
> Regards,
> Matt
>
> On Thu, Jul 12, 2018 at 3:17 AM Carlton Gibson  > wrote:
>
>> Hi Matt, 
>>
>> The Jenkins configuration is not public I'm afraid. 
>>
>> I'm one of the Django Fellows. If you wanted to invite me to your VSTS 
>> project I would be happy to review and offer pointers as you did the actual 
>> work, assuming that would help?
>>
>> The main thing would be setting up the grid of supported versions, 
>> against different Pythons and DBs. I guess I'd start just with Python 3.7 
>> (or 3.6) using SQLite and then grow out from there. 
>>
>> The most basic setup would just be: 
>>
>> * Clone
>> * install pip dependencies
>> * cd tests && ./runtests.py. 
>>
>> If you can set up the grid of supported environments, that becomes 
>> interesting. 
>>
>> I can only presume it's being actively worked on but, VSTS' integration 
>> with GitHub wasn't great last time I looked (?)
>>
>> Finally, I don't think we're looking to move the CI but would Microsoft 
>> provide free resource to the DSF (us) for testing Django? 
>>
>> Thanks. Interesting. 
>>
>> Kind Regards,
>>
>> Carlton
>>
>>
>> On Thursday, 12 July 2018 00:09:19 UTC+2, Matt Cooper wrote:
>>>
>>> Hi Django devs. I've looked all over for the tools that power 
>>> djangoci.com, but I don't see them. Are the Jenkins configuration, 
>>> scripts, and resources available someplace?
>>>
>>> Full disclosure, I'm on the Visual Studio Team Services group at 
>>> Microsoft. We're making sure that our CI system 

Re: A Django Async Roadmap

[Josh Smeaton]

I think most of what you've laid out sounds great and that pursuing async 
Django is in the projects best interests. The sync to async and async to 
sync wrappers that have come out of channels give me much more confidence 
that this project is doable in a reasonable amount of time with backwards 
compatibility being preserved.

The only real concern I have at the moment is around your comments 
regarding on demand foreign key traversal. If existing code running 
synchronously is going to be impacted, that's going to be very difficult 
for a lot of people. If it's only asynchronous traversal that'll have 
issues, then I have no real concern, as on demand foreign key fetching is 
usually a bug anyway.

Having a brief read through the psycopg asynchronous docs[0], it looks like 
a number of features will be impossible or troublesome to use, like 
transactions, executemany, and named cursors (.iterator() with server side 
cursors). We'd also need to investigate how pgbouncer would work in async 
mode, as most large sites using postgres are also using pgbouncer. I would 
expect support can only further improve, especially if there is a driver 
like django pushing. Fallback would just be to run inside a thread pool 
though, so it's not a blocker for the rest of the proposal.

Very exciting times ahead!

[0] http://initd.org/psycopg/docs/advanced.html#asynchronous-support

On Monday, 4 June 2018 23:18:23 UTC+10, Andrew Godwin wrote:
>
> Hello everyone,
>
> For a while now I have been working on potential plans for making Django 
> async-capable, and I finally have a plan I am reasonably happy with and 
> which I think we can actually do.
>
> This proposed roadmap, in its great length, is here:
>
> https://www.aeracode.org/2018/06/04/django-async-roadmap/
>
> I'd like to invite discussion on this potential plan - including:
>
>  - Do we think async is worth going after? Note that this is just async 
> HTTP capability, not WebSockets (that would remain in Channels)
>
>  - Can we do this in a reasonable timeframe? If not, is there a way around 
> that?
>
>  - Are the proposed modifications to how Django runs sensible?
>
>  - How should we fund this?
>
> There's many more potential questions, and I really would love feedback on 
> this. I'm personally pretty convinced that we can and should do this, but 
> this is a decision we cannot take lightly, and I would love to hear what 
> you have to say.
>
> Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d3b3c04e-9db1-4a41-af3a-ad50209a07e4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Proposal: security enhancements

[Josh Smeaton]

Most of this sounds really good. As Jacob mentioned, CSP can be quite 
scary, and sounds like something a novice could try to implement for "good 
security" and end up causing way more issues. I'd really like to see easy 
integration for report only mode, with controls that are harder to turn for 
full blown CSP.

With regard to pyup - the pipenv check command handles this nicely. I'd 
imagine support would be similar - in that we'd list pyup as a required 
dependency for Django, and ship a management command that uses it. The 
deploy check could then hook in.

On Tuesday, 1 May 2018 17:28:21 UTC+10, James Bennett wrote:
>
> I've written this up in pseudo-DEP format partly for ease of
> organization, and partly because I'm unsure whether it would require a
> DEP. Right now I'm just throwing it out here as a proposal, and
> offering to work on implementing it; if you have questions, concerns,
> or suggestions for things to add to it, please let me know :)
>
>
> Introduction
> -
>
> Django has acquired a reputation for being helpful with/good at
> "security". Which has been a focus since at least 1.0; Django goes out
> of its way to protect you out of the box against common security
> problems, and includes a lot of tools geared toward making it easy to
> do the right thing when building apps.
>
> But security is a Red Queen problem; we can't ever pause and rest,
> since as soon as we do we'll fall behind.
>
> In light of which, I'd like to propose a few security-related
> improvements/additions, and offer to put in the time to help implement
> them. My baseline goal here is to set things up such that it's
> possible, using only things that ship with Django itself and an
> SSL-enabled hosting service, to score an A+ rating on
> securityheaders.com. Beyond that, some stretch goals are possible.
>
>
> Content Security Policy
> 
>
> CSP[1] is one of the more important defenses available against XSS and
> a few other types of attacks. Django does not currently have
> out-of-the-box support for it; the usual solution is to install
> django-csp[2], configure it and set up its middlware to emit the CSP
> header.
>
> I'd like to see support for CSP built in to Django. This could be done
> by integrating django-csp, which is open source and uses a compatible
> license, though there'd need to be some updates to its documentation
> (which seems to lag a bit behind what the code supports) and it might
> also be good to switch over to using dictionary-based configuration.
>
>
> Referrer-Policy
> -
>
> The Referrer-Policy header[3] is relatively new, but gaining support
> in major browsers. It helps enhance both security and privacy.
>
> I wrote a package earlier this year[4] that supports sending the
> Referrer-Policy header, and BSD-licensed it. It would be relatively
> easy to integrate this into Django.
>
>
> Subresource integrity
> -
>
> Subresource integrity[5] allows the author of an HTML document to
> supply the expected hashes of resources (scripts, stylesheets, images,
> etc.) referenced in the document. This helps to harden against not
> only local breaches, but also breaches of third-party scripts and
> CDNs.
>
> I don't have a clear vision yet of how it would work, but I'd like to
> see the form media system and the staticfiles app grow support for
> SRI. In the case of form media, the output already generates the full
> HTML and could include the hash. In the case of the {% static %}
> template tag, I'm open to suggestions for how it could
> generate/include the hash. The admin should use this by default.
>
>
> CORS
> -
>
> Cross-Origin Resource Sharing is increasingly important on the modern
> web, and Django does not currently support setting CORS headers. There
> is an open-source package for CORS in Django[6], which could be
> integrated.
>
>
> rel="noopener"
> -
>
> Setting rel="noopener" on links which will open in new tabs/windows is
> an easy way to head off certain types of surprising attacks[7]. Like
> SRI, I don't yet have a great plan for how this could be supported
> easily/automatically in Django, but I'd like to give it a try.
>
>
> Improved system-check integration
> ---
>
> Currently the security system-check is OK, but still a bit
> minimal. I'd like to add support for checking at least everything I've
> proposed adding above, and probably several more items. In my ideal
> world, a service like securityheaders.com could be replicated by the
> security check.
>
> In my magical stretch-goal land, I'd also figure out a way to support
> the pyup safety library[8] to scan for a requirements file and any
> dependencies in setup.py, and warn if known-insecure versions are
> specified.
>
>
> Improved security documentation
> -
>
> Right now we have OK documentation on security, in that we 

Re: #django-dev IRC channel

[Josh Smeaton]

Yes - that's probably a good idea!

On Thursday, 26 April 2018 05:11:33 UTC+10, Kenneth wrote:
>
> Hello,
>
> I was just curious about the #django-dev IRC Channel.
> Currently, the only mention for this channel is on the Code of Conduct FAQ 
>  page.
> Should it also be mentioned on the Contributing to Django 
>  page?
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e163783e-81f1-4080-9aa8-ad09c00b1f8f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: django-firebird test expressions

[Josh Smeaton]

Have you looked into using Cast for this particular query to see if it 
passes?

https://docs.djangoproject.com/en/2.0/ref/models/database-functions/#cast

Cast(RawSQL('%s', ['value']), CharField(max_length=10) ?

If that worked, then perhaps you could find a clever way to wrap problem 
expressions, or at least document the workaround for your users.

On Tuesday, 17 April 2018 21:12:09 UTC+10, Maximiliano Robaina wrote:
>
>
> Hi Shai,
>
> Thank for the tip
>
> El martes, 17 de abril de 2018, 3:01:30 (UTC-3), Shai Berger escribió:
>>
>> Hi Maximiliano, 
>>
>> On Sat, 14 Apr 2018 19:25:48 -0700 (PDT) 
>> Maximiliano Robaina  wrote: 
>>
>> > Hi, 
>> > 
>> > Testing expressions test app, the query generated into 
>> > BasicExpressionsTests.test_annotate_values_filter method: 
>> > 
>> > companies = Company.objects.annotate( 
>> > foo=RawSQL('%s', ['value']), 
>> > ).filter(foo='value').order_by('name') 
>> > 
>> > Generate: 
>> > 
>> > 'SELECT  "EXPRESSIONS_COMPANY"."ID", "EXPRESSIONS_COMPANY"."NAME", 
>> > "EXPRESSIONS_COMPANY"."NUM_EMPLOYEES", 
>> > "EXPRESSIONS_COMPANY"."NUM_CHAIRS", "EXPRESSIONS_COMPANY"."CEO_ID", 
>> > "EXPRESSIONS_COMPANY"."POINT_OF_CONTACT_ID", ? AS "FOO" FROM 
>> > "EXPRESSIONS_COMPANY" WHERE ? = ? ORDER BY 
>> > "EXPRESSIONS_COMPANY"."NAME" ASC' 
>> > 
>> > This sql command has 3 params (?), two of which are out of where 
>> > clause. 
>> > 
>> > ? AS "FOO" 
>> > 
>> > WHERE ? = ? 
>> > 
>>
>> A few years ago something similar was causing issues with the Oracle 
>> backend. To resolve it, Mariusz added a clevar hack, relying on named 
>> parameters -- he made sure that if the same parameter value is used 
>> more than once, then the statement re-uses the parameter name, passing 
>> the value only once. So, in your case, the equivalent would be 
>> something like 
>>
>> ... 
>> "EXPRESSIONS_COMPANY"."POINT_OF_CONTACT_ID", %arg1 AS "FOO" FROM 
>> "EXPRESSIONS_COMPANY" WHERE %arg1 = %arg1 ORDER BY 
>> "EXPRESSIONS_COMPANY"."NAME" ASC' 
>>
>> with 
>>
>>  cursor.execute(sql, [], arg1='value') 
>>
>> This was relatively easy to do in the Oracle backend, which has always 
>> used named parameters under the hood (though the idea to do it was, in 
>> my opinion, surprising and brilliant). If Firebird supports them, you 
>> may be able to borrow this solution. 
>>
>> Shai 
>>
>
> Ok, but it is a cx_Oracle implementation.
> Unfortunately fdb (the firebird python driver) doesn't support this and, 
> in any case, is a FirebirdSQL limitation. [1]
>
> Regards.
>
>
> [1]  
> https://stackoverflow.com/questions/37348807/data-type-unknown-in-case-expression-with-only-parameters-as-values
>
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d2367698-dbf6-4750-88f7-3eba95ac9f2f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: django-firebird test expressions

[Josh Smeaton]

It looks correct to me. RawSQL is defining a parameter to be included in 
the query. The RawSQL expression itself is used in the select and on the 
left hand side of the WHERE clause, so the parameter needs to be included 
there. The right hand side of the WHERE clause is a regular string (which 
just happens to also be named "value") which Django will always 
parameterise and has nothing to do with the RawSQL expression itself.

Is there some reason you think there's an issue with the SQL being 
constructed?

On Sunday, 15 April 2018 12:25:48 UTC+10, Maximiliano Robaina wrote:
>
> Hi,
>
> Testing expressions test app, the query generated into 
> BasicExpressionsTests.test_annotate_values_filter method:
>
> companies = Company.objects.annotate(
> foo=RawSQL('%s', ['value']),
> ).filter(foo='value').order_by('name')
>
> Generate:
>
> 'SELECT  "EXPRESSIONS_COMPANY"."ID", "EXPRESSIONS_COMPANY"."NAME", 
> "EXPRESSIONS_COMPANY"."NUM_EMPLOYEES", "EXPRESSIONS_COMPANY"."NUM_CHAIRS", 
> "EXPRESSIONS_COMPANY"."CEO_ID", 
> "EXPRESSIONS_COMPANY"."POINT_OF_CONTACT_ID", ? AS "FOO" FROM 
> "EXPRESSIONS_COMPANY" WHERE ? = ? ORDER BY "EXPRESSIONS_COMPANY"."NAME" ASC'
>
> This sql command has 3 params (?), two of which are out of where clause.
>
> ? AS "FOO" 
>
> WHERE ? = ?
>
>
> So, the underlying database driver execute this, doing something like this
>
>  cursor.execute(sql, ['value', 'value', 'value'])
>
> Is correct that the 3 params are replaced into the entire sql and not just 
> on where clause ?
> It depend of the  implementation on each database driver?
>
> Best regards
> Maxi
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d63ac393-8e8f-46f6-93d9-d0d1373da0cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Shouldn't manage.py call python3 instead of python?

[Josh Smeaton]

As a datapoint, I've seen roughly 1 person per week in #django IRC confused 
about specific startup exceptions due to them using python 2 rather than 
python 3 on Django >= 2.0. Unsure how many of these are due to the shebang. 
That said, it looks like there are no good solutions other than maybe 
ensuring our docs always show the form *python3 manage.py * rather 
than *./manage.py *.

On Wednesday, 11 April 2018 12:02:31 UTC+10, Bobby Mozumder wrote:
>
> In any case you’re going to see a lot of Django 2.0 developers on Mac OS 
> hit this problem when they install to default Python or use standard Python 
> install convention where Python 3.5 is installed as “python3".
>
> -bobby
>
> On Apr 10, 2018, at 3:46 PM, Aymeric Augustin <
> aymeric@polytechnique.org > wrote:
>
> On 10 Apr 2018, at 17:43, Florian Apolloner  > wrote:
>
> On Tuesday, April 10, 2018 at 1:28:33 PM UTC+2, Tim Allen wrote:
>>
>> Since `django-admin startproject my_project` is created on the fly from 
>> templates, couldn't we dynamically create the `manage.py` executable based 
>> on some system introspection and an agreed upon priority
>>
>
> Wouldn't that result in something along the lines of "works on my system" 
> and breaks elsewhere? after all manage.py is committed into git more often 
> than not.
>
>
> ... which directs us to the correct solution: setting PYTHONPATH and 
> DJANGO_SETTINGS_MODULE correctly and using django-admin instead of 
> manage.py. 
>
> pip / setuptools rewrites the shebang line appropriately when it installs 
> the django-admin script. (I'm not sure how this happens exactly.)
>
> My point is — there's no perfect solution. At best we can aim for a less 
> imperfect solution than the status quo.
>
> -- 
> Aymeric.
>
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-develop...@googlegroups.com .
> To post to this group, send email to django-d...@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/C36A8494-1094-4A03-A402-618BB999F927%40polytechnique.org
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/33ec0ccb-a63b-4451-bfec-815b9e5f52a7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Shouldn't manage.py call python3 instead of python?

[Josh Smeaton]

I think you're right and PEP394 is the relevant 
text: https://www.python.org/dev/peps/pep-0394/

TL;DR

For now, *python* should refer to python2 and *python3* should be used to 
refer to python 3.

On Saturday, 7 April 2018 07:07:35 UTC+10, Bobby Mozumder wrote:
>
> The header of manage.py has: #!/usr/bin/env python
>
> Shoudn’t it be: #!/usr/bin/env python3
>
> Since 2.0 is now only Python3. Both my Mac OS & FreeBSD environments have 
> Python 3.5+ as “python3". (I’m not sure about Linux or other environments).
>
> Is that a bug I need to file?
>
> -bobby
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/d13b010f-df53-46eb-8dab-93b4fc05dd97%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Django ORM Handling of Reverse Relationships and Multi-Value Relationships

[Josh Smeaton]

It sounds like you understand what the current behaviour is, and you think 
that it would be better modelled with Q objects, is that the case? I can 
see where you're coming from, as even the docs explain the difference 
loosely in terms of AND and OR.

Q(entry__headline__contains='Lennon') & Q(entry__pubdate__year=2008) would 
be the filter(cond_a, cond_b).
Q(entry__headline__contains='Lennon') | Q(entry__pubdate__year=2008) would 
be the filter(cond_a).filter(cond_b).

The justification for **not** doing it this way, I suppose, would be that Q 
objects may not have existed at the time, or this was just never 
considered. The justification for not changing it to be this way now would 
be because it'd break user code everywhere.

I agree that the Q &/| logic would be more intuitive, but it's not 
something we can change at this point.

On Friday, 30 March 2018 08:26:44 UTC+11, Andrew Standley wrote:
>
> I have recently become acquainted with some ORM behaviour for reverse 
> relationships that "makes no sense", and I'm hoping someone can explain the 
> justification for the current behaviour. 
>
> This specifically relates to `filter` behaviour referenced in 29271 
> , and 16554 
>  which seems tangentially 
> related to several issues with `exclude` (24421 
> , 14645 
> , 17315 
> ) and aggregate expressions (
> 16603 , 19415 
> )
>
> Most of the confusion about 'intended' behaviour and confirmed 'bugged' 
> behaviour seems to relate to the ORM's use of joins for reverse 
> relationships.
> I think my personal confusion boils down to two questions.
>
> 1) Is there some fundamental limitation in the ORM that prevents reducing 
> the number of joins? Several of these tickets indicate how the ORM could 
> potentially produce similar results with queries that did not use multiple 
> joins. Why is that not desirable behaviour?
>
> 2) Why is the current behaviour of `filter` for multi-value relationships 
> 'intended'? I'm hoping I am missing something obvious but it seems to me 
> that `Q` objects would support the type of behaviour suggested in the 
> spanning 
> multi-valued relationships 
> 
>  documentation 
> in a much more inituative manner. In a test case with models
>
> class Related(models.Model):
> field = models.CharField(max_length=100)
>
> class Main(models.Model):
> field_one = models.CharField(max_length=100)
> field_two = models.CharField(max_length=100)
> related = models.ForeignKey(Related, on_delete=models.CASCADE)
>
>
> both
>
> >>> Related.objects.filter(Q(main__field_two='2')|Q(main__field_one='1'))
>
> SQL:
> SELECT "test_app_related"."id", "test_app_related"."field" FROM 
> "test_app_related" INNER JOIN "test_app_main" ON ("test_app_related"."id"=
>  "test_app_main"."related_id") WHERE ("test_app_main"."field_two" = "2"
>  OR "test_app_main"."field_one" = "1")
>
> and
>
> >>> Related.objects.filter(main__field_two='2').filter(main__field_one='1'
> )
>
> SQL:
> SELECT "test_app_related"."id", "test_app_related"."field" FROM "test_app_
> related" INNER JOIN "test_app_main" ON ("test_app_related"."id"= 
> "test_app_main"."related_id") INNER JOIN "test_app_main" T3 ON ("test_app_
> related"."id" = T3."related_id") WHERE ("test_app_main"."field_two" = 
> "two" AND T3."field_one" = "one")
>
> Produce exactly the same results but the second seems to have an 
> unnecessary extra join, and directly contradicts the behaviour of filter 
> with non multi-valued fields.
>
>
>
> In short, could someone be kind enough to explain the justification for 
> all this weird behaviour with multi-value relationships?
>
>
> Cheers,
>   Andrew
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/81339adc-875e-48ac-a41f-4e45d0c26823%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Gsoc 2018

[Josh Smeaton]

I'm sorry, but the deadline has already passed for applications. It closed 
on the 27th of March (2 days ago) according to the GSOC 
timeline https://developers.google.com/open-source/gsoc/timeline. I do note 
that Django's own timeline mentions April 3rd being the deadline, but I'm 
not entirely sure that's 
accurate https://code.djangoproject.com/wiki/SummerOfCode2018.

That said, I'm unsure if work on the Django Girls website would be eligible 
for the Django GSOC program, so I won't comment too much on eligibility in 
that regard. Perhaps someone else can.

As far as the work itself though, without seeing a timeline of what needs 
to be done or familiarity with the particular issue you're looking at, it's 
hard to say if the feature would be "big enough" for a 12 week development 
cycle. You'd also need to find a mentor that was willing to sponsor the 
application. It might make sense to look at adding someone from the django 
girls organisation that was also a django software foundation member to the 
list of mentors if it was decided that django girls was in scope. I think 
that's something we should address in future years one way or another.

To answer your question specifically though - I don't think your proposal 
is going to happen this year, mostly due to timelines. I'm happy to be 
corrected by others though.

Regards,

On Thursday, 29 March 2018 06:41:09 UTC+11, AGATEVURE GLORY wrote:
>
> I need response from the message I sent. Can I can get a reply please.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/b110b331-5fec-49cd-b7f4-99ef315ce24b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Adding custom join support to Django

[Josh Smeaton]

Someone has again brought up the lack of custom join support in django - 
specifically LEFT OUTER joins: https://code.djangoproject.com/ticket/29262

I figure it's probably something that we'd like the ORM to support but I 
don't think we've ever really landed on a design that's palatable. I'd like 
to try and get a rough consensus for a design and flesh out any likely 
issues. We can then either provide a ready to go syntax that someone could 
implement, or I could put together a DEP. At a minimum, we can point future 
askers towards a design and "PR Welcome" them. Best case someone (possibly 
myself) runs with the design and provides an implementation.

What I'm interested in seeing are specific use cases for customising joins 
as that will drive any kind of design.

I've been playing around with some syntaxes, and this is the kind of thing 
I'm currently leaning toward:

qs = Book.objects.annotate(
my_reviews=joins.Left(
Review.objects.filter(user=u1),
book=OuterRef('pk')
)
)

This supports customising both the join type, and the join conditions (ON 
book.id = review.book_id). One concern with this design is that now the 
result set contains multivalues - where a book might be repeated if the 
number of reviews that match the filter is greater than 1. We could ask the 
user to specify whether the result is potentially multi-valued and convert 
the result into a list, but that would require ordering and comparing the 
current row to the previous row to check if it's a multivalue. These are 
the kind of issues I want to flesh out, and that's only going to be 
possible given a sufficient corpus of examples.

So I'll kick off with an example that seems to be rather common:

*- Add filtered results from a reverse relation / m2m relation to the 
result set without eliminating results using a LEFT JOIN.*

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3c3ce7f4-4138-4a80-b43a-3613e8d65070%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Gsoc 2018

[Josh Smeaton]

Hi there,

I don't think the rules for Google Summer of Code allow for "projects" to 
be documentation only. The extra promo materials would also count as 
documentation I believe. The relevant rule would be 1.26:

https://summerofcode.withgoogle.com/rules/

"Project” means an open source coding project to be worked on by a Student 
> as an individual. For the avoidance of doubt, Projects do not include 
> projects for documentation only.


So I don't think this particular idea would be accepted as a GSOC proposal 
unfortunately. Google want to see code as output in the first instance, 
with supporting documentation to accompany that output.

Regards,


On Monday, 26 March 2018 14:12:55 UTC+11, AGATEVURE GLORY wrote:
>
> Hello, 
>
> Project Idea: Django Girls Tutorial Glossary documentation and Improvement 
> of organizers promotional, design reaources 
>
> About Me: 
> I am Agatevure Glory from Nigeria. I lady interested in contributing to 
> the open source community. 
> I am not an expert python and Django programmer but I have basic knowledge 
> of python and Django. I got to know of Django through the Django Girls 
> workshop. Ever since I got involved with Django Girls I have been thinking 
> how I can contribute to the Django community so thats why I chose to Django 
> for this year's summer of code. If given the opportunity this will be my 
> first contribution to the open source community apart from personal github 
> repos. 
>
> Why Glossary Documentation: I went through the Django Girls github repo 
> and found an issue #123 on that which was raised since 2015 and it was 
> marked as needed help. So I decided to pick it up for this summer of code 
> so the issue can be closed. 
>
> In addition to that would also like to work on improving the promotional 
> designs, banners, backdrop, certificates, and  presentation slides to make 
> it more concise for the duration of the workshop. Which will be presented 
> in different format for ease of editing. 
>
> I can understand that Django and Django girls somehow work hand in hand 
> but what am not sure of is if I will be considered since this is not a 
> Django girls Organization on the Gsoc. 
>
> This is not a formal proposal but if my idea will be considered I would 
> like to submit a more comprehensive proposal base on the stated idea.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5777ad40-beea-41be-8c23-3a9204863e12%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Channels - Groups and discovery

[Josh Smeaton]

Interesting - I hadn't realised the dispatch method was directly on the 
consumer. And I think I agree with you that these things can be handled in 
user code or 3rd party library code quite easily. I guess as channels is 
used more and more in production applications we'll begin to see a number 
of helper libraries or higher level abstractions come along to make it 
easier for users to safely implement their real time apps on top.

Cheers

On Monday, 26 March 2018 13:19:19 UTC+11, Andrew Godwin wrote:
>
> The code that tries to find a handler is just the dispatch method, so it's 
> actually quite easy to override even at a user level rather than in the 
> core itself: 
> https://github.com/django/channels/blob/master/channels/consumer.py#L61
>
> I would like to see a stronger solution for interfaces and type 
> assertions, but I also think this is something that could be done 
> externally quite easily (in fact we have a library we use at work for this 
> very thing for our redis-based message bus: 
> https://github.com/eventbrite/conformity). The channel layers being just 
> a dumb dictionary transport is quite a nice facade in terms of keeping them 
> low maintenance, and also in letting people solve 
> dependency/versioning/schema upgrade issues in the way that fits them best.
>
> Andrew
>
> On Sun, Mar 25, 2018 at 4:08 PM, Josh Smeaton <josh.s...@gmail.com 
> > wrote:
>
>> I see - some kind of python type that can be shared amongst consumers 
>> might make sense at the application level, but it doesn't at the protocol 
>> level. Specific patterns may emerge as application level helpers like 
>> Consumer subclasses that mixin specific Groups, and ASGI won't need to have 
>> any knowledge of these patterns.
>>
>> > There's also the idea of adding an easy way to ignore messages you're 
>> not interested in if you're using groups as a firehose rather than targeted 
>> broadcast
>>
>> Something like this would probably solve the biggest issue. At an 
>> application level I'd imagine it to be something along the lines of 
>> try_get_handler(message_type, consumer, 
>> ignore_missing=consumer.ignore_missing), but no idea if that's something 
>> that could be permitted by the spec/protocol. I'll familiarise myself with 
>> the spec and the implementation so I can understand what might and might 
>> not be possible at each layer before proposing any specific ideas.
>>
>> There's some fantastic stuff in channels though, and was fairly easy to 
>> hook things together after reading the docs. I'm looking forward to using 
>> it more. 
>>
>> On Sunday, 25 March 2018 02:24:00 UTC+11, Andrew Godwin wrote:
>>>
>>> Nobody else has suggested this particular approach yet, and while it 
>>> would definitely make writing applications much more reliable, there's no 
>>> particularly easy way to distribute an interface that I can think of (even 
>>> the ASGI specification ends up being enforced on the receiver side, though 
>>> ultimately it's the conformance test suites that I rely on for safety).
>>>
>>> Without a simple solution to that I think it's more the sort of thing I 
>>> would like to make sure people can hook into themselves to provide 
>>> validation rather than trying to ship something in the core release.
>>>
>>> There's also the idea of adding an easy way to ignore messages you're 
>>> not interested in if you're using groups as a firehose rather than targeted 
>>> broadcast - that does mean silent failures, though, which is my 
>>> least-favourite kind. At least with the current design pattern something 
>>> will complain if it's not working right (even though that thing is on the 
>>> receiving side not the sending side).
>>>
>>> Andrew
>>>
>>> On Sat, Mar 24, 2018 at 4:31 AM, Josh Smeaton <josh.s...@gmail.com> 
>>> wrote:
>>>
>>>> I've finally had the chance to use channels for a project (hack day 
>>>> multiplayer game - hope to release and blog about it some time soon), and 
>>>> I 
>>>> wanted to document some of the rough edges I hit and ask some questions 
>>>> about them. 
>>>>
>>>> Specifically though, I find the mapping of payload.type to a method on 
>>>> the consumer confusing and somewhat brittle.
>>>>
>>>> The design we went with was to have a 
>>>> PlayerConsumer(AsyncWebsocketConsumer) and a GameConsumer(SyncConsumer) 
>>>> running as a worker. The GameConsumer starts the engine in a thread, and 
>>>> let

Re: Channels - Groups and discovery

[Josh Smeaton]

I see - some kind of python type that can be shared amongst consumers might 
make sense at the application level, but it doesn't at the protocol level. 
Specific patterns may emerge as application level helpers like Consumer 
subclasses that mixin specific Groups, and ASGI won't need to have any 
knowledge of these patterns.

> There's also the idea of adding an easy way to ignore messages you're not 
interested in if you're using groups as a firehose rather than targeted 
broadcast

Something like this would probably solve the biggest issue. At an 
application level I'd imagine it to be something along the lines of 
try_get_handler(message_type, consumer, 
ignore_missing=consumer.ignore_missing), but no idea if that's something 
that could be permitted by the spec/protocol. I'll familiarise myself with 
the spec and the implementation so I can understand what might and might 
not be possible at each layer before proposing any specific ideas.

There's some fantastic stuff in channels though, and was fairly easy to 
hook things together after reading the docs. I'm looking forward to using 
it more. 

On Sunday, 25 March 2018 02:24:00 UTC+11, Andrew Godwin wrote:
>
> Nobody else has suggested this particular approach yet, and while it would 
> definitely make writing applications much more reliable, there's no 
> particularly easy way to distribute an interface that I can think of (even 
> the ASGI specification ends up being enforced on the receiver side, though 
> ultimately it's the conformance test suites that I rely on for safety).
>
> Without a simple solution to that I think it's more the sort of thing I 
> would like to make sure people can hook into themselves to provide 
> validation rather than trying to ship something in the core release.
>
> There's also the idea of adding an easy way to ignore messages you're not 
> interested in if you're using groups as a firehose rather than targeted 
> broadcast - that does mean silent failures, though, which is my 
> least-favourite kind. At least with the current design pattern something 
> will complain if it's not working right (even though that thing is on the 
> receiving side not the sending side).
>
> Andrew
>
> On Sat, Mar 24, 2018 at 4:31 AM, Josh Smeaton <josh.s...@gmail.com 
> > wrote:
>
>> I've finally had the chance to use channels for a project (hack day 
>> multiplayer game - hope to release and blog about it some time soon), and I 
>> wanted to document some of the rough edges I hit and ask some questions 
>> about them. 
>>
>> Specifically though, I find the mapping of payload.type to a method on 
>> the consumer confusing and somewhat brittle.
>>
>> The design we went with was to have a 
>> PlayerConsumer(AsyncWebsocketConsumer) and a GameConsumer(SyncConsumer) 
>> running as a worker. The GameConsumer starts the engine in a thread, and 
>> lets the engine fetch the channel layer. The player and engine then 
>> communicate like so:
>>
>> # PlayerConsumer: receive game state updates
>> await self.channel_layer.group_add(
>> self.group_name,
>> self.channel_name
>> )
>>
>> # PlayerConsumer: publish player joining event to game
>> await self.channel_layer.send(
>> 'game_engine',
>> {
>> 'type': 'player.new',
>> 'player': self.user.email,
>> 'channel': self.channel_name,
>> }
>> )
>>
>>
>> # GameConsumer: publish state update
>> async_to_sync(self.channel_layer.group_send)(
>> self.group_name,
>> {
>> 'type': 'game_update',
>> 'state': state_json,
>> }
>> )
>>
>> This works, provided PlayerConsumer has a method:
>>
>> async def game_update(self, event):
>>
>> And the GameConsumer has a method:
>>
>> def player_new(self, event):
>>
>> But if these two consumers are in completely different code 
>> bases/packages, there is no real way to know what the interface is between 
>> these consumers. Worse, it's extremely easy for a bad actor to crash a 
>> listening consumer. Either of the following events will crash the consumer 
>> receiving the message:
>>
>> await self.channel_layer.send(
>> 'game_engine',
>> {
>> 'type': 'i_am_a_bad_consumer'
>> }
>> )
>>
>> async_to_sync(self.channel_layer.group_send)(
>> self.group_name,
>> {
>> 'type': 'gme_update', # typo
>> 'msg': 'hi',
>> }
>> )
>>
>> I'm not so concerned about arbitrary asgi applications gatecrashing my 
>> app as I ultimately have control over what is going to run. But each 
>> consumer participating in a group must support the superset of all message 
>> types that may be sent t

Channels - Groups and discovery

[Josh Smeaton]

I've finally had the chance to use channels for a project (hack day 
multiplayer game - hope to release and blog about it some time soon), and I 
wanted to document some of the rough edges I hit and ask some questions 
about them. 

Specifically though, I find the mapping of payload.type to a method on the 
consumer confusing and somewhat brittle.

The design we went with was to have a 
PlayerConsumer(AsyncWebsocketConsumer) and a GameConsumer(SyncConsumer) 
running as a worker. The GameConsumer starts the engine in a thread, and 
lets the engine fetch the channel layer. The player and engine then 
communicate like so:

# PlayerConsumer: receive game state updates
await self.channel_layer.group_add(
self.group_name,
self.channel_name
)

# PlayerConsumer: publish player joining event to game
await self.channel_layer.send(
'game_engine',
{
'type': 'player.new',
'player': self.user.email,
'channel': self.channel_name,
}
)


# GameConsumer: publish state update
async_to_sync(self.channel_layer.group_send)(
self.group_name,
{
'type': 'game_update',
'state': state_json,
}
)

This works, provided PlayerConsumer has a method:

async def game_update(self, event):

And the GameConsumer has a method:

def player_new(self, event):

But if these two consumers are in completely different code bases/packages, 
there is no real way to know what the interface is between these consumers. 
Worse, it's extremely easy for a bad actor to crash a listening consumer. 
Either of the following events will crash the consumer receiving the 
message:

await self.channel_layer.send(
'game_engine',
{
'type': 'i_am_a_bad_consumer'
}
)

async_to_sync(self.channel_layer.group_send)(
self.group_name,
{
'type': 'gme_update', # typo
'msg': 'hi',
}
)

I'm not so concerned about arbitrary asgi applications gatecrashing my app 
as I ultimately have control over what is going to run. But each consumer 
participating in a group must support the superset of all message types 
that may be sent to that group if it wants to avoid crashing. And it has to 
support the superset of message types without being able to discover what 
they might be. Oh, and they're strings that **might** match a method I have 
if periods are converted to underscores.

This is all very good for adhoc eventing, but I think at some point you're 
going to want to publish an explicit Interface. A GroupInterface might 
define a collection of message types that are valid for members 
participating in that group, and then a consumer could subscribe to 
**that** rather than just a name. The interface members might be optional 
or required, I haven't thought that far ahead, but at least each member 
would be known. Attempting to publish an unknown message to a group would 
crash the **sender** rather than the **receiver**.

Has anything like this come up before?

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/939460e7-45da-4dd0-afb7-f386274faea1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: GSoC 2018

[Josh Smeaton]

This isn't going to work.

1. If you're going to "pre-render" your python files into html files 
**before** they're filled with data, then you will still need to run 
django/jinja templating over this pre-rendered template.
2. If you're **not** going to pre-render your python files into html files, 
then all dynamic data will need to be used in your python file, which will 
generate the HTML at run time. Then this is exactly no different to what we 
currently have, except you'd be using a mimic of html as python objects.
3. In either case, this is useless to frontend devs. In the first case, 
you'd end up overwriting any changes a frontend dev made to the HTML 
pre-rendered output each time you compiled your python files. In the second 
case, the frontend dev never gets to see what the rendered html output is.

Again, thank you for your proposal, but this isn't something the django 
project wants, and is not something we'd sponsor as part of GSoC.


On Saturday, 17 March 2018 07:48:46 UTC+11, Manasvi Saxena wrote:
>
> Hello Sir,
>
> On Saturday, March 17, 2018 at 1:57:00 AM UTC+5:30, Josh Smeaton wrote:
>>
>> I'm confused by your proposal. You state one of the drawbacks with 
>> current templating is the readability issues for frontend developers that 
>> have templating logic defined within the template. But isn't your proposal 
>> to replace html templates with one defined in python? How does this make it 
>> any better for a frontend developer? It sounds like it'd be strictly worse.
>>
>> Further, sites will still need to query the ORM and inject that data into 
>> the template. I don't see how your point number 3 is addressed at all - and 
>> don't even think it can be.
>>
>> A new templating engine will also be incompatible with the thousands of 
>> django and python libraries already available in the wild.
>>
>> It sounds like you're attached to your project, which is a good thing, 
>> but I'd encourage you to work on this separately from GSoC as a learning 
>> experience, as Aymeric mentioned before. This is definitely not a project 
>> that would be accepted for GSoC, without actual proof that the project was 
>> **already used by people** and provably better than the options we already 
>> have.
>>
>> Regards,
>>
>> On Saturday, 17 March 2018 04:09:41 UTC+11, Manasvi Saxena wrote:
>>>
>>> Hello Sir,
>>>
>>> On Thursday, March 15, 2018 at 11:29:46 PM UTC+5:30, Aymeric Augustin 
>>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> 2018-03-15 14:24 GMT+01:00 Manasvi Saxena <msma...@gmail.com>:
>>>>
>>>>> Think of this library as a bridge between Python and HTML, you don't 
>>>>> need to know the syntax of HTML, just need to know which function of the 
>>>>> library creates what output. After you have created the content of your 
>>>>> page just pass it to a function in a list and you have your HTML page.
>>>>> Have a look at the prototype and let me know what you think of it.
>>>>>
>>>>
>>>> Thanks for the example.
>>>>
>>>> To be honest the sample.py file doesn't strike me as a convenient way 
>>>> to write HTML.
>>>>
>>>> I think someone who's confortable writing Python code should be able to 
>>>> learn the HTML syntax very quickly.
>>>>
>>>> content isn't more complicated 
>>>> than pyhtml.tag("content", attr1="value1", attr2="value2"), is it?
>>>>
>>>> This idea is not a good fit for Django. Perhaps it can be a learning 
>>>> experiment for you, though?
>>>>
>>>> Best regards,
>>>>
>>>> -- 
>>>> Aymeric.
>>>>
>>>
>>>
>>> After keeping in mind your valuable feedback I have decided to change my 
>>> project proposal.
>>>
>>> I'm halfway through my formal proposal which contains the details 
>>> required as per the guidelines provided on Django GSoC page.
>>> But would like to discuss with you the idea first for your valuable 
>>> feedback.
>>>  
>>> Django templating engine and jinja2 while being very efficient in their 
>>> way still have some drawbacks.
>>> 1)Not enough freedom to implement logic in a template. Python functions 
>>> can't be called inside the template.
>>> 2)Once the templating language syntax is applied to the template it 
>>> creates readability issues for the front-end developer. This leads to two 
>>> dif

Re: [RFC]GSoC 2018 Test framework improvement

[Josh Smeaton]

Hi Billy,

This plan sounds more like a plan for a plan. If you're serious about 
wanting to participate in GSoC, then you need to have **specific** ideas 
and goals. By this, I mean your proposal should be able to be picked up by 
anyone and implemented. You're missing any actual feature proposals. To be 
successfully chosen, what you'd need to demonstrate is that you have a good 
understanding of how things already work, and the problems that people 
face. You'd also accompany your proposal with code/test examples of **how 
things are now** and **how things will be** once you've completed the 
project.

Regards,

On Saturday, 17 March 2018 04:56:53 UTC+11, Su Billy wrote:
>
> Hello developers, 
>
> My name is BING-LI SU from Taiwan, you can simply call me Billy.
>
> *Background*
> I'm an undergraduate student at National Chiao Tung University, major in 
> Computer Science.
> I have been contributed to several open source projects for one year, such 
> as CDNJS, pyqtgraph, and I host my own project using Tornado Web to build a 
> simple digital signage system which contain a little bit unit test, CI, and 
> docker image for fast deployment.
> I have know Django when I chose the framework of the digital signage, but 
> I haven't used it util these few days, knowing that Django is a more 
> sophisticated and handy framework compared to Tornado. In my opinion, the 
> best way to be familiar with a project is to understand the test of the 
> project (if there has one), therefore I choose test framework as my 
> proposal idea.
>
> *Ideas*
> I'd like to add some new feature to the test framework, it is mainly about 
> provide some groups of common used tests for user to choose, and maybe 
> provide hook function to add their own test before, after, during each 
> group of tests.
>
> *Plan*
> Here is my draft plan to achieve above ideas
> 1. To make me better understand the whole test framework, I will briefly 
> trace the test framework in May
> 2. Find out and discuss with mentor about how to group up the tests and 
> what this feature should look like.
> 3. Implementation and discuss with mentor once or twice a week.
>
> Thanks for any suggestion or comments, i.e: I should start to trace which 
> part of test framework? The idea has some problem...? etc.
>
> sincerely
> Billy Su
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/381ad22a-86f7-4982-bf63-a680274aaa86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: GSoC 2018

[Josh Smeaton]

I'm confused by your proposal. You state one of the drawbacks with current 
templating is the readability issues for frontend developers that have 
templating logic defined within the template. But isn't your proposal to 
replace html templates with one defined in python? How does this make it 
any better for a frontend developer? It sounds like it'd be strictly worse.

Further, sites will still need to query the ORM and inject that data into 
the template. I don't see how your point number 3 is addressed at all - and 
don't even think it can be.

A new templating engine will also be incompatible with the thousands of 
django and python libraries already available in the wild.

It sounds like you're attached to your project, which is a good thing, but 
I'd encourage you to work on this separately from GSoC as a learning 
experience, as Aymeric mentioned before. This is definitely not a project 
that would be accepted for GSoC, without actual proof that the project was 
**already used by people** and provably better than the options we already 
have.

Regards,

On Saturday, 17 March 2018 04:09:41 UTC+11, Manasvi Saxena wrote:
>
> Hello Sir,
>
> On Thursday, March 15, 2018 at 11:29:46 PM UTC+5:30, Aymeric Augustin 
> wrote:
>>
>> Hello,
>>
>> 2018-03-15 14:24 GMT+01:00 Manasvi Saxena :
>>
>>> Think of this library as a bridge between Python and HTML, you don't 
>>> need to know the syntax of HTML, just need to know which function of the 
>>> library creates what output. After you have created the content of your 
>>> page just pass it to a function in a list and you have your HTML page.
>>> Have a look at the prototype and let me know what you think of it.
>>>
>>
>> Thanks for the example.
>>
>> To be honest the sample.py file doesn't strike me as a convenient way to 
>> write HTML.
>>
>> I think someone who's confortable writing Python code should be able to 
>> learn the HTML syntax very quickly.
>>
>> content isn't more complicated 
>> than pyhtml.tag("content", attr1="value1", attr2="value2"), is it?
>>
>> This idea is not a good fit for Django. Perhaps it can be a learning 
>> experiment for you, though?
>>
>> Best regards,
>>
>> -- 
>> Aymeric.
>>
>
>
> After keeping in mind your valuable feedback I have decided to change my 
> project proposal.
>
> I'm halfway through my formal proposal which contains the details required 
> as per the guidelines provided on Django GSoC page.
> But would like to discuss with you the idea first for your valuable 
> feedback.
>  
> Django templating engine and jinja2 while being very efficient in their 
> way still have some drawbacks.
> 1)Not enough freedom to implement logic in a template. Python functions 
> can't be called inside the template.
> 2)Once the templating language syntax is applied to the template it 
> creates readability issues for the front-end developer. This leads to two 
> different development branches.
> If a front-end developer changes some of the parts in his template then 
> the back-end developer has to make amendments manually each time.
> His job is not finished even after he has generated the content of the 
> page dynamically.
> 3)The time required by the engines to generate the HTML content is more 
> for large data. First, the QRM fetches the data from the database, then 
> templating engine populates it on the template.
>
> I intend to first generate the HTML content according to the requirements 
> of the developer using his programming skills in an exact way required and 
> then place it inside the template.
>
> The library will have functions exactly similar to the HTML tags so that a 
> python developer can reproduce the HTML code made by the front-end 
> developer. 
> Or a person with some knowledge of HTML or following the documentation of 
> the library can design the page according to his requirement.
>
> The main benefit of doing this is that freedom in generating any HTML 
> content can be given to the developer.
>
> I'm not experienced enough to comment on any of the drawbacks of 
> templating engine but I listed these points based on my own personal 
> experience I got from some of my past projects.
> And hence I need to know your views on this and would really appreciate if 
> you could suggest me anything from your side.
>
> Regards,
> Manasvi Saxena
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/c4a290e2-ca6a-476e-8339-57c967db6a53%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add Alias or annotations without group-by support?

[Josh Smeaton]

Sure - but you can always save the expression to a variable and use it 
multiple times.

mycalc = MyFunc('a', 'b')
Model.objects.filter(GreaterEqual(mycalc, 0.6)).order_by(mycalc)

I think we already have the building blocks we need to avoid adding another 
queryset method.

On Saturday, 10 March 2018 14:01:41 UTC+11, Cristiano Coelho wrote:
>
> It wouldn't work if you also want to order by the annotated value.
>
> El viernes, 9 de marzo de 2018, 8:27:36 (UTC-3), Josh Smeaton escribió:
>>
>> Would teaching filter() and friends to use expressions directly solve 
>> your issue? You suggested using `alias` upthread, but that's only really 
>> required so you can refer to it later? Unless you wanted to refer to the 
>> field more than once, having each queryset method respect expressions 
>> should be enough I think.
>>
>> https://github.com/django/django/pull/8119 adds boolean expression 
>> support to filter. I believe most other queryset methods have support for 
>> expressions now (order_by, values/values_list).
>>
>> For the alias used multiple times case, it should be enough to annotate 
>> and then restrict with values if you don't actually want it in the 
>> select/group list.
>>
>> On Friday, 9 March 2018 00:22:00 UTC+11, Cristiano Coelho wrote:
>>>
>>> The workaround, although extremely ugly and which will probably cause 
>>> issues in the future (reason I only used it for the model I needed to do 
>>> those odd queries) was to use a custom queryset/manager. Something like 
>>> this.
>>>
>>> class FasterCountQuerySet(QuerySet):
>>> def count(self):
>>> return super(FasterCountQuerySet, self.values('pk')).count()
>>> FasterCountManager = Manager.from_queryset(FasterCountQuerySet)
>>>
>>> But again, this is extremely ugly and will still cause a subquery, but 
>>> without the unnecessary group by and extra function calls.
>>>
>>>
>>> El miércoles, 7 de marzo de 2018, 19:48:01 (UTC-3), Jared Proffitt 
>>> escribió:
>>>>
>>>> I have also run into this exact problem. Would love to get this fixed. 
>>>> Have you found a good workaround?
>>>>
>>>> On Tuesday, December 26, 2017 at 12:37:16 PM UTC-6, Cristiano Coelho 
>>>> wrote:
>>>>>
>>>>> Hello, I'm having a hard time explaining the exact issue but I hope 
>>>>> it's clear enough.
>>>>>
>>>>>
>>>>> Following this issue (
>>>>> https://groups.google.com/forum/#!searchin/django-users/cristiano%7Csort:date/django-users/q6XdfyK29HA/TcE8oFitBQAJ)
>>>>>  
>>>>> from django users and a related ticket (
>>>>> https://code.djangoproject.com/ticket/27719) that seems to be left 
>>>>> out or forgotten already.
>>>>>
>>>>> There has to be a way to alias or annotate a value given an expression 
>>>>> or SQL Function that doesn't necessarily aggregates data but rather work 
>>>>> on 
>>>>> a single value.
>>>>>
>>>>> Right now as shown on the django-users post, using annotate for this 
>>>>> purpose will cause unexpected grouping and sub querying that could result 
>>>>> in very slow and hard to debug queries.
>>>>>
>>>>> The core issue is that using annotate without a previous call either 
>>>>> vaues or values_list, will work as expected, simply annotating a value 
>>>>> and 
>>>>> returning it as an additional column, but if an aggregate is added 
>>>>> afterwards (such as count), the final query ends up being a redundant 
>>>>> query 
>>>>> where the annotated value is added to a group by clause (group by id + 
>>>>> column), to a column as part of the select (function called twice) and 
>>>>> then 
>>>>> wrapped into a select * (subquery), which makes the extra column as part 
>>>>> of 
>>>>> the select and group by useless, unless the query had any kind of 
>>>>> left/inner join in which case the group by might make sense (although not 
>>>>> sure about the column showing up on the select clause)
>>>>>
>>>>> The ugly work around is to simply add a .values('id') at the end so 
>>>>> the annotated value doesn't show on the group by and select sections, 
>>>>> although the nested query still happens.
>>>>>
>>>>>
>

Re: Add Alias or annotations without group-by support?

[Josh Smeaton]

Would teaching filter() and friends to use expressions directly solve your 
issue? You suggested using `alias` upthread, but that's only really 
required so you can refer to it later? Unless you wanted to refer to the 
field more than once, having each queryset method respect expressions 
should be enough I think.

https://github.com/django/django/pull/8119 adds boolean expression support 
to filter. I believe most other queryset methods have support for 
expressions now (order_by, values/values_list).

For the alias used multiple times case, it should be enough to annotate and 
then restrict with values if you don't actually want it in the select/group 
list.

On Friday, 9 March 2018 00:22:00 UTC+11, Cristiano Coelho wrote:
>
> The workaround, although extremely ugly and which will probably cause 
> issues in the future (reason I only used it for the model I needed to do 
> those odd queries) was to use a custom queryset/manager. Something like 
> this.
>
> class FasterCountQuerySet(QuerySet):
> def count(self):
> return super(FasterCountQuerySet, self.values('pk')).count()
> FasterCountManager = Manager.from_queryset(FasterCountQuerySet)
>
> But again, this is extremely ugly and will still cause a subquery, but 
> without the unnecessary group by and extra function calls.
>
>
> El miércoles, 7 de marzo de 2018, 19:48:01 (UTC-3), Jared Proffitt 
> escribió:
>>
>> I have also run into this exact problem. Would love to get this fixed. 
>> Have you found a good workaround?
>>
>> On Tuesday, December 26, 2017 at 12:37:16 PM UTC-6, Cristiano Coelho 
>> wrote:
>>>
>>> Hello, I'm having a hard time explaining the exact issue but I hope it's 
>>> clear enough.
>>>
>>>
>>> Following this issue (
>>> https://groups.google.com/forum/#!searchin/django-users/cristiano%7Csort:date/django-users/q6XdfyK29HA/TcE8oFitBQAJ)
>>>  
>>> from django users and a related ticket (
>>> https://code.djangoproject.com/ticket/27719) that seems to be left out 
>>> or forgotten already.
>>>
>>> There has to be a way to alias or annotate a value given an expression 
>>> or SQL Function that doesn't necessarily aggregates data but rather work on 
>>> a single value.
>>>
>>> Right now as shown on the django-users post, using annotate for this 
>>> purpose will cause unexpected grouping and sub querying that could result 
>>> in very slow and hard to debug queries.
>>>
>>> The core issue is that using annotate without a previous call either 
>>> vaues or values_list, will work as expected, simply annotating a value and 
>>> returning it as an additional column, but if an aggregate is added 
>>> afterwards (such as count), the final query ends up being a redundant query 
>>> where the annotated value is added to a group by clause (group by id + 
>>> column), to a column as part of the select (function called twice) and then 
>>> wrapped into a select * (subquery), which makes the extra column as part of 
>>> the select and group by useless, unless the query had any kind of 
>>> left/inner join in which case the group by might make sense (although not 
>>> sure about the column showing up on the select clause)
>>>
>>> The ugly work around is to simply add a .values('id') at the end so the 
>>> annotated value doesn't show on the group by and select sections, although 
>>> the nested query still happens.
>>>
>>>
>>> For this reason, there's currently no way to achieve the above without 
>>> ugly work arounds or unnecessary database performance hits.
>>>
>>> The easiest option I believe would be to follow the ticket in order to 
>>> implement an alias call that works exactly like annotate but doesn't 
>>> trigger any grouping.
>>>
>>> A more complicated option is probably trying to make annotate/aggregate 
>>> smarter, so all the unnecessary grouping and sub querying doesn't happen 
>>> unless needed, for example, if the queryset didn't call values/values_list 
>>> or if there are no relationships/joins used.
>>>
>>>
>>> Example/demostration:
>>>
>>> Given the following queryset
>>>
>>> query1 = MyModel.objects.annotate(x=MyFunction('a', 'b')).filter(x__gte=
>>> 0.6).order_by('-x')
>>>
>>>
>>> query1 SQL is good and looks like:
>>>
>>> SELECT id, a, b, myfunction(a, b) as x
>>> FROM mymodel
>>> WHERE myfunction(a, b) >= 0.6
>>> ORDER BY x desc
>>>
>>> Notice how there's no group by, the ORM was smart enough to not include 
>>> it since there was no previous call to values/values_list
>>>
>>>
>>> If we run query1.count() the final SQL looks like:
>>>
>>> SELECT COUNT(*) FROM (
>>> SELECT id, myfunction(a, b) as x
>>> FROM mymodel
>>> WHERE myfunction(a ,b) >= 0.6
>>> GROUP BY id, myfunction(a ,b)
>>> ) subquery
>>>
>>> which if myfunction is slow, will add a massive slow down that's not 
>>> even needed, and should actually be just:
>>>
>>> SELECT count(*)
>>> FROM mymodel
>>> WHERE myfunction(a ,b) >= 0.6
>>>
>>>
>>> while the other query should ONLY happen if the group by makes sense 
>>> (i.e, if there's a join 

Re: Consider renaming `mark_safe` to `dangerously_trust_html` (etc)

[Josh Smeaton]

We could use the same name or just |trusted.

I'm not so concerned with the `safe` tag as it's already an adjective, but 
I would be ok with an alias for trusted also.

On Friday, 23 February 2018 08:24:50 UTC+11, Florian Apolloner wrote:
>
>
>
> On Thursday, February 22, 2018 at 9:52:04 PM UTC+1, Josh Smeaton wrote:
>>
>> Or, since this isn't a template tag (facepalm) 
>>
>
> Which raises a perfectly valid point: If we were to change this, we will 
> also need to come up with a new name for the "|safe"-filter. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/f2a90aa8-87b8-476f-b72e-1b1c2e6fe4e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Consider renaming `mark_safe` to `dangerously_trust_html` (etc)

[Josh Smeaton]

Or, since this isn't a template tag (facepalm) 

trusted_html(m.content)
trust_html(m.content) 

On Friday, 23 February 2018 07:46:45 UTC+11, Josh Smeaton wrote:
>
> Yes I'm not a fan of the *dangerously...* names either. I'm still 
> somewhat wary of *trust_html* which is a *verb* and could still confuse 
> users in a similar way (does the api make it trustworthy?). I think I'd 
> prefer something more descriptive like *trusted_html*.
>
> {{ content|trusted_html }}
> vs
> {{ content|trust_html }}
>
> On Friday, 23 February 2018 01:07:12 UTC+11, Adam Johnson wrote:
>>
>> I am also in favour of a rename without deprecating the old name.
>>
>> I like 'trust_html' - it's still similarly short but as Tom says it 
>> implies more than 'mark_safe' does.
>>
>> On 22 February 2018 at 08:30, Tom Forbes <t...@tomforb.es> wrote:
>>
>>> What about just 'trust_html'? The dangerous part is quite context 
>>> dependent (and a bit of mouth-full), but at the core you are trusting the 
>>> HTML. Hopefully it follows that you should not trust html with user input 
>>> that hasn't been escaped.
>>>
>>>
>>> On 22 Feb 2018 13:10, "Anthony King" <anthon...@gmail.com> wrote:
>>>
>>> I entirely agree with renaming `mark_safe`. Though it's name is correct, 
>>> it doesn't convey the gravity of what this actually does.
>>> However I'm unsure on the `dangerously_trust_html` name. It wouldn't be 
>>> dangerous to trust the literal "Some Content", for example.
>>>
>>> Perhaps it could be something a bit more explicit. `no_escape(string)`?
>>> This assumes that most have at least heard of escaping.
>>>
>>>
>>> On 22 February 2018 at 12:16, Josh Smeaton <josh.s...@gmail.com> wrote:
>>>
>>>> The concern isn't overusing an API. It's not understanding the proper 
>>>> use case for it.
>>>>
>>>> "mark safe" can sound like the API is doing sanitation so it can 
>>>> encourage developers to use it incorrectly. I'm fairly sure I've done this 
>>>> myself.
>>>>
>>>> The intended meaning is "this output is **already** safe" but the name 
>>>> doesn't convey that meaning clearly enough.
>>>>
>>>> What the proposal is designed to do is convey the "I trust this output" 
>>>> meaning of the API. I'm just wary of enforcing users to change code when 
>>>> they already use the API correctly.
>>>>
>>>> On Thursday, 22 February 2018 21:08:31 UTC+11, Florian Apolloner wrote:
>>>>>
>>>>> Yeah, I am also worried about the churn for no gain in my eyes. If 
>>>>> users overuse mark_safe, they will overuse dangerously_trust_html too…
>>>>>
>>>>> On Wednesday, February 21, 2018 at 10:41:15 PM UTC+1, Josh Smeaton 
>>>>> wrote:
>>>>>>
>>>>>> I agree that the names are misleading and we should probably provide 
>>>>>> better names. I'm wary of deprecating the old names because it'll create 
>>>>>> a 
>>>>>> lot of churn (some of which would be the right thing to do). Maybe we 
>>>>>> could 
>>>>>> just alias and warn when using the old name, leaving a decision on 
>>>>>> deprecation until some time in the future.
>>>>>>
>>>>>> On Monday, 29 January 2018 03:14:27 UTC+11, Stuart Cox wrote:
>>>>>>>
>>>>>>> In my experience, misuse of mark_safe() — i.e. marking stuff safe 
>>>>>>> which *isn’t* actually safe (e.g. HTML from a rich text input) — is 
>>>>>>> one of the biggest causes of XSS vulnerabilities in Django projects.
>>>>>>>
>>>>>>> The docs warn to be careful, but unfortunately I think Django devs 
>>>>>>> have just got too used to mark_safe() being *the way* to insert 
>>>>>>> HTML in a template. And it’s easy for something that was safe when it 
>>>>>>> was 
>>>>>>> authored (e.g. calling mark_safe() on a hard-coded string) to be 
>>>>>>> copied / repurposed / adapted into a case which is no longer be safe 
>>>>>>> (e.g. 
>>>>>>> that string replaced with a user-provided value).
>>>>>>>
>>>>>>> Some other frameworks use scary sounding names to help reinforce 
>>>>>>> t

Re: Consider renaming `mark_safe` to `dangerously_trust_html` (etc)

[Josh Smeaton]

Yes I'm not a fan of the *dangerously...* names either. I'm still somewhat 
wary of *trust_html* which is a *verb* and could still confuse users in a 
similar way (does the api make it trustworthy?). I think I'd prefer 
something more descriptive like *trusted_html*.

{{ content|trusted_html }}
vs
{{ content|trust_html }}

On Friday, 23 February 2018 01:07:12 UTC+11, Adam Johnson wrote:
>
> I am also in favour of a rename without deprecating the old name.
>
> I like 'trust_html' - it's still similarly short but as Tom says it 
> implies more than 'mark_safe' does.
>
> On 22 February 2018 at 08:30, Tom Forbes <t...@tomforb.es > 
> wrote:
>
>> What about just 'trust_html'? The dangerous part is quite context 
>> dependent (and a bit of mouth-full), but at the core you are trusting the 
>> HTML. Hopefully it follows that you should not trust html with user input 
>> that hasn't been escaped.
>>
>>
>> On 22 Feb 2018 13:10, "Anthony King" <anthon...@gmail.com > 
>> wrote:
>>
>> I entirely agree with renaming `mark_safe`. Though it's name is correct, 
>> it doesn't convey the gravity of what this actually does.
>> However I'm unsure on the `dangerously_trust_html` name. It wouldn't be 
>> dangerous to trust the literal "Some Content", for example.
>>
>> Perhaps it could be something a bit more explicit. `no_escape(string)`?
>> This assumes that most have at least heard of escaping.
>>
>>
>> On 22 February 2018 at 12:16, Josh Smeaton <josh.s...@gmail.com 
>> > wrote:
>>
>>> The concern isn't overusing an API. It's not understanding the proper 
>>> use case for it.
>>>
>>> "mark safe" can sound like the API is doing sanitation so it can 
>>> encourage developers to use it incorrectly. I'm fairly sure I've done this 
>>> myself.
>>>
>>> The intended meaning is "this output is **already** safe" but the name 
>>> doesn't convey that meaning clearly enough.
>>>
>>> What the proposal is designed to do is convey the "I trust this output" 
>>> meaning of the API. I'm just wary of enforcing users to change code when 
>>> they already use the API correctly.
>>>
>>> On Thursday, 22 February 2018 21:08:31 UTC+11, Florian Apolloner wrote:
>>>>
>>>> Yeah, I am also worried about the churn for no gain in my eyes. If 
>>>> users overuse mark_safe, they will overuse dangerously_trust_html too…
>>>>
>>>> On Wednesday, February 21, 2018 at 10:41:15 PM UTC+1, Josh Smeaton 
>>>> wrote:
>>>>>
>>>>> I agree that the names are misleading and we should probably provide 
>>>>> better names. I'm wary of deprecating the old names because it'll create 
>>>>> a 
>>>>> lot of churn (some of which would be the right thing to do). Maybe we 
>>>>> could 
>>>>> just alias and warn when using the old name, leaving a decision on 
>>>>> deprecation until some time in the future.
>>>>>
>>>>> On Monday, 29 January 2018 03:14:27 UTC+11, Stuart Cox wrote:
>>>>>>
>>>>>> In my experience, misuse of mark_safe() — i.e. marking stuff safe 
>>>>>> which *isn’t* actually safe (e.g. HTML from a rich text input) — is 
>>>>>> one of the biggest causes of XSS vulnerabilities in Django projects.
>>>>>>
>>>>>> The docs warn to be careful, but unfortunately I think Django devs 
>>>>>> have just got too used to mark_safe() being *the way* to insert HTML 
>>>>>> in a template. And it’s easy for something that was safe when it was 
>>>>>> authored (e.g. calling mark_safe() on a hard-coded string) to be 
>>>>>> copied / repurposed / adapted into a case which is no longer be safe 
>>>>>> (e.g. 
>>>>>> that string replaced with a user-provided value).
>>>>>>
>>>>>> Some other frameworks use scary sounding names to help reinforce that 
>>>>>> there are dangers around similar features, and that this isn’t something 
>>>>>> you should use in everyday work — e.g. React’s 
>>>>>> dangerouslySetInnerHTML.
>>>>>>
>>>>>> Relatedly, this topic 
>>>>>> <https://groups.google.com/d/msg/django-developers/c4fa2pOcHxo/EtT942WnyiAJ>
>>>>>>  suggested 
>>>>>> making it more explicit that mark_safe() refers to being safe f

Re: Consider renaming `mark_safe` to `dangerously_trust_html` (etc)

[Josh Smeaton]

The concern isn't overusing an API. It's not understanding the proper use 
case for it.

"mark safe" can sound like the API is doing sanitation so it can encourage 
developers to use it incorrectly. I'm fairly sure I've done this myself.

The intended meaning is "this output is **already** safe" but the name 
doesn't convey that meaning clearly enough.

What the proposal is designed to do is convey the "I trust this output" 
meaning of the API. I'm just wary of enforcing users to change code when 
they already use the API correctly.

On Thursday, 22 February 2018 21:08:31 UTC+11, Florian Apolloner wrote:
>
> Yeah, I am also worried about the churn for no gain in my eyes. If users 
> overuse mark_safe, they will overuse dangerously_trust_html too…
>
> On Wednesday, February 21, 2018 at 10:41:15 PM UTC+1, Josh Smeaton wrote:
>>
>> I agree that the names are misleading and we should probably provide 
>> better names. I'm wary of deprecating the old names because it'll create a 
>> lot of churn (some of which would be the right thing to do). Maybe we could 
>> just alias and warn when using the old name, leaving a decision on 
>> deprecation until some time in the future.
>>
>> On Monday, 29 January 2018 03:14:27 UTC+11, Stuart Cox wrote:
>>>
>>> In my experience, misuse of mark_safe() — i.e. marking stuff safe which 
>>> *isn’t* actually safe (e.g. HTML from a rich text input) — is one of 
>>> the biggest causes of XSS vulnerabilities in Django projects.
>>>
>>> The docs warn to be careful, but unfortunately I think Django devs have 
>>> just got too used to mark_safe() being *the way* to insert HTML in a 
>>> template. And it’s easy for something that was safe when it was authored 
>>> (e.g. calling mark_safe() on a hard-coded string) to be copied / 
>>> repurposed / adapted into a case which is no longer be safe (e.g. that 
>>> string replaced with a user-provided value).
>>>
>>> Some other frameworks use scary sounding names to help reinforce that 
>>> there are dangers around similar features, and that this isn’t something 
>>> you should use in everyday work — e.g. React’s dangerouslySetInnerHTML.
>>>
>>> Relatedly, this topic 
>>> <https://groups.google.com/d/msg/django-developers/c4fa2pOcHxo/EtT942WnyiAJ>
>>>  suggested 
>>> making it more explicit that mark_safe() refers to being safe for use 
>>> in *HTML* contexts (rather than JS, CSS, SQL, etc).
>>>
>>> Combining the two, it would be great if Django could rename mark_safe() to 
>>> dangerously_trust_html(), |safe to |dangerously_trust_html, @csrf_exempt to 
>>> @dangerously_csrf_exempt, etc.
>>>
>>> Developers who know what they’re doing with these could then be 
>>> encouraged to create suitable wrappers which handle their use case safely 
>>> internally — e.g.:
>>>
>>> @register.filter
>>> def sanitize_and_trust_html(value):
>>> # Safe because we sanitize before trusting
>>> return dangerously_trust_html(bleach.clean(value))
>>>
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/db4ac958-89e1-4286-a616-99e9854c9bbb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Consider renaming `mark_safe` to `dangerously_trust_html` (etc)

[Josh Smeaton]

I agree that the names are misleading and we should probably provide better 
names. I'm wary of deprecating the old names because it'll create a lot of 
churn (some of which would be the right thing to do). Maybe we could just 
alias and warn when using the old name, leaving a decision on deprecation 
until some time in the future.

On Monday, 29 January 2018 03:14:27 UTC+11, Stuart Cox wrote:
>
> In my experience, misuse of mark_safe() — i.e. marking stuff safe which 
> *isn’t* actually safe (e.g. HTML from a rich text input) — is one of the 
> biggest causes of XSS vulnerabilities in Django projects.
>
> The docs warn to be careful, but unfortunately I think Django devs have 
> just got too used to mark_safe() being *the way* to insert HTML in a 
> template. And it’s easy for something that was safe when it was authored 
> (e.g. calling mark_safe() on a hard-coded string) to be copied / 
> repurposed / adapted into a case which is no longer be safe (e.g. that 
> string replaced with a user-provided value).
>
> Some other frameworks use scary sounding names to help reinforce that 
> there are dangers around similar features, and that this isn’t something 
> you should use in everyday work — e.g. React’s dangerouslySetInnerHTML.
>
> Relatedly, this topic 
>  
> suggested 
> making it more explicit that mark_safe() refers to being safe for use in 
> *HTML* contexts (rather than JS, CSS, SQL, etc).
>
> Combining the two, it would be great if Django could rename mark_safe() to 
> dangerously_trust_html(), |safe to |dangerously_trust_html, @csrf_exempt to 
> @dangerously_csrf_exempt, etc.
>
> Developers who know what they’re doing with these could then be encouraged 
> to create suitable wrappers which handle their use case safely internally — 
> e.g.:
>
> @register.filter
> def sanitize_and_trust_html(value):
> # Safe because we sanitize before trusting
> return dangerously_trust_html(bleach.clean(value))
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/05de4602-5c44-41bf-b675-ab15d69fb46d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Purpose of constant_time_compare?

[Josh Smeaton]

I don't doubt that timing attacks can work in practise, but the link you've 
posted is not this. From the commit message:

> There have not been any reports of attempted exploit in the wild. (The 
vulnerability was discovered through code review.)

On Friday, 9 February 2018 09:27:03 UTC+11, Anssi Porttikivi wrote:
>
>
>> https://github.com/anymail/django-anymail/commit/c07998304b4a31df4c61deddcb03d3607a04691b
>>
> It happened for real.  
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/13837ef0-3bb4-4c14-8c00-f46de8049e91%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: PR 9583 - Ticket 28643

[Josh Smeaton]

I think having Left/Right is the right way to go. If there are 
opportunities to use slicing syntax, then they would resolve internally to 
those Expressions I would think. But that can be done separately, and have 
greater scope such as ArrayField in Simons example.

On Friday, 19 January 2018 08:21:57 UTC+11, charettes wrote:
>
> Hey Matthew,
>
> I could see adding support for F('field')[0:5] being useful and more 
> intuitive
> than using Left/Right/Substr. You should be able to assert the field is 
> sliceable
> (e.g. an instance of CharField/TextField) in the expression returned by
> F.__getitem__ (e.g. Left/Right/Substr) resolve_expression method.
>
> Maybe having a F.__getitem__ return an intermediary Expression subclass
> (e.g. FieldSlice) that resolves to the correct expression based on the 
> type of
> the sliced field and the nature of the slice in resolve_expression would 
> work
> better here.
>
> I could see this having great benefit for usage with nested data structure 
> fields
> such as contrib.postgres ArrayField (e.g F('array')[0:10]) and JSONField 
> where
> we would finally have a way to specify a key without colliding with the 
> builtin
> lookups (e.g. F('data')['exact'], F('data')['exact'][0:10]).
>
> That might be bit too much for this ticket though and you might want to 
> only
> focus on adding Left/Right/Substr for now.
>
> Best,
> Simon
>
>
> Le jeudi 18 janvier 2018 13:06:29 UTC-5, matthew.pava a écrit :
>>
>> Hi everyone,
>>
>> I’ve been working on ticket 28643 specifically adding Ord, Chr, Left, and 
>> Right to Django’s built-in database functions.  I’ve done the PR #9583.  I 
>> am a new contributor, and I’ve never really contributed code to any project 
>> through pull requests ever before.  I appreciate your guidance.  The checks 
>> have passed, and I’ve got a mess of commits that I’ve tried to squash a 
>> couple of times but seem to stick around nonetheless.
>>
>>  
>>
>> I also made a comment on the ticket that maybe instead of using Left, 
>> Right, and Substr functions that we instead implement slicing on fields and 
>> expressions to make things more pythonic (and deprecate Left, Right, and 
>> Substr).  Would such an implementation need a new ticket?  And then would 
>> we need to consider slicing all iterables instead of just only CharFields, 
>> TextFields, and string expressions?
>>
>>  
>>
>> Also, most of the functions in ticket 28643 that go across all supported 
>> Django database backends are complete, with the exception of ROUND.  SQLite 
>> does not come with many built-in math functions, though there are 
>> extensions available.  I was wondering how we should proceed with that 
>> ticket when ROUND is completed.
>>
>>  
>>
>> Thank you!
>>
>>  
>>
>> https://github.com/django/django/pull/9583
>>
>> https://code.djangoproject.com/ticket/28643
>>
>>  
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/af172482-ffb2-4b0c-ba2f-5f16e876302c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Security middleware for django for insecure (http) connections

[Josh Smeaton]

HTTPS is enough. Despite that, how would you handle the **client** doing 
decryption and encryption? I don't think this is an idea you should pursue, 
especially if your expertise is not in security.

On Tuesday, 16 January 2018 10:03:00 UTC+11, Vishwas Mittal wrote:
>
> Hello everyone,
>
> I am a new contributor to django, and would like to propose a new 
> middleware to django that can provide some degree of security for HTTP 
> connections by encrypting the data to be sent in response.
>
> This middleware will work on the principles of HTTPS but the main 
> difference is there will no certification authority involved.
>
> This will include the middleware inplementation and also a user side code/ 
> implementaion which can be used to complete the encryption-decryption 
> couple.
>
> Please present your thoughts and make necessary suggestions regarding this.
>
>
> Regards
> Vishwas
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/184bbb3a-b7b8-4ef0-bc02-b5e888ffa549%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Fellow Report — January 12, 2018

[Josh Smeaton]

Welcome, and nice work! I've seen some of the reviews you've made and 
you're straight into the deep end.

On Saturday, 13 January 2018 04:01:06 UTC+11, Carlton Gibson wrote:
>
> Hi all, 
>
> With help from Tim, Aymeric and Frank, I got started on the fellowship 
> this week. 
> (Thank you all!)
>
>
> I've Reviewed Patches on: 
>
> https://code.djangoproject.com/ticket/24747 -- Allowed transforms in 
> QuerySet.order_by()
> https://code.djangoproject.com/ticket/29002 -- Corrected cached template 
> loader docs about when it's automatically enabled
>
> https://code.djangoproject.com/ticket/28977 -- Change Local Memory Cache 
> to Use LRU
> https://code.djangoproject.com/ticket/29004 -- Add optional database view 
> support to inspectdb
> https://code.djangoproject.com/ticket/28761 -- Document prefixes in 
> InlineFormsets
> https://code.djangoproject.com/ticket/28885 -- Bottom parts of 
> "Congratulations!" page cut off internationalized content
>
>
> I'm still finding my feet (exactly how things are done etc) so all input 
> is welcome and appreciated (!!!)
>
> Kind Regards,
>
> Carlton
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/da0e883c-80ff-4626-b06a-092bc001b187%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Should ModelField.choices accept a callable?

[Josh Smeaton]

Migrations are mainly for tracking database changes. Model attributes not 
SQL related are pushed into the migrations also, presumably for the 
purposes of data migrations (maybe post-hoc justification, only a few 
people would actually know). Providing a callable still provides access to 
the data as defined as migration time. But for dynamic content like 
choices, the set of production values can and will differ from the set of 
dev values. 

Perhaps there should be a more general mechanism for opting specific kwargs 
out of migrations - I don't know.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/382314ae-b3e4-4ce6-be34-135060978317%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Should ModelField.choices accept a callable?

[Josh Smeaton]

https://code.djangoproject.com/ticket/22837 was closed as invalid. It 
concerns itself with migrations being detected when model field choices 
(which may be dynamic) change. The workaround suggested is to pass a 
callable to choices with a link 
to https://code.djangoproject.com/ticket/13181. However, that ticket only 
concerns itself with FormField.choices accepting a callable.

There are many times where dynamic choices causes unnecessary migrations to 
be detected, sometimes in external apps:

- pytz gains new timezone codes
- country codes are added or removed
- new states are added to an internal library
- settings determine the set of choices

I would like to reconsider reopening 
https://code.djangoproject.com/ticket/22837 with the goal of making 
ModelField.choices accept a callable. There is sufficient frustration 
within the community to warrant such a change I feel.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/27d18006-0a5d-4441-92ce-205a42c3bef2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Change Local Memory Cache to Use LRU #28977

[Josh Smeaton]

Yup, good analysis, good numbers, thanks for running those. I'm happy with 
those results and think we should proceed.

On Thursday, 11 January 2018 05:11:56 UTC+11, Adam Johnson wrote:
>
> Grant, you're a star. I think the tradeoff is acceptable too.
>
> On 10 January 2018 at 17:05, Grant Jenks <grant...@gmail.com 
> > wrote:
>
>> I was able to run the more extensive benchmarks under no-contention and 
>> high-contention scenarios with measurements at the 50th, 90th, 99th, and 
>> 100th percentiles. I updated the ticket at 
>> https://code.djangoproject.com/ticket/28977 with the results.
>>
>> Under high-contention scenarios, the RWLock did serve a purpose, albeit a 
>> rather limited one. There's a small tradeoff to be had but I think it's 
>> easy to accept. Note that the benchmark does not include a miss-rate 
>> penalty which the LRU-eviction policy will almost certainly improve 
>> compared with the current random-eviction policy.
>>
>> Grant
>>
>>
>> On Mon, Jan 8, 2018 at 2:25 PM, Josh Smeaton <josh.s...@gmail.com 
>> > wrote:
>>
>>> Nice to meet you too. Your benchmarking code was extremely handy when I 
>>> was profiling lru-cache-backend, so thank you!
>>>
>>> Are you able to run the same benchmarks using this version of the cache 
>>> to see how it performs in low/medium/high eviction scenarios? I think those 
>>> benchmarks will be nicer than the simpler cache.get() ones you have on the 
>>> ticket.
>>>
>>> I'll get to reviewing that change some time this week regardless.
>>>
>>>
>>>
>>>
>>> On Tuesday, 9 January 2018 08:58:55 UTC+11, Grant Jenks wrote:
>>>>
>>>> Josh, it's nice to meet you here. I cited your django-lrucache-backend 
>>>> project in the original post of the Trac ticket. I'm also the author of 
>>>> DiskCache http://www.grantjenks.com/docs/diskcache/ which your project 
>>>> refs for benchmarks :) I added some benchmark data to the Trac ticket 
>>>> which 
>>>> may interest you.
>>>>
>>>> Thank you, Josh and Adam, for the +1's. I've summarized the feedback 
>>>> and updated the ticket at https://code.djangoproject.com/ticket/28977 
>>>> It's now marked as Accepted and Has patch. I've created a pull request at 
>>>> https://github.com/django/django/pull/9555 Please review when able.
>>>>
>>>> Grant
>>>>
>>>> Ps. As always the docs are excellent. Even the docs for contributing 
>>>> are extremely helpful!
>>>>
>>>>
>>>> On Friday, January 5, 2018 at 1:46:38 PM UTC-8, Adam Johnson wrote:
>>>>>
>>>>> I'm +1 for moving to LRU too, the eviction algorithm has always looked 
>>>>> weird to me. And Josh's library shows there are valid uses of local 
>>>>> memory 
>>>>> caching in applications - perhaps moreso these days than when Django 
>>>>> added 
>>>>> caching and memcached was the latest thing.
>>>>>  
>>>>>
>>>>>> You can also get a very nice bump in throughput if you eliminate the 
>>>>>> `validate_key` check
>>>>>
>>>>>
>>>>> +1 to adding an option to disable the check as well. If you're using a 
>>>>> LocMemCache in production, you probably don't care about compatibility 
>>>>> with 
>>>>> memcached, because you'll be using it for different types of data.
>>>>>
>>>>> On 5 January 2018 at 02:53, Josh Smeaton <josh.s...@gmail.com> wrote:
>>>>>
>>>>>> To lend some weight to this, I've implemented an LRU loc mem cache 
>>>>>> and have done some benchmarking. There are some graphs in the readme: 
>>>>>> https://github.com/kogan/django-lrucache-backend - which I've 
>>>>>> written a little about 
>>>>>> https://devblog.kogan.com/blog/a-smarter-local-memory-django-cache-backend
>>>>>>  
>>>>>> (I don't think my particular implementation is useful to core, as it 
>>>>>> relies 
>>>>>> on a 3rd party C lib, but using OrderedDict is cool!).
>>>>>>
>>>>>> You can also get a very nice bump in throughput if you eliminate the 
>>>>>> `validate_key` check, which does a character by character check of the 
>>>>>> key 
>>>>>> to avoid issues wi

Re: Change Local Memory Cache to Use LRU #28977

[Josh Smeaton]

Nice to meet you too. Your benchmarking code was extremely handy when I was 
profiling lru-cache-backend, so thank you!

Are you able to run the same benchmarks using this version of the cache to 
see how it performs in low/medium/high eviction scenarios? I think those 
benchmarks will be nicer than the simpler cache.get() ones you have on the 
ticket.

I'll get to reviewing that change some time this week regardless.



On Tuesday, 9 January 2018 08:58:55 UTC+11, Grant Jenks wrote:
>
> Josh, it's nice to meet you here. I cited your django-lrucache-backend 
> project in the original post of the Trac ticket. I'm also the author of 
> DiskCache http://www.grantjenks.com/docs/diskcache/ which your project 
> refs for benchmarks :) I added some benchmark data to the Trac ticket which 
> may interest you.
>
> Thank you, Josh and Adam, for the +1's. I've summarized the feedback and 
> updated the ticket at https://code.djangoproject.com/ticket/28977 It's 
> now marked as Accepted and Has patch. I've created a pull request at 
> https://github.com/django/django/pull/9555 Please review when able.
>
> Grant
>
> Ps. As always the docs are excellent. Even the docs for contributing are 
> extremely helpful!
>
>
> On Friday, January 5, 2018 at 1:46:38 PM UTC-8, Adam Johnson wrote:
>>
>> I'm +1 for moving to LRU too, the eviction algorithm has always looked 
>> weird to me. And Josh's library shows there are valid uses of local memory 
>> caching in applications - perhaps moreso these days than when Django added 
>> caching and memcached was the latest thing.
>>  
>>
>>> You can also get a very nice bump in throughput if you eliminate the 
>>> `validate_key` check
>>
>>
>> +1 to adding an option to disable the check as well. If you're using a 
>> LocMemCache in production, you probably don't care about compatibility with 
>> memcached, because you'll be using it for different types of data.
>>
>> On 5 January 2018 at 02:53, Josh Smeaton <josh.s...@gmail.com> wrote:
>>
>>> To lend some weight to this, I've implemented an LRU loc mem cache and 
>>> have done some benchmarking. There are some graphs in the readme: 
>>> https://github.com/kogan/django-lrucache-backend - which I've written a 
>>> little about 
>>> https://devblog.kogan.com/blog/a-smarter-local-memory-django-cache-backend 
>>> (I don't think my particular implementation is useful to core, as it relies 
>>> on a 3rd party C lib, but using OrderedDict is cool!).
>>>
>>> You can also get a very nice bump in throughput if you eliminate the 
>>> `validate_key` check, which does a character by character check of the key 
>>> to avoid issues with particular characters in memcache.
>>>
>>> We don't want to be promoting locmemcache too much, but that said, if we 
>>> can provide a better default then we should in my opinion.
>>>
>>> On Friday, 5 January 2018 10:12:39 UTC+11, Grant Jenks wrote:
>>>>
>>>> Hi all--
>>>>
>>>> Long time user, first time poster, here. Thank you all for Django!
>>>>
>>>> The current local memory cache (locmem) in Django uses a pseudo-random 
>>>> culling strategy. Rather than random, the OrderedDict data type can be 
>>>> used 
>>>> to implement an LRU eviction policy. A prototype implementation is already 
>>>> used by functools.lru_cache and Python 3 now supports 
>>>> OrderedDict.move_to_end and OrderedDict.popitem to ease the implementation.
>>>>
>>>> I have created an example set of changes at 
>>>> https://github.com/grantjenks/django/tree/ticket_28977 in commit 
>>>> https://github.com/grantjenks/django/commit/b06574f6713d4b7d367d7a11e0268fb62f5fd1d1
>>>>
>>>> Is there a consensus as to the value of these changes?
>>>>
>>>> Sincerely,
>>>> Grant Jenks
>>>>
>>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com.
>>> To post to this group, send email to django-d...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/fdb91901-c378-4258-9201-d24a9f5f103e%40googlegroups.com
>>>  
>>> <https://groups.google.

Re: Change Local Memory Cache to Use LRU #28977

[Josh Smeaton]

To lend some weight to this, I've implemented an LRU loc mem cache and have 
done some benchmarking. There are some graphs in the 
readme: https://github.com/kogan/django-lrucache-backend - which I've 
written a little about 
https://devblog.kogan.com/blog/a-smarter-local-memory-django-cache-backend 
(I don't think my particular implementation is useful to core, as it relies 
on a 3rd party C lib, but using OrderedDict is cool!).

You can also get a very nice bump in throughput if you eliminate the 
`validate_key` check, which does a character by character check of the key 
to avoid issues with particular characters in memcache.

We don't want to be promoting locmemcache too much, but that said, if we 
can provide a better default then we should in my opinion.

On Friday, 5 January 2018 10:12:39 UTC+11, Grant Jenks wrote:
>
> Hi all--
>
> Long time user, first time poster, here. Thank you all for Django!
>
> The current local memory cache (locmem) in Django uses a pseudo-random 
> culling strategy. Rather than random, the OrderedDict data type can be used 
> to implement an LRU eviction policy. A prototype implementation is already 
> used by functools.lru_cache and Python 3 now supports 
> OrderedDict.move_to_end and OrderedDict.popitem to ease the implementation.
>
> I have created an example set of changes at 
> https://github.com/grantjenks/django/tree/ticket_28977 in commit 
> https://github.com/grantjenks/django/commit/b06574f6713d4b7d367d7a11e0268fb62f5fd1d1
>
> Is there a consensus as to the value of these changes?
>
> Sincerely,
> Grant Jenks
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fdb91901-c378-4258-9201-d24a9f5f103e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Sealing or locking QuerySets -- a suggestion to prevent surprise N+1 queries.

[Josh Smeaton]

I wasn't aware of this new feature Shai, thanks for pointing it out!

For this particular case I'd prefer locking to be bound to a particular 
queryset rather than the database as a whole. I would also expect it to 
fail loudly when accessing a non-fetched related object (book.author), 
which can be a common cause of pain.

I'm also still very interested in auto-prefetch Adam, is there any help I 
can lend?

On Thursday, 4 January 2018 17:32:19 UTC+11, Shai Berger wrote:
>
> Hi all, 
>
> Django 2.0 has a new feature[1] which allows you to say "I expect no 
> actual 
> database queries in this piece of code". It is not designed to stop a 
> specific 
> queryset from spawning requests, so getting it to do exactly what's asked 
> for 
> in this thread may be a little involved, but if your goal is to prevent 
> surprise queries, I think it is actually better than sealing a single 
> queryset. 
>
> HTH, 
> Shai 
>
> [1] https://docs.djangoproject.com/en/2.0/topics/db/instrumentation/ 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7441a83e-e459-4612-8dd7-cff48f5af7f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: GLOBAL_PERMS

[Josh Smeaton]

>
>
> - I dislike the seetings approach of GLOBAL_PERMS and would rather see 
>   users writing explicit data migrations. 
>

I don't favour either setting or migration based perms, but if you're going 
to go with migration based, then please also consider adding a first class 
permission type, to avoid the ceremony of creating forward/back functions, 
and having to remember the model layout for Permissions.

Something like:

migrations.CreatePermission(app_label, permission_name)

or django.contrib.auth.CreatePermission (which is actually an Operation) if 
we want to avoid contrib polluting migrations proper. 

I haven't fully thought through the implications, but every time I need to 
write a data migration I spend more time than I'd like to admit searching 
for a previous data migration from which to copy.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/32bb32b4-bf1a-494e-8588-df70f8abfef7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving Backward Compatibility

[Josh Smeaton]

> We'd likely deprecate fewer features if we had to reach consensus on
django-developers every time

I’m not sure if you consider this a good thing or a bad thing Aymeric :)

I haven’t fully thought through a process but I’m not even sure a full
consensus would be required. But it’d help to think through options that
may not have been considered on the ticket which is a smaller audience.

Perhaps a “we’re proposing deprecating this way, unless others have a
better idea how we can proceed” would be enough?
On Thu, 7 Dec 2017 at 18:12, Aymeric Augustin <
aymeric.augus...@polytechnique.org> wrote:

> Hello,
>
> The duration of the deprecation period doesn't make a big difference
> because few people update their code until they have no other choice.
> Keeping deprecated code around longer means it's harder to track down the
> release where the corresponding changes were introduced and figure out what
> changes are needed. This effect already occurs in Django 2.0 where some
> removals are related to 1.9 and others to 1.10.
>
> Requiring a thread of django-developers for deprecations sounds
> interesting. 20 to 40 features were deprecated in the recent releases.
> That's one feature per week or fortnight. We'd likely deprecate fewer
> features if we had to reach consensus on django-developers every time.
>
> Best regards,
>
> --
> Aymeric.
>
>
>
> On 7 Dec 2017, at 04:09, Josh Smeaton <josh.smea...@gmail.com> wrote:
>
> Following on from Collin, another ticket that generated a large amount of
> code churn for little perceived benefit was
> https://code.djangoproject.com/ticket/21127 (enforce on_delete). I know
> there are many in the community that rail against any and all changes (I
> had an argument with such a person on the 2.0 reddit thread recently), but
> we should endeavour to make churn as painless and infrequent as possible.
> As a concrete example with the on_delete ticket, we could have used
> settings to control the default on_delete behaviour per nullable and
> non-nullable foreign key. I know we don't like adding extra settings, and
> perhaps another path forward could have sufficed, but requiring everyone to
> update every foreign key definition is quite hostile.
>
> In that light, I think a thread per deprecation on django-dev is a great
> idea, which will both help to keep us honest, and give those in the
> community that strongly dislike backward incompatible changes a chance to
> propose alternatives.
>
> I'm not so keen on the idea of keeping deprecations going for as long as
> Collin suggests. There should be little harm in 3rd party libraries
> declaring **this is the final version to support 1.8**, and then bumping
> the major version to support the next major Django series. The new LTS to
> LTS policy that Django has taken up is new - but should give 3rd party
> libraries a bit of certainty on how to support Django moving forward.
>
> The only concern I have is 3rd party libraries might begin to stop
> supporting versions of X.0 and X.1, and instead just support the LTS (X.2).
> That'll mean fewer users to run and test newer versions in production, if
> 3rd party ecosystem isn't keeping current. I don't know if that'll be an
> actual problem in practise though. I don't think it will be.
>
> For renames and moves, have we considered leaving around aliases
> indefinitely? How big is the overhead when the modules themselves are kept?
> We could generate an `ObsoleteWarning` + undocument to discourage usage. I
> don't wish to see or maintain large or complicated deprecations. At the
> same time, dropping renames seems like overkill. Could we maintain a
> catalog of renames in a single spot, then have the startup machinary patch
> old names? At least we'd avoid polluting the "real code" with shims.
>
>
> On Thursday, 7 December 2017 09:08:09 UTC+11, Tim Graham wrote:
>
> I don't see a strong reason to make it easy to support 1.8 and 2.0 at the
>> same time. Support for Django 1.8 ends 4 months after the release of 2.0.
>> As the 2.0 release notes say, "Following the release of Django 2.0, we
>> suggest that third-party app authors drop support for all versions of
>> Django prior to 1.11. At that time, you should be able to run your
>> package’s tests using python -Wdso that deprecation warnings do appear.
>> After making the deprecation warning fixes, your app should be compatible
>> with Django 2.0."
>>
>
>>
>> Third-party packages aren't required to support a certain version of
>> Django until it's completely unsupported. In fact, a package dropping
>> support for a version of Django that's nearing it's end-of-life is a good
>> reminder for users to upgrade their Django.
>>
&g

Re: Improving Backward Compatibility

[Josh Smeaton]

Following on from Collin, another ticket that generated a large amount of 
code churn for little perceived benefit 
was https://code.djangoproject.com/ticket/21127 (enforce on_delete). I know 
there are many in the community that rail against any and all changes (I 
had an argument with such a person on the 2.0 reddit thread recently), but 
we should endeavour to make churn as painless and infrequent as possible. 
As a concrete example with the on_delete ticket, we could have used 
settings to control the default on_delete behaviour per nullable and 
non-nullable foreign key. I know we don't like adding extra settings, and 
perhaps another path forward could have sufficed, but requiring everyone to 
update every foreign key definition is quite hostile.

In that light, I think a thread per deprecation on django-dev is a great 
idea, which will both help to keep us honest, and give those in the 
community that strongly dislike backward incompatible changes a chance to 
propose alternatives.

I'm not so keen on the idea of keeping deprecations going for as long as 
Collin suggests. There should be little harm in 3rd party libraries 
declaring **this is the final version to support 1.8**, and then bumping 
the major version to support the next major Django series. The new LTS to 
LTS policy that Django has taken up is new - but should give 3rd party 
libraries a bit of certainty on how to support Django moving forward.

The only concern I have is 3rd party libraries might begin to stop 
supporting versions of X.0 and X.1, and instead just support the LTS (X.2). 
That'll mean fewer users to run and test newer versions in production, if 
3rd party ecosystem isn't keeping current. I don't know if that'll be an 
actual problem in practise though. I don't think it will be.

For renames and moves, have we considered leaving around aliases 
indefinitely? How big is the overhead when the modules themselves are kept? 
We could generate an `ObsoleteWarning` + undocument to discourage usage. I 
don't wish to see or maintain large or complicated deprecations. At the 
same time, dropping renames seems like overkill. Could we maintain a 
catalog of renames in a single spot, then have the startup machinary patch 
old names? At least we'd avoid polluting the "real code" with shims.

On Thursday, 7 December 2017 09:08:09 UTC+11, Tim Graham wrote:
>
> I don't see a strong reason to make it easy to support 1.8 and 2.0 at the 
> same time. Support for Django 1.8 ends 4 months after the release of 2.0. 
> As the 2.0 release notes say, "Following the release of Django 2.0, we 
> suggest that third-party app authors drop support for all versions of 
> Django prior to 1.11. At that time, you should be able to run your 
> package’s tests using python -Wd so that deprecation warnings do appear. 
> After making the deprecation warning fixes, your app should be compatible 
> with Django 2.0."
>
> Third-party packages aren't required to support a certain version of 
> Django until it's completely unsupported. In fact, a package dropping 
> support for a version of Django that's nearing it's end-of-life is a good 
> reminder for users to upgrade their Django.
>
> In the past few weeks, I sent a few pull requests to various third-party 
> packages following the "drop support for versions older than 1.11" 
> guideline and I thought this process was very clean. I'm not enthusiastic 
> about the idea of making the deprecation process even more lengthy. I think 
> it will mostly promote the continued use of deprecation features resulting 
> in messier code and encourage prolonged use of unsupported versions of 
> Django.
>
> Personally, I think asking developers to do a bit of house cleaning every 
> three years if you want to upgrade from LTS to then next LTS is reasonable. 
> What's your experience been like?
>
> On Wednesday, December 6, 2017 at 4:12:00 PM UTC-5, Collin Anderson wrote:
>>
>> Hi All,
>>
>> I think it would help if Django was better at *deprecations and backward 
>> compatibility*. I’ve brought this up [before 
>> ],
>>  
>> but didn’t get any feedback, so here’s another try at it, with some 
>> *specific 
>> ideas of how to improve things below*.
>>
>> *Background*:
>>
>> The [API docs ] 
>> say “*If, for some reason, an API declared stable must be removed or 
>> replaced*, it will be declared deprecated but will remain in the API for 
>> at least two feature releases.” - I’ve always thought of the “must be 
>> removed” as “there’s really no alternative”. I think that if it's not 
>> broken, Django should avoid breaking it.
>>
>> I maintain lots of Django projects, and upgrading is not a small task. 
>> There are enough changes to undocumented APIs already happening that it 
>> would be great if the documented ones didn't change as much.
>>
>> I also think it’s a little disappointing that 

Re: Additional PostgreSQL-specific functions

[Josh Smeaton]

Hey Joey,

I think you can create a new ticket for this function. If there is a way to 
approximate or implement this function in all 4 supported backends then 
that would be preferred. If it can only really be implemented in postgres, 
then document in the ticket why, and you can add the new function to 
contrib.postgres. Hope that helps?

On Friday, 1 December 2017 05:22:32 UTC+11, Joey Wilhelm wrote:
>
> Ping!
>
> Is there any chance of getting some additional guidance on these? I'd love 
> to get these contributed, if they are welcome!
>
> -Joey Wilhelm
>
> On Fri, Nov 17, 2017 at 1:22 PM, Joey Wilhelm  > wrote:
>
>> I did come upon that ticket, but I wasn't sure this necessarily belonged 
>> as part of it, given that these are not methods which are widely available 
>> across different engines. That said, I would gladly add them there. I'm 
>> just eager to spin up a PR or two. :-)
>>
>> On Fri, Nov 17, 2017 at 1:12 PM, Matthew Pava > > wrote:
>>
>>> I wonder if we should put that in this ticket:
>>>
>>> https://code.djangoproject.com/ticket/28643
>>>
>>>  
>>>
>>>  
>>>
>>> *From:* django-d...@googlegroups.com  [mailto:
>>> django-d...@googlegroups.com ] *On Behalf Of *Joey Wilhelm
>>> *Sent:* Friday, November 17, 2017 2:02 PM
>>> *To:* django-d...@googlegroups.com 
>>> *Subject:* Additional PostgreSQL-specific functions
>>>
>>>  
>>>
>>> Greetings,
>>>
>>>  
>>>
>>> Yesterday I opened a ticket[1] for a "RegexpReplace" method for 
>>> PostgreSQL. After some talk and encouragement over in IRC, I have 
>>> additionally created a "SubstrFrom" function for PostgreSQL, which allows 
>>> you to use a regex match in order to extract your substring.
>>>
>>>  
>>>
>>> So at this point, I have a couple of questions.
>>>
>>>  
>>>
>>> 1) Are these things which would likely get integrated into core?
>>>
>>> 2) If so, should I put them into separate tickets or would I be able to 
>>> combine them into a single?
>>>
>>> 3) Should they be added to the PostgreSQL specific library? Oracle does 
>>> have both of these pieces of functionality, but I believe it is the only 
>>> popular engine aside from PostgreSQL which does.
>>>
>>>  
>>>
>>> I have the code for both of these implemented in my local project, along 
>>> with tests, so I would love to contribute them if possible.
>>>
>>>  
>>>
>>> Thank you!
>>>
>>>  
>>>
>>>  
>>>
>>> [1]: https://code.djangoproject.com/ticket/28805
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com .
>>> To post to this group, send email to django-d...@googlegroups.com 
>>> .
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/CADBkHd%2Be3GodbKvDFXq0dAgwzs4n%3DFgLGHTYsbhuiQWDOcB64Q%40mail.gmail.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Django developers (Contributions to Django itself)" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to django-develop...@googlegroups.com .
>>> To post to this group, send email to django-d...@googlegroups.com 
>>> .
>>> Visit this group at https://groups.google.com/group/django-developers.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/django-developers/fa1db399bca544eebeeb0b261752ffeb%40ISS1.ISS.LOCAL
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6560985c-9d33-4837-9af5-ee467ae5369b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Feature request: get_first_or_404

[Josh Smeaton]

>From my experiences, all projects I've worked on have implemented 
get_first_or_none and get_first_or_404, and usually slightly differently. I 
think these shortcuts would make sense for Django. Interested to hear how 
common the usage would be from others on the list though.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fcabccc7-3da2-456f-ad92-90145b700635%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: models.CalculatedField feature

[Josh Smeaton]

Introducing an `as_python` to the expression API would probably work. It'd 
involve a lot of work to implement for all supportable expression types. 
Considering there are many 3rd party / user code expressions in the wild 
without as_python, we'd need to find a way to fail gracefully.

However, for our core types, we can mimic SQL behaviour for nulls/empty 
strings, and other weird backend behaviour.

It might be interesting to see a proof of concept implementing support for 
some of the more complex expressions like Case/When and datetime truncation 
with timezones. Then there are Exists() subqueries and Window functions 
that can't work without consulting the database, and that should probably 
not be implemented to fetch row by row on demand (like foreign keys without 
select related).

If some kind of rough consensus can be reached on an implementation I think 
we'd probably want to see a DEP considering the tradeoffs and different 
opinions raised in this thread.

On Sunday, 19 November 2017 22:14:46 UTC+11, ilya.ka...@jetbrains.com wrote:
>
> Ok, I see not everyone are happy with idea of ORM expression calculated on 
> Python side:)
> We can:
>
> * Provide two different backends (Database-based and Python-based or  "run 
> python in database" backend) for calculated field
> * Or provide python-based backend as third party lib
> * Or provide whole solution as third party lib (if Django API is flexible 
> enough for that)
>
> I still do not like database approach.
> * If I have list of objects, do you want to have O(n) queries?
> * If calculated field is function of only model fields (no foreign keys 
> etc) it should work locally: some projects may have no database at all.
>
>
>
> >>Even simple comparisons can be difficult to simulate in Python. As an 
> example, some databases treat empty strings as null
> But ORM should handle it somehow, should not it?
> I want to belive that "filter(alias__isnull=True)" works identically 
> across all databases.
> So, we need to do same on Python side.
>
> >>To be honest, I'm curious why the SQLAlchemy approach isn't being 
> discussed more here.
> I think SQLAlchemy approach is good, but we should use Django ORM 
> expressions language.
> See: we need some high-level expression language to describe functions of 
> models that could be 
> calculated both on SQL side and Python side.
> And we already have it: it is Django ORM expressions language:)
> We have some kind of AST for all those Q, F expressions etc, so we only 
> need to create evaluator for this language. 
>
> No need to use external language, based on Python, as SQLAlchemy does.
>
> Why do we need
> from sqlalchemy import func
> func.concat(cls.first_name, ' ', cls.last_name)
>
> if we already have
> F('first_name') + F('last_name')
> ?
>
>
> >>and supplying both the SQL side (via a method which returns a Q object, 
> most likely) and the Python side 
> But it ruin DRY idea by making author to write same logic twice (for 
> Python and SQL), is not it?
>
> Still, user may have ability to provide custom expression, 
> but for simple cases like "age__gt=18" we can do it "automagically".
>
> Django generates SQL code from its expressions. Sometimes this code is 
> sophisticated.
> Why can't we do that for Python, if we do it for SQL?
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/730ceeaa-0d14-456f-a52d-8981c015093f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Using server side cursors for values and values_list

[Josh Smeaton]

I can't think of any reasons why not to do this off the top of my head. 
Have you looked into the code? Is there any technical reason you can see 
that might prevent chunked fetching from working there?

On Saturday, 18 November 2017 03:27:13 UTC+11, Dražen Odobašić wrote:
>
> Hi, 
>
> at the moment `chunked_fetch` (server side cursors) are only initialized 
> for 
> ModelIterable class. Is there any reason why we should not do the same for 
> other 
> BaseIterable classes (ValuesIterable, ValuesListIterable and 
> FlatValuesListIterable). 
>
> The idea is that you could then use QuerySet.values_list('id').iterator() 
> and 
> actually use server side cursors. 
>
> Dražen 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1e6a2114-cf33-4059-8a83-ec42a24f3256%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [off topic] Re: models.CalculatedField feature

[Josh Smeaton]

Re-reading your reply, I think I've misunderstood your intent there (I 
didn't realise who the author was). Please disregard my previous reply.

On Sunday, 19 November 2017 09:59:44 UTC+11, Josh Smeaton wrote:
>
> Until you want to aggregate or filter on that calculation, and perform 
> that filtering in the database rather than pulling back millions of records 
> to do so in python. The database *must* be involved.
>
> On Sunday, 19 November 2017 06:32:48 UTC+11, ilya.ka...@jetbrains.com 
> wrote:
>>
>> Such computations always end up slightly different in some 
>>> edge cases 
>>>
>> I agree.
>> We can document this fact. Model authors should be aware that they are 
>> not allowed to use anything but literals. All other functions are used on 
>> their own risc. I believe that comparition of two integers or strings 
>> should work stable across all modern platforms.
>>
>> I still believe database shall not be involved in this process.I do not 
>> need database to compare two digits. 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/33fd5466-f790-4bdd-941e-80e81701e2af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.