Re: Making max_length argument optional

[Aymeric Augustin]

Hello,

This thread is getting long. It contains lots of valid arguments. In the 
interest of seeking consensus, here’s a tentative summary.


Problem

Typing `foo = models.CharField(max_length=100)` gets old quickly. Can we skip 
the `max_length` argument in many cases?


Constraints

Character limits and their consequences on performance vary widely across 
databases, which makes it impossible for Django to provide a “large enough and 
fast enough for everyone” default.

The lowest limit mentioned in the thread is 191 on MySQL. No one talked about 
third-party backends; we can’t be sure that this value  is compatible with 
unique indexing on all current backends.

Making the default limit backend-dependent sounds like a bad idea, especially 
for pluggable applications. The implementation also looks complicated compared 
to the expected benefits.

PostgreSQL is a bit of an outlier. As far as I understand, there’s no 
performance difference VARCHAR(N) and VARCHAR(). The former just validates the 
length of the data.

The argument for not providing a default max_length is to have developers think 
about their data. Unfortunately, in practice, most just write 
`models.CharField(max_length=255)` and move one.

(FWIW this limit is either a cargo-cult from the 90s 
 or a micro-optimization for MySQL 
.)


Consensus (?)

In order to resolve this discussion, I think we must split it in two questions.


1) Should Django provide a reasonable default for CharField.max_length?

It seems to me that the resistance to this idea weakened as the discussion 
unfolded. Perhaps it’s a losing fight in the long run anyway…

A default value could be justified as the “reasonable length for a line of 
text”. Considering that the optimal line length 
 is around 66 characters, 
100 could be a good limit. 120 would work as well.

It seems more future-proof and less debatable not to tie this limit to a 
technical limitation of the current version of a particular database engine.

On one hand, the default value matters because it will be widely used. On the 
other hand it doesn’t matter because developers who care about the length of 
their CharFields should set them explicitly.


2) How can we make it easy for PostgreSQL users to just use VARCHAR()?

Since this is a PostgreSQL-specific feature, having a variant of CharField in 
django.contrib.postgres that supports and perhaps even defaults to unlimited 
length shouldn’t be controversial.


I hope this helps!

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/A2F207B6-42EF-4E7E-80B8-6FE4C6FD8014%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Translate permission Django

[Marcos Serrano]

Hello,

How to translate the default permissions django . Keywords ( Add, Delete ... )

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/cc4a2daf-103f-476f-a507-50ba104e6510%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Add support for relative imports in django.conf.urls.include()?

[Tim Graham]

For example, within myproject.urls:


urlpatterns = (
url(r'', include('.myapp.urls')),
)


.. becomes equivalent to::

urlpatterns = (
url(r'', include('myproject.myapp.urls')),
)
 

Whilst a contrived example, this can make heavy-nested apps look far, far 
cleaner. For example, within myproject.foo.foo_bar.foo_bar_baz.urls, one 
only needs to include .foo_bar_baz_qux.urls to get to the "next" level, 
rather than the significantly more unwieldy 
myproject.foo.foo_bar.foo_bar_baz.foo.bar_baz_qux.urls.



What do you think of this proposal from 
https://code.djangoproject.com/ticket/26288? I don't know if "nested apps" 
are very common and/or something that should be promoted with a change like 
this.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5a92150c-4969-4432-b47b-0a02ce889312%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add support for relative imports in django.conf.urls.include()?

[Aymeric Augustin]

I’m +0 on this change, for consistency with Python’s import statement.

-- 
Aymeric.

> On 29 Feb 2016, at 14:17, Tim Graham  wrote:
> 
> 
> For example, within myproject.urls:
> 
> urlpatterns = (
> url(r'', include('.myapp.urls')),
> )
> 
> .. becomes equivalent to::
> urlpatterns = (
> url(r'', include('myproject.myapp.urls')),
> )
>  
> Whilst a contrived example, this can make heavy-nested apps look far, far 
> cleaner. For example, within myproject.foo.foo_bar.foo_bar_baz.urls, one only 
> needs to include .foo_bar_baz_qux.urls to get to the "next" level, rather 
> than the significantly more unwieldy 
> myproject.foo.foo_bar.foo_bar_baz.foo.bar_baz_qux.urls.
> 
> 
> 
> What do you think of this proposal from 
> https://code.djangoproject.com/ticket/26288 
> ? I don't know if "nested apps" 
> are very common and/or something that should be promoted with a change like 
> this.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-developers+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to django-developers@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-developers 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/5a92150c-4969-4432-b47b-0a02ce889312%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/79F34550-4CC6-4E2A-8AF7-584D9FC796E5%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.

Re: Add support for relative imports in django.conf.urls.include()?

[Marten Kenbeek]

The current approach can lead to surprising errors if include() is called 
by a helper function, rather than directly called in your urlconf. The 
relative import would be resolved in relation to the module where the 
helper function is defined. I'm not sure how much of an actual problem that 
is in practice, but it's something to keep in mind. 

On Monday, February 29, 2016 at 2:20:23 PM UTC+1, Aymeric Augustin wrote:
>
> I’m +0 on this change, for consistency with Python’s import statement.
>
> -- 
> Aymeric.
>
> On 29 Feb 2016, at 14:17, Tim Graham  
> wrote:
>
>
> For example, within myproject.urls:
>
>
> urlpatterns = (
> url(r'', include('.myapp.urls')),
> )
>
>
> .. becomes equivalent to::
>
> urlpatterns = (
> url(r'', include('myproject.myapp.urls')),
> )
>  
>
> Whilst a contrived example, this can make heavy-nested apps look far, far 
> cleaner. For example, within myproject.foo.foo_bar.foo_bar_baz.urls, one 
> only needs to include .foo_bar_baz_qux.urls to get to the "next" level, 
> rather than the significantly more unwieldy 
> myproject.foo.foo_bar.foo_bar_baz.foo.bar_baz_qux.urls.
>
> 
>
> What do you think of this proposal from 
> https://code.djangoproject.com/ticket/26288? I don't know if "nested 
> apps" are very common and/or something that should be promoted with a 
> change like this.
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to django-develop...@googlegroups.com .
> To post to this group, send email to django-d...@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/django-developers/5a92150c-4969-4432-b47b-0a02ce889312%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/187d6ad9-289d-446a-9932-a2c57ef3912c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Translate permission Django

[Marcos Serrano]

El lunes, 29 de febrero de 2016, 10:20:36 (UTC+1), Marcos Serrano escribió:
>
> Hello,
>
> How to translate the default permissions django . Keywords ( Add, Delete ... )
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/04078718-fc4b-4655-b249-c2234cbc2fae%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Making max_length argument optional

[Shai Berger]

Hi,

Thank you, Aymeric, for summing up the discussion this way. The division into 
two separate problems is indeed required, and I fully support the idea of 
setting max_length's default to 100 or 120.

There seem to be just two points worth adding to your summary:

On Monday 29 February 2016 11:19:02 Aymeric Augustin wrote:
> 
> 2) How can we make it easy for PostgreSQL users to just use VARCHAR()?
> 
> Since this is a PostgreSQL-specific feature, having a variant of CharField
> in django.contrib.postgres that supports and perhaps even defaults to
> unlimited length shouldn’t be controversial.
> 

The first -- I believe it was raised very early on by Christophe Pettus -- is 
that Django already has a field that manifests on PG as VARCHAR(), and that is 
TextField. However, I don't like the idea that PG users should be using 
TextField(widget=TextInput) as a replacement for CharField; I find that 
counter-intuitive -- even if just because it is a "bad name". Names are 
important.

The second -- in response to a comment made by Josh Smeaton -- is that having 
django.db.models.CharField with default max_lenth=N (for some finite N) and 
django.contrib.postgres.CharField with default max_length=None (meaning 
infinity) sounds like a bad idea.

> 
> I hope this helps!

I'm certain it did!

Shai.

Re: Improving django.setup() and app loading performance

[Rich Jones]

Hey all!

Let me clarify a few of the terms here and describe a bit how Django 
operates in this context.

"Serverless" in this contexts means "without any permanent infrastructure". 
The server is created _after_ the request comes in, and it dies once the 
response is returned. The means that we never have to worry about server 
operations or horizontal scalability, and we pay far, far far less of a 
cost, as we only pay for server time by the millisecond. It's also 
radically easier to deploy - a single 'python manage.py deploy production' 
gives you an infinitely scalable, zero-maintenance  app. Basically, Zappa 
is what comes after Heroku.

To do this, we use two services - Amazon Lambda and Amazon API Gateway.

The first, AWS Lamdba - allows us to define any arbitrary function, which 
Amazon will then cache to memory and execute in response to any AWS system 
event (S3 uploads, Emails, SQS events, etc.) This was designed for small 
functions, but I've been able to squeeze all of Django into it.

The other piece, API Gateway, allows us to turn HTTP requests into AWS 
events - in this case our Lambda context. This requires using a nasty 
language called 'VTL', but you don't need to worry about this.

Zappa converts this API Gateway request into a 'normal' Python WSGI 
request, feeds it to Django, gets the response back, and performs some 
magic on it that lets it get back out through API Gateway.

You can see my slides about this here: 
http://jsbin.com/movecayuba/1/edit?output
and a screencast here: https://www.youtube.com/watch?v=plUrbPN0xc8

Now, this comes with a cost, but that's the trade off. The flip side is 
that it also means we need to call Django's 'setup()' method every time. 
All of this currently takes about ~150ms - the majority of which is spent 
setting up the apps. If we could do that in parallel, this would greatly 
increase the performance of every django-zappa request. Make sense?



*We also have a Slack channel  
where we are working on this if you want to come by! *R

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/69403014-64e1-473a-86fc-88ccaea7ac91%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Rich Jones]

For those who are still following along, these are the lines in question:

https://github.com/Miserlou/django-zappa/blob/master/django_zappa/handler.py#L31
https://github.com/Miserlou/django-zappa/blob/master/django_zappa/handler.py#L68

It's very possible there are ways of significantly improving this. 
Suggestions always welcome!

R

On Monday, February 29, 2016 at 11:26:17 PM UTC+1, Rich Jones wrote:
>
> Hey all!
>
> Let me clarify a few of the terms here and describe a bit how Django 
> operates in this context.
>
> "Serverless" in this contexts means "without any permanent 
> infrastructure". The server is created _after_ the request comes in, and it 
> dies once the response is returned. The means that we never have to worry 
> about server operations or horizontal scalability, and we pay far, far far 
> less of a cost, as we only pay for server time by the millisecond. It's 
> also radically easier to deploy - a single 'python manage.py deploy 
> production' gives you an infinitely scalable, zero-maintenance  app. 
> Basically, Zappa is what comes after Heroku.
>
> To do this, we use two services - Amazon Lambda and Amazon API Gateway.
>
> The first, AWS Lamdba - allows us to define any arbitrary function, which 
> Amazon will then cache to memory and execute in response to any AWS system 
> event (S3 uploads, Emails, SQS events, etc.) This was designed for small 
> functions, but I've been able to squeeze all of Django into it.
>
> The other piece, API Gateway, allows us to turn HTTP requests into AWS 
> events - in this case our Lambda context. This requires using a nasty 
> language called 'VTL', but you don't need to worry about this.
>
> Zappa converts this API Gateway request into a 'normal' Python WSGI 
> request, feeds it to Django, gets the response back, and performs some 
> magic on it that lets it get back out through API Gateway.
>
> You can see my slides about this here: 
> http://jsbin.com/movecayuba/1/edit?output
> and a screencast here: https://www.youtube.com/watch?v=plUrbPN0xc8
>
> Now, this comes with a cost, but that's the trade off. The flip side is 
> that it also means we need to call Django's 'setup()' method every time. 
> All of this currently takes about ~150ms - the majority of which is spent 
> setting up the apps. If we could do that in parallel, this would greatly 
> increase the performance of every django-zappa request. Make sense?
>
>
>
> *We also have a Slack channel  
> where we are working on this if you want to come by! *R
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/eee4285f-243b-494a-ae20-7678c8d4cb7c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add support for relative imports in django.conf.urls.include()?

[Josh Smeaton]

Going off topic a little bit but.. have we considered deprecating string 
based includes in favour of actually importing the urls module?

from . import myapp
urlpatterns = (
 url(r'^myapp/', include(myapp.urls.urlpatterns)),
)


Then users can relatively import their nested app patterns as they will.

Even if we didn't deprecate string based imports I'm wary of increasing 
their utility.

On Tuesday, 1 March 2016 00:17:39 UTC+11, Tim Graham wrote:
>
> For example, within myproject.urls:
>
>
> urlpatterns = (
> url(r'', include('.myapp.urls')),
> )
>
>
> .. becomes equivalent to::
>
> urlpatterns = (
> url(r'', include('myproject.myapp.urls')),
> )
>  
>
> Whilst a contrived example, this can make heavy-nested apps look far, far 
> cleaner. For example, within myproject.foo.foo_bar.foo_bar_baz.urls, one 
> only needs to include .foo_bar_baz_qux.urls to get to the "next" level, 
> rather than the significantly more unwieldy 
> myproject.foo.foo_bar.foo_bar_baz.foo.bar_baz_qux.urls.
>
> 
>
> What do you think of this proposal from 
> https://code.djangoproject.com/ticket/26288? I don't know if "nested 
> apps" are very common and/or something that should be promoted with a 
> change like this.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/20f41a42-55ec-45b1-912f-f261e2450f6b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Add support for relative imports in django.conf.urls.include()?

[Marten Kenbeek]

Hi Josh,

On Monday, February 29, 2016 at 11:34:18 PM UTC+1, Josh Smeaton wrote:
>
> Going off topic a little bit but.. have we considered deprecating string 
> based includes in favour of actually importing the urls module?
>

I've tried to remove them as a proof of concept, but that's not possible 
without some small regressions and test failures. Currently, urlpatterns 
are loaded lazily, and when they're loaded, the complete content of the 
namespace is loaded. This allows you to recursively include the same 
URLconf, but only when you include the URLconf using a namespace (otherwise 
the code attempts to load every level, and you'll run into a maximum 
recursion depth error). This is done in the test suite as well[1], hence 
the test failures.

I haven't been able to think of any practical use-cases, but you never know 
what's out there. The best I can come up with is to include an arbitrary 
number of positional arguments, but I think that's better handled by 
capturing them as one argument and splitting in the view, or even better, 
by using GET parameters.

I don't think string based imports are much of a problem in the case of 
include. The import is done immediately, and the errors are clear. 
Deprecating does have the benefit of increased consistency and less magic, 
but otherwise I don't see any practical benefits.

Then users can relatively import their nested app patterns as they will.
>
 

Even if we didn't deprecate string based imports I'm wary of increasing 
> their utility.


This just made me realize that the whole problem can already be fixed from 
the user's perspective by importing the module instead of using string 
based imports. That is possible and has been possible for a long time. In 
that light, I don't see the benefit of supporting relative string based 
imports with some confusing edge-cases. 

[1] 
https://github.com/django/django/blob/eac1423f9ebcf432dc5be95d605d124a05ab2686/tests/urlpatterns_reverse/namespace_urls.py#L57
 

> On Tuesday, 1 March 2016 00:17:39 UTC+11, Tim Graham wrote:
>>
>> For example, within myproject.urls:
>>
>>
>> urlpatterns = (
>> url(r'', include('.myapp.urls')),
>> )
>>
>>
>> .. becomes equivalent to::
>>
>> urlpatterns = (
>> url(r'', include('myproject.myapp.urls')),
>> )
>>  
>>
>> Whilst a contrived example, this can make heavy-nested apps look far, far 
>> cleaner. For example, within myproject.foo.foo_bar.foo_bar_baz.urls, one 
>> only needs to include .foo_bar_baz_qux.urls to get to the "next" level, 
>> rather than the significantly more unwieldy 
>> myproject.foo.foo_bar.foo_bar_baz.foo.bar_baz_qux.urls.
>>
>> 
>>
>> What do you think of this proposal from 
>> https://code.djangoproject.com/ticket/26288? I don't know if "nested 
>> apps" are very common and/or something that should be promoted with a 
>> change like this.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/e50521cf-5535-45dc-8d50-a10073de385b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Cristiano Coelho]

Sorry if this looks like a retarded question, but have you tried the setup 
calls to the top of the lambda_handler module? I'm not sure why you need 
the settings data as an argument to the lambda handler, but if you find a 
way to move those 4 lines near setup(), you will only load the whole django 
machinery once (or until AWS decides to kill your instance). I have a wild 
guess that this is related to the way you have implemented the "publish to 
aws" process. But the main issue here is that you are calling setup() on 
every request, where you really shouldn't be doing that, but rather do it 
at module level.

I'm sorry if this goes away too far from the actual thread, since this 
looks more like a response to a zappa forum :)



El lunes, 29 de febrero de 2016, 19:34:16 (UTC-3), Rich Jones escribió:
>
> For those who are still following along, these are the lines in question:
>
>
> https://github.com/Miserlou/django-zappa/blob/master/django_zappa/handler.py#L31
>
> https://github.com/Miserlou/django-zappa/blob/master/django_zappa/handler.py#L68
>
> It's very possible there are ways of significantly improving this. 
> Suggestions always welcome!
>
> R
>
> On Monday, February 29, 2016 at 11:26:17 PM UTC+1, Rich Jones wrote:
>>
>> Hey all!
>>
>> Let me clarify a few of the terms here and describe a bit how Django 
>> operates in this context.
>>
>> "Serverless" in this contexts means "without any permanent 
>> infrastructure". The server is created _after_ the request comes in, and it 
>> dies once the response is returned. The means that we never have to worry 
>> about server operations or horizontal scalability, and we pay far, far far 
>> less of a cost, as we only pay for server time by the millisecond. It's 
>> also radically easier to deploy - a single 'python manage.py deploy 
>> production' gives you an infinitely scalable, zero-maintenance  app. 
>> Basically, Zappa is what comes after Heroku.
>>
>> To do this, we use two services - Amazon Lambda and Amazon API Gateway.
>>
>> The first, AWS Lamdba - allows us to define any arbitrary function, which 
>> Amazon will then cache to memory and execute in response to any AWS system 
>> event (S3 uploads, Emails, SQS events, etc.) This was designed for small 
>> functions, but I've been able to squeeze all of Django into it.
>>
>> The other piece, API Gateway, allows us to turn HTTP requests into AWS 
>> events - in this case our Lambda context. This requires using a nasty 
>> language called 'VTL', but you don't need to worry about this.
>>
>> Zappa converts this API Gateway request into a 'normal' Python WSGI 
>> request, feeds it to Django, gets the response back, and performs some 
>> magic on it that lets it get back out through API Gateway.
>>
>> You can see my slides about this here: 
>> http://jsbin.com/movecayuba/1/edit?output
>> and a screencast here: https://www.youtube.com/watch?v=plUrbPN0xc8
>>
>> Now, this comes with a cost, but that's the trade off. The flip side is 
>> that it also means we need to call Django's 'setup()' method every time. 
>> All of this currently takes about ~150ms - the majority of which is spent 
>> setting up the apps. If we could do that in parallel, this would greatly 
>> increase the performance of every django-zappa request. Make sense?
>>
>>
>>
>> *We also have a Slack channel  
>> where we are working on this if you want to come by! *R
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/6acb21e3-701e-4aee-b7c7-7bd9a51f1ace%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Making max_length argument optional

[Cristiano Coelho]

I find that using TextField rather than CharField just to make postgres use 
varchar() is a terrible idea, if you are implementing an reusable app and 
it is used on a backend like MySQL where TextFields are created as text 
columns which are horribly inneficient and should be avoided at any cost 
you will have a really bad time.
I'm not sure about postgres but I want to believe that using varchar 
without limits has also some performance considerations that should be 
taken care of.

El lunes, 29 de febrero de 2016, 17:58:33 (UTC-3), Shai Berger escribió:
>
> Hi, 
>
> Thank you, Aymeric, for summing up the discussion this way. The division 
> into 
> two separate problems is indeed required, and I fully support the idea of 
> setting max_length's default to 100 or 120. 
>
> There seem to be just two points worth adding to your summary: 
>
> On Monday 29 February 2016 11:19:02 Aymeric Augustin wrote: 
> > 
> > 2) How can we make it easy for PostgreSQL users to just use VARCHAR()? 
> > 
> > Since this is a PostgreSQL-specific feature, having a variant of 
> CharField 
> > in django.contrib.postgres that supports and perhaps even defaults to 
> > unlimited length shouldn’t be controversial. 
> > 
>
> The first -- I believe it was raised very early on by Christophe Pettus -- 
> is 
> that Django already has a field that manifests on PG as VARCHAR(), and 
> that is 
> TextField. However, I don't like the idea that PG users should be using 
> TextField(widget=TextInput) as a replacement for CharField; I find that 
> counter-intuitive -- even if just because it is a "bad name". Names are 
> important. 
>
> The second -- in response to a comment made by Josh Smeaton -- is that 
> having 
> django.db.models.CharField with default max_lenth=N (for some finite N) 
> and 
> django.contrib.postgres.CharField with default max_length=None (meaning 
> infinity) sounds like a bad idea. 
>
> > 
> > I hope this helps! 
>
> I'm certain it did! 
>
> Shai. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/37cd6e9c-ec5e-4a69-99a5-84ca11853afe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Rich Jones]

Haven't tried that! I don't _think_ that'll work.. but worth a shot (I 
don't think they only cache the handler.. I think they cache the whole 
environment). Will report back! And as you're mentioning, we really 
shouldn't be doing it every request, so if there were even a way to cache 
that manually rather than calculate it every time, that'd be just as 
valuable.

There is no "zappa forum" - it's just me and a few other contributors in a 
Slack channel! And all of this stuff is super new, and I'm sure that you 
guys know a lot more about the Django internals than I do, so all 
suggestions are welcome!

R

On Tuesday, March 1, 2016 at 12:57:28 AM UTC+1, Cristiano Coelho wrote:
>
> Sorry if this looks like a retarded question, but have you tried the setup 
> calls to the top of the lambda_handler module? I'm not sure why you need 
> the settings data as an argument to the lambda handler, but if you find a 
> way to move those 4 lines near setup(), you will only load the whole django 
> machinery once (or until AWS decides to kill your instance). I have a wild 
> guess that this is related to the way you have implemented the "publish to 
> aws" process. But the main issue here is that you are calling setup() on 
> every request, where you really shouldn't be doing that, but rather do it 
> at module level.
>
> I'm sorry if this goes away too far from the actual thread, since this 
> looks more like a response to a zappa forum :)
>
>
>
> El lunes, 29 de febrero de 2016, 19:34:16 (UTC-3), Rich Jones escribió:
>>
>> For those who are still following along, these are the lines in question:
>>
>>
>> https://github.com/Miserlou/django-zappa/blob/master/django_zappa/handler.py#L31
>>
>> https://github.com/Miserlou/django-zappa/blob/master/django_zappa/handler.py#L68
>>
>> It's very possible there are ways of significantly improving this. 
>> Suggestions always welcome!
>>
>> R
>>
>> On Monday, February 29, 2016 at 11:26:17 PM UTC+1, Rich Jones wrote:
>>>
>>> Hey all!
>>>
>>> Let me clarify a few of the terms here and describe a bit how Django 
>>> operates in this context.
>>>
>>> "Serverless" in this contexts means "without any permanent 
>>> infrastructure". The server is created _after_ the request comes in, and it 
>>> dies once the response is returned. The means that we never have to worry 
>>> about server operations or horizontal scalability, and we pay far, far far 
>>> less of a cost, as we only pay for server time by the millisecond. It's 
>>> also radically easier to deploy - a single 'python manage.py deploy 
>>> production' gives you an infinitely scalable, zero-maintenance  app. 
>>> Basically, Zappa is what comes after Heroku.
>>>
>>> To do this, we use two services - Amazon Lambda and Amazon API Gateway.
>>>
>>> The first, AWS Lamdba - allows us to define any arbitrary function, 
>>> which Amazon will then cache to memory and execute in response to any AWS 
>>> system event (S3 uploads, Emails, SQS events, etc.) This was designed for 
>>> small functions, but I've been able to squeeze all of Django into it.
>>>
>>> The other piece, API Gateway, allows us to turn HTTP requests into AWS 
>>> events - in this case our Lambda context. This requires using a nasty 
>>> language called 'VTL', but you don't need to worry about this.
>>>
>>> Zappa converts this API Gateway request into a 'normal' Python WSGI 
>>> request, feeds it to Django, gets the response back, and performs some 
>>> magic on it that lets it get back out through API Gateway.
>>>
>>> You can see my slides about this here: 
>>> http://jsbin.com/movecayuba/1/edit?output
>>> and a screencast here: https://www.youtube.com/watch?v=plUrbPN0xc8
>>>
>>> Now, this comes with a cost, but that's the trade off. The flip side is 
>>> that it also means we need to call Django's 'setup()' method every time. 
>>> All of this currently takes about ~150ms - the majority of which is spent 
>>> setting up the apps. If we could do that in parallel, this would greatly 
>>> increase the performance of every django-zappa request. Make sense?
>>>
>>>
>>>
>>> *We also have a Slack channel  
>>> where we are working on this if you want to come by! *R
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/156e4949-d69c-4d15-9bbf-f0510902921b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Cristiano Coelho]

I'm almost sure that right now you are calling setup() with django already 
initialized in some cases where the enviorment is reused, I'm amazed django 
doesn't complain when setup() is called twice.

El lunes, 29 de febrero de 2016, 21:07:32 (UTC-3), Rich Jones escribió:
>
> Haven't tried that! I don't _think_ that'll work.. but worth a shot (I 
> don't think they only cache the handler.. I think they cache the whole 
> environment). Will report back! And as you're mentioning, we really 
> shouldn't be doing it every request, so if there were even a way to cache 
> that manually rather than calculate it every time, that'd be just as 
> valuable.
>
> There is no "zappa forum" - it's just me and a few other contributors in a 
> Slack channel! And all of this stuff is super new, and I'm sure that you 
> guys know a lot more about the Django internals than I do, so all 
> suggestions are welcome!
>
> R
>
> On Tuesday, March 1, 2016 at 12:57:28 AM UTC+1, Cristiano Coelho wrote:
>>
>> Sorry if this looks like a retarded question, but have you tried the 
>> setup calls to the top of the lambda_handler module? I'm not sure why you 
>> need the settings data as an argument to the lambda handler, but if you 
>> find a way to move those 4 lines near setup(), you will only load the whole 
>> django machinery once (or until AWS decides to kill your instance). I have 
>> a wild guess that this is related to the way you have implemented the 
>> "publish to aws" process. But the main issue here is that you are calling 
>> setup() on every request, where you really shouldn't be doing that, but 
>> rather do it at module level.
>>
>> I'm sorry if this goes away too far from the actual thread, since this 
>> looks more like a response to a zappa forum :)
>>
>>
>>
>> El lunes, 29 de febrero de 2016, 19:34:16 (UTC-3), Rich Jones escribió:
>>>
>>> For those who are still following along, these are the lines in question:
>>>
>>>
>>> https://github.com/Miserlou/django-zappa/blob/master/django_zappa/handler.py#L31
>>>
>>> https://github.com/Miserlou/django-zappa/blob/master/django_zappa/handler.py#L68
>>>
>>> It's very possible there are ways of significantly improving this. 
>>> Suggestions always welcome!
>>>
>>> R
>>>
>>> On Monday, February 29, 2016 at 11:26:17 PM UTC+1, Rich Jones wrote:

 Hey all!

 Let me clarify a few of the terms here and describe a bit how Django 
 operates in this context.

 "Serverless" in this contexts means "without any permanent 
 infrastructure". The server is created _after_ the request comes in, and 
 it 
 dies once the response is returned. The means that we never have to worry 
 about server operations or horizontal scalability, and we pay far, far far 
 less of a cost, as we only pay for server time by the millisecond. It's 
 also radically easier to deploy - a single 'python manage.py deploy 
 production' gives you an infinitely scalable, zero-maintenance  app. 
 Basically, Zappa is what comes after Heroku.

 To do this, we use two services - Amazon Lambda and Amazon API Gateway.

 The first, AWS Lamdba - allows us to define any arbitrary function, 
 which Amazon will then cache to memory and execute in response to any AWS 
 system event (S3 uploads, Emails, SQS events, etc.) This was designed for 
 small functions, but I've been able to squeeze all of Django into it.

 The other piece, API Gateway, allows us to turn HTTP requests into AWS 
 events - in this case our Lambda context. This requires using a nasty 
 language called 'VTL', but you don't need to worry about this.

 Zappa converts this API Gateway request into a 'normal' Python WSGI 
 request, feeds it to Django, gets the response back, and performs some 
 magic on it that lets it get back out through API Gateway.

 You can see my slides about this here: 
 http://jsbin.com/movecayuba/1/edit?output
 and a screencast here: https://www.youtube.com/watch?v=plUrbPN0xc8

 Now, this comes with a cost, but that's the trade off. The flip side is 
 that it also means we need to call Django's 'setup()' method every time. 
 All of this currently takes about ~150ms - the majority of which is spent 
 setting up the apps. If we could do that in parallel, this would greatly 
 increase the performance of every django-zappa request. Make sense?



 *We also have a Slack channel  
 where we are working on this if you want to come by! *R



-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at 

Re: Improving django.setup() and app loading performance

[Rich Jones]

As I suspected, moving setup() outside of the handler had a negligible 
effect - in fact the test showed a slight drop in performance. :(

Testing from httping. From Berlin to US-East-1:

Before:
--- 
https://arb9clq9k9.execute-api.us-east-1.amazonaws.com/unicode/json_example/ 
ping statistics ---
52 connects, 52 ok, 0.00% failed, time 56636ms
round-trip min/avg/max = 59.1/104.8/301.9 ms

After:
--- 
https://arb9clq9k9.execute-api.us-east-1.amazonaws.com/unicode/json_example/ 
ping statistics ---
51 connects, 51 ok, 0.00% failed, time 57306ms
round-trip min/avg/max = 61.8/128.7/523.2 ms

It was a nice thought though!

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/489869be-cde5-4833-9406-1be565f7900d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Cristiano Coelho]

That's quite odd, I recall testing this once, where I created a lambda 
which had a datetime.now() at the top, and just returned that value. Out of 
a few calls, it returned two different results, meaning the module was re 
used "most" of the time. This was tested calling the lambda from the AWS 
Test itself and not through API Gateway, so perhaps API Gateway is 
preventing the module from being re used? Could be there anything else that 
might prevent AWS from re using the module?

El lunes, 29 de febrero de 2016, 21:27:28 (UTC-3), Rich Jones escribió:
>
> As I suspected, moving setup() outside of the handler had a negligible 
> effect - in fact the test showed a slight drop in performance. :(
>
> Testing from httping. From Berlin to US-East-1:
>
> Before:
> --- 
> https://arb9clq9k9.execute-api.us-east-1.amazonaws.com/unicode/json_example/ 
> ping statistics ---
> 52 connects, 52 ok, 0.00% failed, time 56636ms
> round-trip min/avg/max = 59.1/104.8/301.9 ms
>
> After:
> --- 
> https://arb9clq9k9.execute-api.us-east-1.amazonaws.com/unicode/json_example/ 
> ping statistics ---
> 51 connects, 51 ok, 0.00% failed, time 57306ms
> round-trip min/avg/max = 61.8/128.7/523.2 ms
>
> It was a nice thought though!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/bfd3e770-2b09-4129-9ebd-cd1246a9c33d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Cristiano Coelho]

I have repeated the test this time through API Gateway, and out of many 
calls I only got two different dates that were instantiated at module 
level, meaning my module was only imported twice. I fail to see why it 
doesn't behave the same with your code.

El lunes, 29 de febrero de 2016, 21:33:05 (UTC-3), Cristiano Coelho 
escribió:
>
> That's quite odd, I recall testing this once, where I created a lambda 
> which had a datetime.now() at the top, and just returned that value. Out of 
> a few calls, it returned two different results, meaning the module was re 
> used "most" of the time. This was tested calling the lambda from the AWS 
> Test itself and not through API Gateway, so perhaps API Gateway is 
> preventing the module from being re used? Could be there anything else that 
> might prevent AWS from re using the module?
>
> El lunes, 29 de febrero de 2016, 21:27:28 (UTC-3), Rich Jones escribió:
>>
>> As I suspected, moving setup() outside of the handler had a negligible 
>> effect - in fact the test showed a slight drop in performance. :(
>>
>> Testing from httping. From Berlin to US-East-1:
>>
>> Before:
>> --- 
>> https://arb9clq9k9.execute-api.us-east-1.amazonaws.com/unicode/json_example/ 
>> ping statistics ---
>> 52 connects, 52 ok, 0.00% failed, time 56636ms
>> round-trip min/avg/max = 59.1/104.8/301.9 ms
>>
>> After:
>> --- 
>> https://arb9clq9k9.execute-api.us-east-1.amazonaws.com/unicode/json_example/ 
>> ping statistics ---
>> 51 connects, 51 ok, 0.00% failed, time 57306ms
>> round-trip min/avg/max = 61.8/128.7/523.2 ms
>>
>> It was a nice thought though!
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/a423a181-66c1-48da-89b2-b2cf32479351%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Rich Jones]

That certainly could have something to do with it - there isn't very much 
transparency about how API Gateway works. It's super new and pretty janky, 
TBQH. However, I think that the behavior describing is not what's expected 
- the caching seems to be for the assets of the whole environment, not of 
anything that's computed - whether or not they are held in memory or read 
from disk.

[[ Also, obviously it's not a fair comparison, but I thought I'd include 
these numbers for reference:
--- http://djangoproject.com/ ping statistics ---
52 connects, 52 ok, 0.00% failed, time 68473ms
round-trip min/avg/max = 227.9/329.3/1909.3 ms ]]

So, I think the interesting things to explore would be:
 - Loading apps in parallel
 - "Pre-loading" apps before app deployment, then loading that cached state 
at runtime. I guess I'd need to know more about what it means to "load" an 
app to see if that makes any sense at all.

I imagine the former is probably more feasible. I understand the aversion 
to non-determinism, but I think that shouldn't matter as long as there is a 
way to define inter-app dependencies.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/fbc32a62-4d29-4ecf-bbd6-a5a2256b564b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Cristiano Coelho]

Rich, I have just performed a real test, with a simple lambda and a 
datetime.now() defined at the top of the module as I said, and out of 100 
requests, this was the result:
{u'2016-03-01T00:37:30.476828': [43], u'2016-03-01T00:36:51.536025': [58]}
Where the date is the datetime.now() defined at the module top, and the 
number is the amount of times that same value was returned (oddly it sums 
101, I think I did an additional one). So the module is really being 
reused, even with API Gateway (I did the test from python to a remote 
address). So in your case there must be something else going on or I did 
this test wrongly


El lunes, 29 de febrero de 2016, 21:44:43 (UTC-3), Rich Jones escribió:
>
> That certainly could have something to do with it - there isn't very much 
> transparency about how API Gateway works. It's super new and pretty janky, 
> TBQH. However, I think that the behavior describing is not what's expected 
> - the caching seems to be for the assets of the whole environment, not of 
> anything that's computed - whether or not they are held in memory or read 
> from disk.
>
> [[ Also, obviously it's not a fair comparison, but I thought I'd include 
> these numbers for reference:
> --- http://djangoproject.com/ ping statistics ---
> 52 connects, 52 ok, 0.00% failed, time 68473ms
> round-trip min/avg/max = 227.9/329.3/1909.3 ms ]]
>
> So, I think the interesting things to explore would be:
>  - Loading apps in parallel
>  - "Pre-loading" apps before app deployment, then loading that cached 
> state at runtime. I guess I'd need to know more about what it means to 
> "load" an app to see if that makes any sense at all.
>
> I imagine the former is probably more feasible. I understand the aversion 
> to non-determinism, but I think that shouldn't matter as long as there is a 
> way to define inter-app dependencies.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/587ceaf9-9881-497c-a207-fc5f6c27199c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Making max_length argument optional

[Markus Holtermann]

>From what I understand you will have a hard time doing that at all:

On PG could go with a 'max_length=None' in a 3rd party app. But that 
wouldn't be supported on any other database backend. Which means you're 
limiting your app to PG. On the other hand you could make your app database 
independent by using a TextField which has pretty much the same performance 
on PG as a VARCHAR() and is available on other backends. But may perform 
bad on other backends. But there's no way to have a text of unlimited 
length on MySQL other than a TEXT column.

TL;DR: you can't achieve both ways anyway. Neither now nor if we change the 
behavior of Django's max_length attribute on CharFields.

/Markus

On Tuesday, March 1, 2016 at 11:00:29 AM UTC+11, Cristiano Coelho wrote:
>
> I find that using TextField rather than CharField just to make postgres 
> use varchar() is a terrible idea, if you are implementing an reusable app 
> and it is used on a backend like MySQL where TextFields are created as text 
> columns which are horribly inneficient and should be avoided at any cost 
> you will have a really bad time.
> I'm not sure about postgres but I want to believe that using varchar 
> without limits has also some performance considerations that should be 
> taken care of.
>
> El lunes, 29 de febrero de 2016, 17:58:33 (UTC-3), Shai Berger escribió:
>>
>> Hi, 
>>
>> Thank you, Aymeric, for summing up the discussion this way. The division 
>> into 
>> two separate problems is indeed required, and I fully support the idea of 
>> setting max_length's default to 100 or 120. 
>>
>> There seem to be just two points worth adding to your summary: 
>>
>> On Monday 29 February 2016 11:19:02 Aymeric Augustin wrote: 
>> > 
>> > 2) How can we make it easy for PostgreSQL users to just use VARCHAR()? 
>> > 
>> > Since this is a PostgreSQL-specific feature, having a variant of 
>> CharField 
>> > in django.contrib.postgres that supports and perhaps even defaults to 
>> > unlimited length shouldn’t be controversial. 
>> > 
>>
>> The first -- I believe it was raised very early on by Christophe Pettus 
>> -- is 
>> that Django already has a field that manifests on PG as VARCHAR(), and 
>> that is 
>> TextField. However, I don't like the idea that PG users should be using 
>> TextField(widget=TextInput) as a replacement for CharField; I find that 
>> counter-intuitive -- even if just because it is a "bad name". Names are 
>> important. 
>>
>> The second -- in response to a comment made by Josh Smeaton -- is that 
>> having 
>> django.db.models.CharField with default max_lenth=N (for some finite N) 
>> and 
>> django.contrib.postgres.CharField with default max_length=None (meaning 
>> infinity) sounds like a bad idea. 
>>
>> > 
>> > I hope this helps! 
>>
>> I'm certain it did! 
>>
>> Shai. 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/71e5e743-29bb-4dba-9bf4-6948ab2d1fa9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Rich Jones]

Hm. This is the downside of cloud services - we cannot look under the hood.

Since I think that since this is something we _want_ cached, and because it 
will make the function being executed shorter in length - it is a good code 
decision to make. Thank you for the idea! However, it looks like the actual 
result is negligible in terms of round-trip time.

Maybe you can try with httping and show me your times? Or give django-zappa 
a shot yourself! This is the app I'm testing with: 
https://github.com/Miserlou/django-helloworld

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/aa2ce7e4-647b-4f3e-ad2a-732f0d6ee0ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Cristiano Coelho]

I think I have found your issue, if I'm not wrong, django won't initialize 
twice if you call it twice (which was your previous behaviour) at least not 
completely, since apps registry has a "ready" flag.
Check this line: 
https://github.com/django/django/blob/master/django/apps/registry.py#L66

So basically having setup() on top or inside the handler, should yield 
almost the same results since django won't re load apps that are already 
loaded, and that's why the change didn't yield any different results. Does 
the slow down come from somewhere else rather than setup()

>From here doing 100 requests these are the results, times are quite higher 
since I have at least 200ms round trip from here to the USA.

min: 0.34s
max: 3.58
avg:  0.57s

El lunes, 29 de febrero de 2016, 22:21:11 (UTC-3), Rich Jones escribió:
>
> Hm. This is the downside of cloud services - we cannot look under the hood.
>
> Since I think that since this is something we _want_ cached, and because 
> it will make the function being executed shorter in length - it is a good 
> code decision to make. Thank you for the idea! However, it looks like the 
> actual result is negligible in terms of round-trip time.
>
> Maybe you can try with httping and show me your times? Or give 
> django-zappa a shot yourself! This is the app I'm testing with: 
> https://github.com/Miserlou/django-helloworld
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/1e45f5b8-e40a-4d97-bc88-d139550426cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Rich Jones]

Ah, interesting! Thanks for tracking that down.

In chat we basically discovered that the intercontinental latency and 
shoddy wifi connections were responsible for a lot of the confusion.

Testing from a US-based fiber connection, we got results of ~40ms in both 
scenarios.

 --- 
https://arb9clq9k9.execute-api.us-east-1.amazonaws.com/unicode/json_example/ 
ping statistics ---
16 connects, 16 ok, 0.00% failed, time 15989ms
round-trip min/avg/max = 32.7/42.3/142.4 ms

So, there you have it! As if you needed more of a reason to get down with 
Zappa?! :-D

(Now, that's with a naive-example. Expect this thread to get bumped when we 
start really hitting the databases..)

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/32553dde-3106-4e65-b7a9-d05ad0256b50%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Cristiano Coelho]

In my opinion, those latencies are still a bit high, how much is really 
used on python/lambda code?  On a project of mine without hitting the 
database and django-rest-framework my times were around 1-4ms excluding any 
network latency. Debug and loggers might have high impact on your times if 
it is using cloud watch.

An interesting way of testing concurrency is to use apache benchmark. You 
can do it easily with an EC2 machine, ideally on the same region as your 
lambdas to reduce network latency to 0. There you can test on a nearly real 
scenario as this tool is quite good for load testing. Be careful to not do 
a huge test as you might start to get charged :D

I have added a github issue on zappa from reading your handler.py code with 
a few suggestions that may help you improve performance even further. Sorry 
that I'm lazy to make a real pull request.

El lunes, 29 de febrero de 2016, 22:47:43 (UTC-3), Rich Jones escribió:
>
> Ah, interesting! Thanks for tracking that down.
>
> In chat we basically discovered that the intercontinental latency and 
> shoddy wifi connections were responsible for a lot of the confusion.
>
> Testing from a US-based fiber connection, we got results of ~40ms in both 
> scenarios.
>
>  --- 
> https://arb9clq9k9.execute-api.us-east-1.amazonaws.com/unicode/json_example/ 
> ping statistics ---
> 16 connects, 16 ok, 0.00% failed, time 15989ms
> round-trip min/avg/max = 32.7/42.3/142.4 ms
>
> So, there you have it! As if you needed more of a reason to get down with 
> Zappa?! :-D
>
> (Now, that's with a naive-example. Expect this thread to get bumped when 
> we start really hitting the databases..)
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5ced17b0-b5cc-44bb-aebe-3a7605c77945%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Improving django.setup() and app loading performance

[Aymeric Augustin]

Hello Rich,


> On 01 Mar 2016, at 01:44, Rich Jones  wrote:
> 
> I think the interesting things to explore would be:
>  - Loading apps in parallel
>  - "Pre-loading" apps before app deployment, then loading that cached state 
> at runtime. I guess I'd need to know more about what it means to "load" an 
> app to see if that makes any sense at all.


As of Django 1.7, loading an app means:

1. importing its app config class (what the dotted Python path in 
INSTALLED_APPS points to; if INSTALLED_APPS points to a Python package, an app 
config is generated on the fly)
2. importing its models module, if there is one (determined from the app config)
3. running the ready() method of the app config

In practice Django does 1. for all apps, then 2. for all apps, then 3. for all 
apps.

If “load an app” confuses you, read it as “import an app” or “import Python 
modules comprising this app”. Really there isn’t much going on aside from 
importing a bunch of Python modules. Some datastructures are built dynamically 
but I’d be surprised if it took a measurable amount of the startup time.

As far as I can tell from the rest of the discussion, the result of 
django.setup() is already cached and reused anyway, but I thought I’d demystify 
things a bit!


> I imagine the former is probably more feasible. I understand the aversion to 
> non-determinism, but I think that shouldn't matter as long as there is a way 
> to define inter-app dependencies.


Non-deterministic software has this strange tendency to deterministically work 
on developers’ laptops and deterministically fail on production servers ;-)

More seriously, there are cases where importing a Django project works when app 
A is imported before app B but deadlocks when app B is imported before app A. 
Apps that attempt to perform auto-discovery on other apps (such as the admin, 
debug-toolbar, haystack, etc.) are especially prone to this. This isn’t a 
theoretical argument. We’ve had these bugs and I spent hundreds of hours 
refactoring the app-loading process to eliminate them.

Also it’s unclear to me how developers could be sure that they’ve defined all 
required inter-app dependencies.


Best regards,

-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/2114E152-2353-4A51-BDA3-E25C96D5F56A%40polytechnique.org.
For more options, visit https://groups.google.com/d/optout.