Re: Django Channels - Throttling/Traffic Management?

2017-08-10 Thread Andrew Godwin
Hi Jamie,

I don't use Stack Overflow, so feel free to answer there yourself for
posterity.

As for the load-balancing bit, what I mean is that because the Channels
model is that all workers draw from a single work-queue, there are no
balancing artifacts like you can get from a plain round-robin model; the
tradeoff being, of course, more limited capacity.

If you wanted separate workers for the HTTP and WebSocket handling, right
now that can be one using the --only-channels and --exclude-channels
arguments to runworker to specialise them. This will change with the
rewrite as the handling will be local to the socket-terminating process
like WSGI, so you will be able to load balance with Nginx/HAProxy like
anything else. There's no dynamic scaling support at all because that's
something that's far too specialised for a framework to ship.

As for the rewrite, the main source of info right now is my blog post at
https://www.aeracode.org/2017/7/11/towards-channels-20/ - more docs will
come as we finalise the design, but that will take a bit longer.

Andrew

On Tue, Aug 8, 2017 at 1:43 PM, Jamie Counsell  wrote:

> Hey Andrew,
>
> Thanks for the prompt response. If you'd like to post this as an answer to the
> StackOverflow question
> ,
> that would be great. Otherwise, I will answer my own question.
>
> I'm not sure what you mean by workers being load-balanced? Right now, when
> I run daphne, I run several workers on a system level using the runworker
> command. What if I wanted to start more workers under certain conditions?
> Also, if workers are all assigned to work on every channel, then I agree
> this is the case, but if I want to make sure that the HTTP channel never
> fails to respond even if the websocket channel gets hammered, I can start a
> worker that will only work on the HTTP channels. Is there any way to do
> this dynamically?
>
> I understand that this is out of scope of channels, and I appreciate your
> willingness to work with me on this. The main thing I think would be
> helpful is middleware that could be assigned to each channel, similar to
> Django's current HTTP middleware.
>
> Lastly, do you have any documentation or public discussions on this
> rewrite? I'm working on a project right now that I'm hoping will see the
> light of day in production, but would hate to get version-locked if the
> rewrite changes too much.
>
> Thank you! And keep up the great work.
>
> Jamie
>
> On Tuesday, 8 August 2017 15:45:03 UTC-4, Andrew Godwin wrote:
>>
>> Hi Jamie,
>>
>> At the moment Channels has quite limited support for throttling - it
>> pretty much consists of an adjustable channel size for incoming connections
>> which, when full, will cause the server to return a 503 error. Workers are
>> load-balanced based on availability due to the channels design, so there's
>> no risk of a worker gaining a larger queue than others.
>>
>> Providing more advanced DoS or DDoS protection is probably not something
>> we can do within the scope of Channels itself, but I'd like to make sure we
>> provide the appropriate hooks. Were there particular things you think we
>> could implement that would help you write some of the things you need?
>>
>> (It's also worth bearing in mind that right now we're changing the
>> worker/consumer layout substantially as part of a major rewrite, which is
>> going to mean different considerations when scaling, so I don't want to
>> give too precise advice just yet)
>>
>> Andrew
>>
>> On Tue, Aug 8, 2017 at 10:19 AM, Jamie Counsell  wrote:
>>
>>> Hey folks,
>>>
>>> I'm working on a Django Channels project and was wondering if there was
>>> (or was a roadmap for) features that could help us control load issues at
>>> the application level. I asked on StackOverflow
>>> 
>>>  but
>>> made the mistake of including the term "DDoS" and so I received a bunch of
>>> generic DDoS explanations. I know that DDoS is in some cases a different
>>> beast entirely, but I also think that smaller attacks or load issues
>>> related to any traffic increase could be handled at the application level.
>>> For example, Django Rest Framework's Throttling support provides a great
>>> way to do things like prioritize or not throttle authenticated users, but
>>> throttle anonymous users. I'm wondering if Channels or Daphne has support
>>> for something like:
>>>
>>>- Throttling (per channel, ideally)
>>>- Dynamic worker assignment based on queue size, response time, etc.
>>>- Middleware to provide priority to requests based on some metric
>>>(authenticated, for example)
>>>- Priority decorators for consumers that could tell workers not to
>>>run certain consumers if the queue is large
>>>
>>> Right now I have a design that could cause the socket channel to
>>> experience 

Re: Row based permissions: In DB or App? ... farewell pain

2017-08-10 Thread Vijay Khemlani
It's still implemented like that in the background, but I'm not sure why do
you call it slow.

Did you run benchmarks? Profiling?

Usually your own business logic will be the bottleneck of your application,
not your permission checking.

On Thu, Aug 10, 2017 at 5:15 AM, guettli 
wrote:

>
>
> Am Mittwoch, 9. August 2017 18:03:44 UTC+2 schrieb Vijay Khemlani:
>>
>> https://django-guardian.readthedocs.io/en/stable/userguide/
>> check.html#get-objects-for-user
>>
>> projects = get_objects_for_user(request.user, 'projects.view_project')
>>
>>
>>
> Thank you for providing this link. I guess it is new to many developers.
>
> I worked with this method some years ago and it was slow. It was slow
> since the permission checking
> gets done at several ways (via is_superuser, via perms for user, via perms
> for group, ...) But this is old and outdated
> information. I guess this is not true any more.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/django-users/afc9ac15-8e61-400a-a36c-c62b5bac22dc%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CALn3ei3u%3Dk4Js%3DknDkWVAwPSfYVbx5fL78BeFabkAkD1mOT%2Biw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: Row based permissions: In DB or App? ... farewell pain

2017-08-10 Thread guettli


Am Mittwoch, 9. August 2017 18:03:44 UTC+2 schrieb Vijay Khemlani:
>
>
> https://django-guardian.readthedocs.io/en/stable/userguide/check.html#get-objects-for-user
>
> projects = get_objects_for_user(request.user, 'projects.view_project')
>
>
>
Thank you for providing this link. I guess it is new to many developers.

I worked with this method some years ago and it was slow. It was slow since 
the permission checking
gets done at several ways (via is_superuser, via perms for user, via perms 
for group, ...) But this is old and outdated
information. I guess this is not true any more.

 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/afc9ac15-8e61-400a-a36c-c62b5bac22dc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Re: Migration of auth.user to custom user

2017-08-10 Thread James Schneider
On Aug 4, 2017 12:36 AM, "Mike Dewhirst"  wrote:

I have read  widely (including the docs) and been daunted by the notion of
coverting auth.user  into a custom user. I have data and wonder if there
have been any recent recipes for doing this?


On a related note, I'd recommend implementing a custom user model
immediately on every new project, even if you don't intend to change
anything on the built-in user immediately. It is as simple as inheriting
from django.contrib.auth.models.AbstractUser like this:

class MyUser(AbstractUser):
pass

And then setting AUTH_USER_MODEL to MyUser. At that point, changes to the
user model are handled via standard migrations.

Obviously this doesn't help existing projects. You'll still be stuck with
the foreign key shell game that is implementing a custom user after the
fact.

-James

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2Be%2BciWneHQykX-zDFWni8d7O%2BLwGbJNRzknxnNKw_KJftZRug%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


Re: How to Minify, obfuscate, and compress django code

2017-08-10 Thread Antonis Christofides
Hi,

while it is not entirely clear what the original poster wants, if he's referring
to JavaScript then django-compressor provides some support for that.

Regards,

Antonis

Antonis Christofides
http://djangodeployment.com

On 2017-08-10 03:43, Bernd Wechner wrote:
> I'm a little confused as to what this has to do with Django, which runs server
> side. The code it emits to the client is limited to what you put in your
> templates, including perhaps the widgets that django provides for forms.
>
> I use a lot of javascript in my forms, and it's my responsibility to minify
> that and include links to the minified code in my templates in production (and
> unminified in development) but these are all django unrelated build issues.
> And solving them depends on django unrelated tool sets you're using.
>
> -- 
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com
> .
> To post to this group, send email to django-users@googlegroups.com
> .
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/6a306671-399e-40bc-bee0-cf5837d7c5f6%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/4a43b8fb-9181-07d4-a6fd-8a3c357f6115%40djangodeployment.com.
For more options, visit https://groups.google.com/d/optout.


Re: Re: Migration of auth.user to custom user

2017-08-10 Thread Mike Dewhirst

On 7/08/2017 11:55 PM, Tim Graham wrote:

Some steps are outlined in https://code.djangoproject.com/ticket/25313.


OK thanks Tim. I'll start the planning. I'll presume Django 1.10 isn't 
sufficiently different from 1.8 for which the instructions are written.


I wish I wasn't an underconfident wuss!



On Friday, August 4, 2017 at 3:37:08 AM UTC-4, Mike Dewhirst wrote:

I have read  widely (including the docs) and been daunted by the
notion
of coverting auth.user  into a custom user. I have data and wonder if
there have been any recent recipes for doing this?

Thanks

Mike

--
You received this message because you are subscribed to the Google 
Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to django-users+unsubscr...@googlegroups.com 
.
To post to this group, send email to django-users@googlegroups.com 
.

Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/c29496a8-1294-481a-8316-5c288b0af6f9%40googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.



--

Climate Pty Ltd
PO Box 308
Mount Eliza
Vic 3930
Australia +61

T: 03 9034 3977
M: 0411 704 143


--
You received this message because you are subscribed to the Google Groups "Django 
users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/3bade296-ace2-d356-d1bf-d5aa3c25b201%40dewhirst.com.au.
For more options, visit https://groups.google.com/d/optout.