Re: RFC732 (a.k.a. brotli) support

[Václav Dohnal]

Hello everyone and sorry for the late response. I'm the author of 
django-brotli package. Creating a single middleware that can handle 
multiple algorithms sounds like a great idea to me. If you need any kind of 
assistance and/or code from django-brotli package, I'll be happy to help 
you.

Vašek Dohnal

Dne pátek 14. září 2018 10:01:00 UTC+2 Johannes Hoppe napsal(a):
>
> Hi there,
>
> we all know the the wonderful GZip middleware that allows response 
> compression. Which is great especially if you are serving large responses.
>
> Now gzip is only one of many supported encoding types supported by many 
> browsers. The coolest kid on the block seems to be brotli right now.
> It is both faster and smaller in most cases then gzip and has been 
> standardized in https://www.ietf.org/rfc/rfc7932.txt
>
> Of course there is already a package that adds brotli support to Django:
> https://pypi.org/project/django-brotli/
>
> But I would propose to refactor the GZip middleware into something like 
> "CompressionMiddlware" and add support for both GZip and Brotli
> in the same middleware.
>
> I have the feeling something basing like response encoding should be build 
> into Django. Of course I would not add a hard dependency to brotli,
> but add all methods to `Accept-Encoding` that are actually installed.
>
> What do you guys think? Is this worth a little refactoring?
>
> Best
> -Joe
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/5ac0cf5a-41e5-4ff1-a728-e16164d71270%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: RFC732 (a.k.a. brotli) support

[Adam Johnson]

>
> I would require some kind of configuration to define which response
> content-type should be encoded by which algorithm.


I think there's room there to support custom tactics, for example content
length may also be useful to determine which algorithms are applicable, or
perhaps compression could be disabled for certain URL's. GZipMiddleware
doesn't support this but if we're going to add flexibility we should make
it easy to customize, at least by subclassing.

On Sat, 15 Sep 2018 at 11:37, Johannes Hoppe  wrote:

> Hi Curtis,
>
> very good remarks. I would make sense to provide the best possible preset
> for such a middleware. That could even be content-type sensitive.
> I would however add any settings to overwrite the preset. I believe if
> someone want to mingle with those things, one should inherit and override.
>
> But even with a preset, I would follow content negotiation tactics, where
> the request can define preference over the content encoding.
> https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3
>
> Best
> -Joe
>
> On Friday, September 14, 2018 at 12:53:34 PM UTC+2, Curtis Maloney wrote:
>>
>> On 09/14/2018 08:24 PM, Aymeric Augustin wrote:
>> > Hello,
>> >
>> > I would like to be sure that Brotli is actually faster. In my
>> > experience, while decompression is fast, compression can be extremely
>> slow.
>>
>> https://blogs.akamai.com/2016/02/understanding-brotlis-potential.html
>>
>> It's primarily tuned for fast decompression, making it especially good
>> for offline compression (i.e. compression of static assets on deploy)
>>
>> At lower levels it can get as good compression as gzip, but much faster,
>> or similar speed for better results. IIRC the general "wisdom" is that
>> brotli at level 4 is typically faster than gzip, and gives better
>> results.
>>
>> I like the general idea of an "extensible" or plugin driven
>> Content-Encoding handler, though it would require at least some
>> preferencing engine so you can influence which algorithm is chosen by
>> (a) browser supported, (b) local preference, and (c) content type.
>>
>> --
>> Curtis
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/3673361a-dc59-4d27-8050-1ab2cae5d11c%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM0UEYxm%3DK4FNVG-KKSbYRCf8sLwZYijdbhGjnriEhhUxQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: RFC732 (a.k.a. brotli) support

[Johannes Hoppe]

Hi Curtis,

very good remarks. I would make sense to provide the best possible preset 
for such a middleware. That could even be content-type sensitive.
I would however add any settings to overwrite the preset. I believe if 
someone want to mingle with those things, one should inherit and override.

But even with a preset, I would follow content negotiation tactics, where 
the request can define preference over the content encoding.
https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3

Best
-Joe

On Friday, September 14, 2018 at 12:53:34 PM UTC+2, Curtis Maloney wrote:
>
> On 09/14/2018 08:24 PM, Aymeric Augustin wrote: 
> > Hello, 
> > 
> > I would like to be sure that Brotli is actually faster. In my 
> > experience, while decompression is fast, compression can be extremely 
> slow. 
>
> https://blogs.akamai.com/2016/02/understanding-brotlis-potential.html 
>
> It's primarily tuned for fast decompression, making it especially good 
> for offline compression (i.e. compression of static assets on deploy) 
>
> At lower levels it can get as good compression as gzip, but much faster, 
> or similar speed for better results. IIRC the general "wisdom" is that 
> brotli at level 4 is typically faster than gzip, and gives better results. 
>
> I like the general idea of an "extensible" or plugin driven 
> Content-Encoding handler, though it would require at least some 
> preferencing engine so you can influence which algorithm is chosen by 
> (a) browser supported, (b) local preference, and (c) content type. 
>
> -- 
> Curtis 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3673361a-dc59-4d27-8050-1ab2cae5d11c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: RFC732 (a.k.a. brotli) support

[Johannes Hoppe]

Good Point Aymeric,

it will probably not always be a good idea. I would require some kind of 
configuration to define which response content-type should be encoded by 
which algorithm.
It might also be interesting if you could use pre-compressed full page 
caching. Which I think does currently not work. This could however be a 
good way to reduce CPU time.

Curtis also makes a good point on what kind of configraution should be 
supported by such a middlware.

Best
-Joe

On Friday, September 14, 2018 at 12:25:12 PM UTC+2, Aymeric Augustin wrote:
>
> Hello,
>
> I would like to be sure that Brotli is actually faster. In my experience, 
> while decompression is fast, compression can be extremely slow.
>
> This is fine for static assets compressed by a build pipeline or at least 
> when the compressed version is cached. It would be a problem for on-the-fly 
> compression of dynamic content, which is what we're talking about here.
>
> I expect there's a setting for trading off compression time vs. 
> compression ratio. It should be set appropriately. It isn't worth spending 
> a lot more CPU to save a tiny bit of data transfer. The result could be 
> slower, especially for users with fast connections.
>
> I would like to see a quick performance benchmark of gzip as currently 
> implemented in Django vs. brotli with a setting you recommend, to see the 
> effect on compression time (CPU use) and compression ratio.
>
> Best regards,
>
> -- 
> Aymeric.
>
> Le ven. 14 sept. 2018 à 10:01, Johannes Hoppe  > a écrit :
>
>> Hi there,
>>
>> we all know the the wonderful GZip middleware that allows response 
>> compression. Which is great especially if you are serving large responses.
>>
>> Now gzip is only one of many supported encoding types supported by many 
>> browsers. The coolest kid on the block seems to be brotli right now.
>> It is both faster and smaller in most cases then gzip and has been 
>> standardized in https://www.ietf.org/rfc/rfc7932.txt
>>
>> Of course there is already a package that adds brotli support to Django:
>> https://pypi.org/project/django-brotli/
>>
>> But I would propose to refactor the GZip middleware into something like 
>> "CompressionMiddlware" and add support for both GZip and Brotli
>> in the same middleware.
>>
>> I have the feeling something basing like response encoding should be 
>> build into Django. Of course I would not add a hard dependency to brotli,
>> but add all methods to `Accept-Encoding` that are actually installed.
>>
>> What do you guys think? Is this worth a little refactoring?
>>
>> Best
>> -Joe
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/29cf5142-fa82-4d74-9d8f-5fb0c8be34d0%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> -- 
> Aymeric.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/3e7c5b75-fb08-40f1-9394-97fcd33ff6d3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: RFC732 (a.k.a. brotli) support

[Johannes Hoppe]

Hi Adam,

no I would not take any of his code. I would certainly contact him as a 
reviewer or co-author of a possible patch.

In general I would like to consolidate the two middlewares into a single 
response compression middleware that support multiple algorithms. All 
except gzip would require users to install additional dependencies. But I 
believe this to be acceptable. Same as you need to install Pillow if you 
want Image support in Django.

Best
-Joe

On Friday, September 14, 2018 at 11:31:43 AM UTC+2, Adam Johnson wrote:
>
> Sounds like a good idea to me.
>
> Would you be looking at merging any code from django-brotli, and have you 
> been in contact with its creator and license holder Vasek Dohnal?
>
> And what kind of methods around Accept-Encoding are you imagining?
>
> On Fri, 14 Sep 2018 at 09:01, Johannes Hoppe  > wrote:
>
>> Hi there,
>>
>> we all know the the wonderful GZip middleware that allows response 
>> compression. Which is great especially if you are serving large responses.
>>
>> Now gzip is only one of many supported encoding types supported by many 
>> browsers. The coolest kid on the block seems to be brotli right now.
>> It is both faster and smaller in most cases then gzip and has been 
>> standardized in https://www.ietf.org/rfc/rfc7932.txt
>>
>> Of course there is already a package that adds brotli support to Django:
>> https://pypi.org/project/django-brotli/
>>
>> But I would propose to refactor the GZip middleware into something like 
>> "CompressionMiddlware" and add support for both GZip and Brotli
>> in the same middleware.
>>
>> I have the feeling something basing like response encoding should be 
>> build into Django. Of course I would not add a hard dependency to brotli,
>> but add all methods to `Accept-Encoding` that are actually installed.
>>
>> What do you guys think? Is this worth a little refactoring?
>>
>> Best
>> -Joe
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers (Contributions to Django itself)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to django-develop...@googlegroups.com .
>> To post to this group, send email to django-d...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/django-developers.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/django-developers/29cf5142-fa82-4d74-9d8f-5fb0c8be34d0%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
> -- 
> Adam
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/7b2c1967-ad49-42c6-967e-f91ec0e41a5e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: RFC732 (a.k.a. brotli) support

[Curtis Maloney]

On 09/14/2018 08:24 PM, Aymeric Augustin wrote:

Hello,

I would like to be sure that Brotli is actually faster. In my 
experience, while decompression is fast, compression can be extremely slow.


https://blogs.akamai.com/2016/02/understanding-brotlis-potential.html

It's primarily tuned for fast decompression, making it especially good 
for offline compression (i.e. compression of static assets on deploy)


At lower levels it can get as good compression as gzip, but much faster, 
or similar speed for better results. IIRC the general "wisdom" is that 
brotli at level 4 is typically faster than gzip, and gives better results.


I like the general idea of an "extensible" or plugin driven 
Content-Encoding handler, though it would require at least some 
preferencing engine so you can influence which algorithm is chosen by 
(a) browser supported, (b) local preference, and (c) content type.


--
Curtis

--
You received this message because you are subscribed to the Google Groups "Django 
developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/836eb496-a2bf-5362-695c-673ae9ccd948%40tinbrain.net.
For more options, visit https://groups.google.com/d/optout.

Re: RFC732 (a.k.a. brotli) support

[Aymeric Augustin]

Hello,

I would like to be sure that Brotli is actually faster. In my experience,
while decompression is fast, compression can be extremely slow.

This is fine for static assets compressed by a build pipeline or at least
when the compressed version is cached. It would be a problem for on-the-fly
compression of dynamic content, which is what we're talking about here.

I expect there's a setting for trading off compression time vs. compression
ratio. It should be set appropriately. It isn't worth spending a lot more
CPU to save a tiny bit of data transfer. The result could be slower,
especially for users with fast connections.

I would like to see a quick performance benchmark of gzip as currently
implemented in Django vs. brotli with a setting you recommend, to see the
effect on compression time (CPU use) and compression ratio.

Best regards,

-- 
Aymeric.

Le ven. 14 sept. 2018 à 10:01, Johannes Hoppe  a
écrit :

> Hi there,
>
> we all know the the wonderful GZip middleware that allows response
> compression. Which is great especially if you are serving large responses.
>
> Now gzip is only one of many supported encoding types supported by many
> browsers. The coolest kid on the block seems to be brotli right now.
> It is both faster and smaller in most cases then gzip and has been
> standardized in https://www.ietf.org/rfc/rfc7932.txt
>
> Of course there is already a package that adds brotli support to Django:
> https://pypi.org/project/django-brotli/
>
> But I would propose to refactor the GZip middleware into something like
> "CompressionMiddlware" and add support for both GZip and Brotli
> in the same middleware.
>
> I have the feeling something basing like response encoding should be build
> into Django. Of course I would not add a hard dependency to brotli,
> but add all methods to `Accept-Encoding` that are actually installed.
>
> What do you guys think? Is this worth a little refactoring?
>
> Best
> -Joe
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/29cf5142-fa82-4d74-9d8f-5fb0c8be34d0%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Aymeric.

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CANE-7mUesYZ-tmwoXBXFhA4xf3xvdeFaL4O42_5kG_MSBaE7nw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: RFC732 (a.k.a. brotli) support

[Adam Johnson]

Sounds like a good idea to me.

Would you be looking at merging any code from django-brotli, and have you
been in contact with its creator and license holder Vasek Dohnal?

And what kind of methods around Accept-Encoding are you imagining?

On Fri, 14 Sep 2018 at 09:01, Johannes Hoppe  wrote:

> Hi there,
>
> we all know the the wonderful GZip middleware that allows response
> compression. Which is great especially if you are serving large responses.
>
> Now gzip is only one of many supported encoding types supported by many
> browsers. The coolest kid on the block seems to be brotli right now.
> It is both faster and smaller in most cases then gzip and has been
> standardized in https://www.ietf.org/rfc/rfc7932.txt
>
> Of course there is already a package that adds brotli support to Django:
> https://pypi.org/project/django-brotli/
>
> But I would propose to refactor the GZip middleware into something like
> "CompressionMiddlware" and add support for both GZip and Brotli
> in the same middleware.
>
> I have the feeling something basing like response encoding should be build
> into Django. Of course I would not add a hard dependency to brotli,
> but add all methods to `Accept-Encoding` that are actually installed.
>
> What do you guys think? Is this worth a little refactoring?
>
> Best
> -Joe
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/29cf5142-fa82-4d74-9d8f-5fb0c8be34d0%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Adam

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/CAMyDDM24cy5oo-pvUutFVx1UTdggFmqXkJ8tjK3x8t689Hz-vg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

RFC732 (a.k.a. brotli) support

[Johannes Hoppe]

Hi there,

we all know the the wonderful GZip middleware that allows response 
compression. Which is great especially if you are serving large responses.

Now gzip is only one of many supported encoding types supported by many 
browsers. The coolest kid on the block seems to be brotli right now.
It is both faster and smaller in most cases then gzip and has been 
standardized in https://www.ietf.org/rfc/rfc7932.txt

Of course there is already a package that adds brotli support to Django:
https://pypi.org/project/django-brotli/

But I would propose to refactor the GZip middleware into something like 
"CompressionMiddlware" and add support for both GZip and Brotli
in the same middleware.

I have the feeling something basing like response encoding should be build 
into Django. Of course I would not add a hard dependency to brotli,
but add all methods to `Accept-Encoding` that are actually installed.

What do you guys think? Is this worth a little refactoring?

Best
-Joe

-- 
You received this message because you are subscribed to the Google Groups 
"Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-developers/29cf5142-fa82-4d74-9d8f-5fb0c8be34d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.