Re: [PATCH] Introducing mod_brotli

[Jim Jagielski]

Ahh... that's... interesting.

> On Oct 19, 2016, at 8:37 PM, Jacob Champion  wrote:
> 
> On 09/16/2016 05:32 AM, Evgeny Kotkov wrote:
>> This patch adds a module for dynamic Brotli (RFC 7932) compression in httpd.
> 
> Just in case someone else runs into this: I gave mod_brotli a shot after a 
> user on #httpd-dev asked about Brotli compression, and it immediately ran my 
> test server out of memory. :(
> 
> The issue appears to be that the libbrotli ABI changes depending on the C 
> standard in use. (Specifically, the BROTLI_BOOL type is either C99's _Bool or 
> an enum.) So mod_brotli, compiled in maintainer-mode (C89), thinks that 
> "false" boolean values coming back from the library are "true". This leads to 
> some fun infinite loops.
> 
> I've commented upstream, since it looks like the switch from 'int' to 
> 'BROTLI_BOOL' was explicitly requested by another user:
> 
>https://github.com/google/brotli/issues/384
> 
> --Jacob

Re: [PATCH] Introducing mod_brotli

[Jacob Champion]

On 10/19/2016 06:13 PM, William A Rowe Jr wrote:

enum boolval {
   false = 0;

is not that challenging.


Yeah, but I tend to turn up my nose at enums in ABIs too, because the 
question "how big is an enum?" has a more complicated answer than the 
question "how big is an int?".


Ah well, it's not our project. I'll be interested to see how they respond.

--Jacob

Re: [PATCH] Introducing mod_brotli

[William A Rowe Jr]

enum boolval {
   false = 0;

is not that challenging.

On Oct 19, 2016 7:38 PM, "Jacob Champion"  wrote:

> On 09/16/2016 05:32 AM, Evgeny Kotkov wrote:
>
>> This patch adds a module for dynamic Brotli (RFC 7932) compression in
>> httpd.
>>
>
> Just in case someone else runs into this: I gave mod_brotli a shot after a
> user on #httpd-dev asked about Brotli compression, and it immediately ran
> my test server out of memory. :(
>
> The issue appears to be that the libbrotli ABI changes depending on the C
> standard in use. (Specifically, the BROTLI_BOOL type is either C99's _Bool
> or an enum.) So mod_brotli, compiled in maintainer-mode (C89), thinks that
> "false" boolean values coming back from the library are "true". This leads
> to some fun infinite loops.
>
> I've commented upstream, since it looks like the switch from 'int' to
> 'BROTLI_BOOL' was explicitly requested by another user:
>
> https://github.com/google/brotli/issues/384
>
> --Jacob
>

Re: [PATCH] Introducing mod_brotli

[Jacob Champion]

On 09/16/2016 05:32 AM, Evgeny Kotkov wrote:

This patch adds a module for dynamic Brotli (RFC 7932) compression in httpd.


Just in case someone else runs into this: I gave mod_brotli a shot after 
a user on #httpd-dev asked about Brotli compression, and it immediately 
ran my test server out of memory. :(


The issue appears to be that the libbrotli ABI changes depending on the 
C standard in use. (Specifically, the BROTLI_BOOL type is either C99's 
_Bool or an enum.) So mod_brotli, compiled in maintainer-mode (C89), 
thinks that "false" boolean values coming back from the library are 
"true". This leads to some fun infinite loops.


I've commented upstream, since it looks like the switch from 'int' to 
'BROTLI_BOOL' was explicitly requested by another user:


https://github.com/google/brotli/issues/384

--Jacob

Re: [PATCH] Introducing mod_brotli

[Evgeny Kotkov]

Evgeny Kotkov  writes:

>>> Wow! This is great stuff. Brotli support has been in my TODO
>>> queue for awhile.
>>>
>>> Thanks!
>>
>> +1, cool stuff and thanks!
>
> Glad to hear that, thanks everyone.
>
> I would be happy to continue the work on this module, for instance, by
> adding the necessary documentation and the ability to log compression ratio.

So, my current plan is to commit the V1 patch, prepare the necessary bits
of documentation (marking the module as experimental), and then continue
shaping it up by adding the optional, but important things, such as the
compression ratio logging.

I'll start doing this tomorrow.


Regards,
Evgeny Kotkov

Re: [PATCH] Introducing mod_brotli

[Evgeny Kotkov]

Reindl Harald  writes:

> agreed - however, below some configs where my brain rumours how have that
> identically behavior by just use "brotli" compression in case the cient
> supports it - maybe someone with deeper insights as my pure adiminstrator
> view has a idea by looking at it
>
> the "no-gzip dont-vary" stuff is for long running scripts with
> output-flushing to give "realtime" feedback instead have it all buffered
>
> one brainstorming: "AddOutputCompressionByType" provided by whatever module,
> proceed the Accept-Encoding of the client and deciding the compression algo

Adding the new output filter to the AddOutputFilterByType directive will
result in what you're looking for.  For instance, changing

  AddOutputFilterByType DEFLATE text/html

to

  AddOutputFilterByType BROTLI_COMPRESS;DEFLATE text/html

means that the server will send Brotli-compressed data to clients that have
"br" in their Accept-Encoding request header, and Deflate data to other
clients that indicate gzip/deflate support.

> SetEnvIfNoCase Request_URI (download.php)$ no-gzip dont-vary
> SetEnvIfNoCase Request_URI (download_imgzip.php)$ no-gzip dont-vary
> SetEnvIfNoCase Request_URI (presse_download_zip.php)$ no-gzip dont-vary
> SetEnvIfNoCase Request_URI (content_sub_move.php)$ no-gzip dont-vary
> SetEnvIfNoCase Request_URI (synch.php)$ no-gzip dont-vary
> SetEnvIfNoCase Request_URI (import.php)$ no-gzip dont-vary
> SetEnvIfNoCase Request_URI (admin_imagecopyrights.php)$ no-gzip dont-vary
> SetEnvIfNoCase Request_URI (newsletter.php)$ no-gzip dont-vary
> SetEnvIfNoCase Request_URI (importer.php)$ no-gzip dont-vary
> SetEnvIfNoCase Request_URI (create.php)$ no-gzip dont-vary

The V1 patch doesn't add the equivalent of "no-gzip" (like, "no-brotli") env
variable.  This is left out for future work, and until then these statements
cannot be migrated seamlessly.  However, I think that the wanted behavior
here could be achieved with mod_filter's FilterProvider directive using
an expression that doesn't install any compression filters for the scripts.


Regards,
Evgeny Kotkov

Re: [PATCH] Introducing mod_brotli

[Reindl Harald]

Am 19.09.2016 um 19:56 schrieb Jacob Champion:

On 09/19/2016 10:12 AM, Eric Covener wrote:


I would prefer to keep them separate even if we have to teach something
to coordinate them (a module, some new support in mod_filter, some
kind of hook?)



+1. (If it proves difficult to make separate compression modules play
well together, that's a problem we should fix.)


agreed - however, below some configs where my brain rumours how have 
that identically behavior by just use "brotli" compression in case the 
cient supports it - maybe someone with deeper insights as my pure 
adiminstrator view has a idea by looking at it


the "no-gzip dont-vary" stuff is for long running scripts with 
output-flushing to give "realtime" feedback instead have it all buffered


one brainstorming: "AddOutputCompressionByType" provided by whatever 
module, proceed the Accept-Encoding of the client and deciding the 
compression algo


__

Logging to have the compression ratio in the access-logs

DeflateFilterNote   Ratio ratio_info
LogFormat   "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" 
\"%{User-Agent}i\" (%{ratio_info}n%%)" combined

ErrorLog"/Volumes/dune/www-servers/_logs/apache_error.log"
CustomLog 
"/Volumes/dune/www-servers/_logs/apache_access.log" combined

LogLevelnotice core:info
__

[root@testserver:~]$ cat /etc/httpd/conf/httpd-deflate.conf
DeflateCompressionLevel 2
DeflateBufferSize 32768


 AddOutputFilterByType DEFLATE text/html
 AddOutputFilterByType DEFLATE text/javascript
 AddOutputFilterByType DEFLATE text/css
 AddOutputFilterByType DEFLATE text/plain
 AddOutputFilterByType DEFLATE text/xml
 AddOutputFilterByType DEFLATE text/x-component
 AddOutputFilterByType DEFLATE application/xhtml+xml
 AddOutputFilterByType DEFLATE application/xml+rss
 AddOutputFilterByType DEFLATE application/rss+xml
 AddOutputFilterByType DEFLATE application/xml
 AddOutputFilterByType DEFLATE application/javascript
 AddOutputFilterByType DEFLATE application/x-javascript
 AddOutputFilterByType DEFLATE application/msword
 AddOutputFilterByType DEFLATE application/msexcel
 AddOutputFilterByType DEFLATE application/mspowerpoint
 AddOutputFilterByType DEFLATE application/msaccess
 AddOutputFilterByType DEFLATE application/mshelp
 AddOutputFilterByType DEFLATE application/pdf
 AddOutputFilterByType DEFLATE application/postscript
 AddOutputFilterByType DEFLATE audio/x-wav
 AddOutputFilterByType DEFLATE text/rtf
 AddOutputFilterByType DEFLATE text/comma-separated-values
 AddOutputFilterByType DEFLATE text/tab-separated-values
 AddOutputFilterByType DEFLATE text/vnd.wap.wml
 AddOutputFilterByType DEFLATE text/vnd.wap.wmlscript
 AddOutputFilterByType DEFLATE text/vnd.wap.wmlscript
 AddOutputFilterByType DEFLATE application/vnd.wap.wmlc
 AddOutputFilterByType DEFLATE text/x-setext
 AddOutputFilterByType DEFLATE text/x-sgml
 AddOutputFilterByType DEFLATE text/x-speech
 AddOutputFilterByType DEFLATE application/x-sh
 AddOutputFilterByType DEFLATE application/x-latex
 AddOutputFilterByType DEFLATE application/x-httpd-php-source
 AddOutputFilterByType DEFLATE application/vnd.ms-fontobject
 AddOutputFilterByType DEFLATE font/ttf
 AddOutputFilterByType DEFLATE font/otf
 AddOutputFilterByType DEFLATE font/x-woff
 AddOutputFilterByType DEFLATE image/svg+xml
 AddOutputFilterByType DEFLATE 
application/vnd.ms-word.document.macroEnabled.12
 AddOutputFilterByType DEFLATE 
application/vnd.openxmlformats-officedocument.wordprocessingml.document
 AddOutputFilterByType DEFLATE 
application/vnd.ms-word.template.macroEnabled.12
 AddOutputFilterByType DEFLATE 
application/vnd.openxmlformats-officedocument.wordprocessingml.template
 AddOutputFilterByType DEFLATE 
application/vnd.ms-powerpoint.template.macroEnabled.12
 AddOutputFilterByType DEFLATE 
application/vnd.openxmlformats-officedocument.presentationml.template
 AddOutputFilterByType DEFLATE 
application/vnd.ms-powerpoint.addin.macroEnabled.12
 AddOutputFilterByType DEFLATE 
application/vnd.ms-powerpoint.slideshow.macroEnabled.12
 AddOutputFilterByType DEFLATE 
application/vnd.openxmlformats-officedocument.presentationml.slideshow
 AddOutputFilterByType DEFLATE 
application/vnd.ms-powerpoint.presentation.macroEnabled.12
 AddOutputFilterByType DEFLATE 
application/vnd.openxmlformats-officedocument.presentationml.presentation
 AddOutputFilterByType DEFLATE 
application/vnd.ms-excel.addin.macroEnabled.12
 AddOutputFilterByType DEFLATE 
application/vnd.ms-excel.sheet.binary.macroEnabled.12
 AddOutputFilterByType DEFLATE 
application/vnd.ms-excel.sheet.macroEnabled.12
 AddOutputFilterByType DEFLATE 
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
 AddOutputFilterByType DEFLATE 
application/vnd.ms-excel.template.macroEnabled.12
 AddOutputFilterByType DEFLATE 
application/vnd.openxmlformats-officedocument.spreadsheetml.template



SetEnvIfNoCase Request_URI (download.php)$ no-gzip 

Re: [PATCH] Introducing mod_brotli

[Jacob Champion]

On 09/19/2016 10:12 AM, Eric Covener wrote:


I would prefer to keep them separate even if we have to teach something
to coordinate them (a module, some new support in mod_filter, some
kind of hook?)



+1. (If it proves difficult to make separate compression modules play 
well together, that's a problem we should fix.)


--Jacob

Re: [PATCH] Introducing mod_brotli

[Eric Covener]

On Mon, Sep 19, 2016 at 11:35 AM, Reindl Harald  wrote:
> just an idea - wouldn't it make sense to add 'br' support for mod_deflate
> and have it preferred when the client says in it's request headers that it
> supports the encoding instead having two modules for the same thing just
> using different comrepssion algos?

> if not i am fine and grateful too - though about that again after
> considering how to include it and prefer mod_brotli over mod_defalte for
> clients whcih support it while mod_deflate is still needed for old clients
> only knowing about gzip/deflate

I would prefer to keep them separate even if we have to teach something
to coordinate them (a module, some new support in mod_filter, some
kind of hook?)

-- 
Eric Covener
cove...@gmail.com

Re: [PATCH] Introducing mod_brotli

[Reindl Harald]

Am 19.09.2016 um 16:14 schrieb Evgeny Kotkov:

Eric Covener  writes:


Wow! This is great stuff. Brotli support has been in my TODO
queue for awhile.

Thanks!


+1, cool stuff and thanks!


Glad to hear that, thanks everyone.

I would be happy to continue the work on this module, for instance, by
adding the necessary documentation and the ability to log compression ratio


just an idea - wouldn't it make sense to add 'br' support for 
mod_deflate and have it preferred when the client says in it's request 
headers that it supports the encoding instead having two modules for the 
same thing just using different comrepssion algos?


if not i am fine and grateful too - though about that again after 
considering how to include it and prefer mod_brotli over mod_defalte for 
clients whcih support it while mod_deflate is still needed for old 
clients only knowing about gzip/deflate

Re: [PATCH] Introducing mod_brotli

[Evgeny Kotkov]

Eric Covener  writes:

>> Wow! This is great stuff. Brotli support has been in my TODO
>> queue for awhile.
>>
>> Thanks!
>
> +1, cool stuff and thanks!

Glad to hear that, thanks everyone.

I would be happy to continue the work on this module, for instance, by
adding the necessary documentation and the ability to log compression ratio.


Regards,
Evgeny Kotkov

Re: [PATCH] Introducing mod_brotli

[Eric Covener]

On Mon, Sep 19, 2016 at 9:55 AM, Jim Jagielski  wrote:
> Wow! This is great stuff. Brotli support has been in my TODO
> queue for awhile.
>
> Thanks!

+1, cool stuff and thanks!

-- 
Eric Covener
cove...@gmail.com

Re: [PATCH] Introducing mod_brotli

[Jim Jagielski]

Wow! This is great stuff. Brotli support has been in my TODO
queue for awhile.

Thanks!

> On Sep 16, 2016, at 8:32 AM, Evgeny Kotkov  
> wrote:
> 
> Hi all,
> 
> This patch adds a module for dynamic Brotli (RFC 7932) compression in httpd.
> 
> The new compression format is supported by Mozilla Firefox since 44.0 and
> by Google Chrome since 50.0 [1, 2], and both nginx and IIS have modules that
> offer Brotli compression.
> 
> With the new module, existing mod_deflate installations can benefit from
> better compression ratio by sending Brotli-compressed data to the clients
> that support it:
> 
>LoadModule brotli_module modules/mod_brotli.so
>LoadModule deflate_module modules/mod_deflate.so
>SetOutputFilter BROTLI_COMPRESS;DEFLATE
> 
> This module features zero-copy processing, which is only possible with the
> new API from the upcoming 1.0.x series of brotli [3].
> 
> The Linux makefile works against libbrotli [4], as currently the core brotli
> repository doesn't offer a way to build a library [5].  Enabling mod_brotli
> can be done with:
> 
>configure --enable-brotli=shared [--with-brotli=]
> 
> CMake build is supported as well.  Please note that the patch doesn't include
> the documentation updates and other types of makefiles, but I will do it
> separately.
> 
> The second patch adds a couple of tests to the test framework.
> 
> [1] https://www.mozilla.org/en-US/firefox/44.0/releasenotes/
> [2] https://www.chromestatus.com/feature/5420797577396224
> [3] https://github.com/google/brotli
> [4] https://github.com/bagder/libbrotli
> [5] https://github.com/google/brotli/pull/332
> 
> 
> Regards,
> Evgeny Kotkov
> 

Re: [PATCH] Introducing mod_brotli

[Evgeny Kotkov]

Reindl Harald  writes:

> how is the ordering?
> defined by SetOutputFilter or client?

Currently, the order is defined by SetOutputFilter, because AFAIK there
is no centralized way to handle Accept-Encoding priorities (like ;q=0.7).

> does it also support (%{ratio_info}n%%) in the log configuration?
>
> LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"
> (%{ratio_info}n%%)" combined
>
> leads to:
>
> 213.47.77.186 - - [16/Sep/2016:15:13:28 +0200] "GET / HTTP/1.1" 200 4133 ""
> "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.36 (KHTML,
> like Gecko) Chrome/49.0.2623.112 Safari/537.36" (38%)
>
> would especially when you compare clients with and without support nice to
> see the difference here

It would certainly be nice, but this version of the patch doesn't have it
yet.  I could do this separately or provide the V2 patch that supports
ratio_info.


Regards,
Evgeny Kotkov

Re: [PATCH] Introducing mod_brotli

[Reindl Harald]

Am 16.09.2016 um 14:59 schrieb Stefan Eissing:

Sweet!


Am 16.09.2016 um 14:32 schrieb Evgeny Kotkov :

Hi all,

This patch adds a module for dynamic Brotli (RFC 7932) compression in httpd.

The new compression format is supported by Mozilla Firefox since 44.0 and
by Google Chrome since 50.0 [1, 2], and both nginx and IIS have modules that
offer Brotli compression.

With the new module, existing mod_deflate installations can benefit from
better compression ratio by sending Brotli-compressed data to the clients
that support it:

   LoadModule brotli_module modules/mod_brotli.so
   LoadModule deflate_module modules/mod_deflate.so
   SetOutputFilter BROTLI_COMPRESS;DEFLATE


sounds good - 20% better compression AFAIK

how is the ordering?
defined by SetOutputFilter or client?

looked at my firefox request headers and "br" is at the last position
Accept-Encoding: gzip, deflate, br
__

does it also support (%{ratio_info}n%%) in the log configuration?

LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" 
(%{ratio_info}n%%)" combined


leads to:

213.47.77.186 - - [16/Sep/2016:15:13:28 +0200] "GET / HTTP/1.1" 200 4133 
"" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.36 
(KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36" (38%)


would especially when you compare clients with and without support nice 
to see the difference here