Re: [Patch] Async write completion for the full connection filter stack

[Graham Leggett]

Hi all,

I think I have cracked the async problem for both request and connection output 
filters with this patch.

It provides three new functions:

- ap_filter_reinstate_brigade() - Used at the start of a filter, brigades that 
were set aside earlier are reinstated for sending.

- ap_filter_should_yield() - Returns true if downstream filters have set aside 
data. A filter would typically respond by setting aside the data it is working 
with and returning in the expectation of being called again.

- ap_filter_setaside_brigade() - Used at the end of a filter, any brigades that 
were not processed can be set aside to continue the job when called later.

The magic happens in the MPM itself. The first time 
ap_filter_setaside_brigade() is called the filter is added to a hashtable and a 
cleanup is registered to have the filter removed when the request and/or 
connection is cleared. The MPM iterates through this hashtable, and sends empty 
brigades to any filter with setaside data.

Key to this technique is that existing filters remain unaffected - in the 
absence of any changes to a filter the whole brigade will be processed and sent 
downstream, and existing behaviour is maintained. Same with FLUSH buckets - as 
expected, flush behaviour remains unchanged.

If however the filters in the chain are able to setaside buckets, they can 
defer themselves to the write completion phase which in turn can take full 
advantage of the event MPM. These filters will be expected to handle an empty 
brigade to “kick” them back into life and continue the writing of their 
setaside data. As soon as their setaside brigade becomes empty the kicks then 
stop. All filters with setaside data get kicked exactly once, so none of the 
filters should get starved. Filters that are removed from the stack get their 
setaside emptied, and so become ineligible for kicks. When the pool cleanup 
gets triggered, the filter is permanently removed from the connection and 
disappears.

Over and above the network filter the first filter to be modified is mod_ssl, 
and this will allow files served over SSL to take advantage of write 
completion. Another filter that will benefit from using the above calls is 
mod_deflate.

I have an additional goal of adding an ap_filter_suspend() method that will 
allow us to suspend a filter for a given period of time or until some callback 
is triggered, but that will be a separate patch.

Regards,
Graham
--


httpd-async-fullstack-ssl6.patch
Description: Binary data

Re: Lua coroutines

[Jim Jagielski]

The more I think of it, leveraging Lua into the core of httpd would
be a super win. If httpd knew that Lua was embedded, even if mod_lua
itself wasn't enabled, think of the cool stuff we could do.

Also, fwiw, libmill went 1.0 GA today. My main focus has been on
making it not gcc dependent and to work on a larger group of
systems, and not so much on the actual functionality itself,
mostly to make it more possible as a httpd dependent lib :)

> On Sep 27, 2015, at 3:28 AM, Daniel Gruno  wrote:
> 
> Not to mention that should we opt for the Lua coroutines, this would also 
> indirectly mean it would be possible to dynamically alter the HTTP process 
> using custom scripts, which could be very interesting :) sort of like mod_lua 
> on steroids ;). Other ways this could be used would be unit testing, where 
> you could inject things into every crevice of httpd dynamically without 
> having to recompile when you change the test.
> 
> With regards,
> Daniel.
> 
> On 2015-09-26 19:10, Jim Jagielski wrote:
>> I've been keen on libmill, which provides a golang-like
>> goroutine library. It seems that such functionality would
>> be cool for httpd
>> 
>> Then I started thinking: Lua also has a great coroutine impl
>> and it is also MIT licensed and might be easier to fold
>> into httpd. Anyone ever look at that? Having something like
>> ap_routine() would be awful cool. We already have some Lua
>> expertise here and being able to use native coroutines in httpd
>> itself would be very neat, esp for 2.6/3.0
> 

Re: Lua coroutines

[Daniel Gruno]

Not to mention that should we opt for the Lua coroutines, this would 
also indirectly mean it would be possible to dynamically alter the HTTP 
process using custom scripts, which could be very interesting :) sort of 
like mod_lua on steroids ;). Other ways this could be used would be unit 
testing, where you could inject things into every crevice of httpd 
dynamically without having to recompile when you change the test.


With regards,
Daniel.

On 2015-09-26 19:10, Jim Jagielski wrote:

I've been keen on libmill, which provides a golang-like
goroutine library. It seems that such functionality would
be cool for httpd

Then I started thinking: Lua also has a great coroutine impl
and it is also MIT licensed and might be easier to fold
into httpd. Anyone ever look at that? Having something like
ap_routine() would be awful cool. We already have some Lua
expertise here and being able to use native coroutines in httpd
itself would be very neat, esp for 2.6/3.0

Re: svn commit: r1705492 - in /httpd/httpd/branches/2.4.x: ./ docs/manual/mod/ include/ server/ server/mpm/event/ server/mpm/netware/ server/mpm/prefork/ server/mpm/winnt/ server/mpm/worker/

[Marion & Christophe JAILLET]

Hi,

should r1629916 also be included?
The changelog says that it is a follow up to r1629909, which is included 
in the patch below.


CJ

Le 27/09/2015 00:20, minf...@apache.org a écrit :

Author: minfrin
Date: Sat Sep 26 22:20:14 2015
New Revision: 1705492

URL: http://svn.apache.org/viewvc?rev=1705492=rev
Log:
MPMs: Support SO_REUSEPORT to create multiple duplicated listener
records for scalability.

Submitted by: Yingqi Lu , Jeff Trawick,
   Jim Jagielski, Yann Ylavic

Reviewed by: ylavic, jim, minfrin

Modified:
 httpd/httpd/branches/2.4.x/CHANGES
 httpd/httpd/branches/2.4.x/STATUS
 httpd/httpd/branches/2.4.x/docs/manual/mod/mpm_common.xml
 httpd/httpd/branches/2.4.x/include/ap_listen.h
 httpd/httpd/branches/2.4.x/include/ap_mmn.h
 httpd/httpd/branches/2.4.x/include/http_log.h
 httpd/httpd/branches/2.4.x/include/scoreboard.h
 httpd/httpd/branches/2.4.x/server/listen.c
 httpd/httpd/branches/2.4.x/server/log.c
 httpd/httpd/branches/2.4.x/server/mpm/event/event.c
 httpd/httpd/branches/2.4.x/server/mpm/netware/mpm_netware.c
 httpd/httpd/branches/2.4.x/server/mpm/prefork/prefork.c
 httpd/httpd/branches/2.4.x/server/mpm/winnt/mpm_winnt.c
 httpd/httpd/branches/2.4.x/server/mpm/worker/worker.c

Bug report for Apache httpd-2 [2015/09/27]

[bugzilla]

+---+
| Bugzilla Bug ID   |
| +-+
| | Status: UNC=Unconfirmed NEW=New ASS=Assigned|
| | OPN=ReopenedVER=Verified(Skipped Closed/Resolved)   |
| |   +-+
| |   | Severity: BLK=Blocker CRI=Critical  REG=Regression  MAJ=Major   |
| |   |   MIN=Minor   NOR=NormalENH=Enhancement TRV=Trivial |
| |   |   +-+
| |   |   | Date Posted |
| |   |   |  +--+
| |   |   |  | Description  |
| |   |   |  |  |
| 7483|Ass|Enh|2002-03-26|Add FileAction directive to assign a cgi interpret|
| 8713|Inf|Min|2002-05-01|No Errorlog on PROPFIND/Depth:Infinity|
| 8867|Opn|Cri|2002-05-07|exports.c generation fails when using a symlink to|
|10747|New|Maj|2002-07-12|ftp SIZE command and 'smart' ftp servers results i|
|11294|New|Enh|2002-07-30|desired vhost_alias option|
|11580|Opn|Enh|2002-08-09|generate Content-Location headers |
|12033|Opn|Nor|2002-08-26|Graceful restart immediately result in [warn] long|
|12680|New|Enh|2002-09-16|Digest authentication with integrity protection   |
|13599|Inf|Nor|2002-10-14|autoindex formating broken for multibyte sequences|
|13661|Ass|Enh|2002-10-15|Apache cannot not handle dynamic IP reallocation  |
|14104|Opn|Enh|2002-10-30|not documented: must restart server to load new CR|
|14496|New|Enh|2002-11-13|Cannot upgrade any version on Windows. Must uninst|
|14922|Inf|Enh|2002-11-28| is currently hardcoded to 'apache2'  |
|15719|Inf|Nor|2002-12-30|WebDAV MOVE to destination URI which is content-ne|
|16761|Inf|Nor|2003-02-04|CustomLog with pipe spawns process during config  |
|16802|New|Enh|2003-02-05|Additional AllowOverride directive "Restrict" |
|16811|Ass|Maj|2003-02-05|mod_autoindex always return webpages in UTF-8.|
|17107|New|Min|2003-02-16|Windows should not install printenv   |
|17114|New|Enh|2003-02-17|Please add strip and install-strip targets to Make|
|17244|Ass|Nor|2003-02-20|./configure --help gives false information regardi|
|17497|Opn|Nor|2003-02-27|mod_mime_magic generates incorrect response header|
|18325|New|Enh|2003-03-25|PAM support for suEXEC|
|18334|Inf|Cri|2003-03-25|Server crashes when authenticating users against L|
|18497|New|Min|2003-03-30|configure --help gives wrong default for sysconfdi|
|19043|New|Min|2003-04-15|Interesting interaction between cern_meta module a|
|19670|New|Enh|2003-05-05|content type header supplied upon PUT is thrown aw|
|20036|Ass|Nor|2003-05-19|Trailing Dots stripped from PATH_INFO environment |
|21253|New|Nor|2003-07-01|Mime magic doesn't continue if type is specifed fo|
|21260|New|Nor|2003-07-02|CacheMaxExpire directive not enforced !   |
|21533|Ass|Cri|2003-07-11|Multiple levels of htacces files can cause mod_aut|
|22138|Inf|Cri|2003-08-05|Webdav is not preccessing special chars right.|
|22237|New|Enh|2003-08-08|option to disable ServerSignature on index pages  |
|22484|Opn|Maj|2003-08-16|semaphore problem takes httpd down|
|22686|Opn|Nor|2003-08-25|ab: apr_poll: The timeout specified has expired (7|
|22898|Opn|Nor|2003-09-02|nph scripts with two HTTP header  |
|23167|Inf|Cri|2003-09-14|--enable-layout never goes to apr apr-util|
|23181|New|Nor|2003-09-15|Status 304 (Not modified) and chunking leads to an|
|23238|New|Cri|2003-09-18|non-async-signal-safe operations from signal handl|
|23330|New|Enh|2003-09-22|Enhance ApacheMonitor to view and control Tomcat s|
|23911|Opn|Cri|2003-10-18|CGI processes left defunct/zombie under 2.0.54|
|24031|New|Enh|2003-10-23|Passphrase protected private key in SSLProxyMachin|
|24095|Opn|Cri|2003-10-24|ERROR "Parent: child process exited with status 32|
|24437|Opn|Nor|2003-11-05|mod_auth_ldap doubly-escapes backslash (\) charact|
|24890|Opn|Nor|2003-11-21|Apache config parser should not be local aware ( g|
|25014|New|Enh|2003-11-26|A flexible interface for mod_log_config   |
|25201|New|Enh|2003-12-04|Provide Cache Purge operation |
|25240|Inf|Enh|2003-12-05|SSL Library Error: 336105671 logged as information|
|25435|New|Enh|2003-12-11|sethandler and directoryindex not playing nice|
|25469|Opn|Enh|2003-12-12|create AuthRoot for defining paths to auth files  |
|25484|Ass|Nor|2003-12-12|Non-service Apache cannot be stopped in WinXP |
|25543|Inf|Nor|2003-12-15|mod_proxy_ajp overwrites existing response headers|